8000 feat(smb): import files added over the SMB share by adeelahmad · Pull Request #519 · adeelahmad/lense · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,13 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version
Markdown or plain text and exports it, questions and code blocks have their own copy buttons, and the conversation's
header has Export; without a PDF printer on the server, the browser prints it instead
([Copy and export](docs/assistant.md#copy-and-export)).
- **Files added over SMB are imported.** Copy files into a collection's folder on the SMB share and each is imported
when it's closed, as an upload is; a new folder under `Collections` is a new collection. Files of a type Lens doesn't
take are refused as they're made, files over the upload limit are cut off as they're written, and a file deleted as
it's closed or left open when its client goes away isn't imported. The system's own files (`.DS_Store`, `Thumbs.db`,
Office's `~$` files) are kept aside, and renaming a file saved under a temporary name imports it. **Import files
people add** in Settings, SMB share (`smb.add_files`, on) turns it off and makes the share read-only again ([Adding
files](docs/smb.md#adding-files)).
- **News feeds.** Subscribe to a blog's or a news site's RSS or Atom feed: add it on the Sources page (its address,
or just the site's) and watch it, and each new post comes into a namespace as text, with its link, author and date,
so answers cite it. Feeds are asked for again only when they've changed, and a private feed's address is kept like a
Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,8 +70,8 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p
answer OpenAI's chat completions API for any app, cited from that namespace alone, each off until its owner turns it
on, with API keys that open that one namespace and nothing else. The same folders and files can be mounted over S3,
WebDAV, NFS or SMB (S3 and WebDAV also at the web app's address, through 8000 remote access), so file managers and scripts
read Lens too, and a file copied in over S3 or WebDAV is imported where it was put. The web app runs on the same REST
API, so an agent can do what a person can. ([MCP](docs/mcp.md), [one namespace's MCP
read Lens too, and a file copied in over S3, WebDAV or SMB is imported where it was put. The web app runs on the same
REST API, so an agent can do what a person can. ([MCP](docs/mcp.md), [one namespace's MCP
server](docs/mcp.md#one-namespaces-mcp-server), [chat completions](docs/api.md#chat-completions-for-any-app),
[API](docs/api.md), [the graph as files](docs/file-tree.md), [authentication](docs/authentication.md);
[checks](docs/requirements.md#connect-your-agents))
Expand Down
14 changes: 7 additions & 7 deletions docs/file-tree.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,15 +3,15 @@
Lens shows its graph as an ordinary tree of folders and files: graph nodes (collections, topics, entities) are
folders, nested along the graph's links, and every asset (a recording, document or image) is a few files inside the
folders it belongs to, its metadata in a sidecar beside it. Agents and tools that already understand files can read
Lens without learning its API, and add to it the same way: a file copied into a collection's folder over S3 or WebDAV
is imported there ([Adding files](#adding-files)).
Lens without learning its API, and add to it the same way: a file copied into a collection's folder over S3, WebDAV or
SMB is imported there ([Adding files](#adding-files)).

| Where | Status |
| --- | --- |
| The API (`GET /api/v1/tree`, `GET /api/v1/tree/file`) and the MCP tools `list_files` and `read_file` | built |
| An [S3 server](#s3) and a [WebDAV server](#webdav) (off until turned on); files added through them are imported | built |
| An [NFS share](file-shares.md) (read-only, off until turned on, allowed networks only) | built |
| An [SMB share](smb.md) (read-only, off until turned on) | built |
| An [SMB share](smb.md) (off until turned on); files added through it are imported | built |

Every way in serves the same tree, built by `app/domain/filetree.py`.

Expand Down Expand Up @@ -47,7 +47,7 @@ Names are made safe for every file system: `/ \ : * ? " < > |` and control chara
Exactly what the web app shows them ([Access](access.md)): an admin sees every namespace; others see the namespaces
they have a role in, and in a namespace where they only have roles on some collections, those collections and the
topics and entities of the assets in them (with the topics broader than those). Anything else answers "not found". The API only
reads; files are added through the [S3](#s3) and [WebDAV](#webdav) servers.
reads; files are added through the [S3](#s3) and [WebDAV](#webdav) servers and the [SMB share](smb.md).

## The API

Expand Down Expand Up @@ -118,7 +118,7 @@ behind TLS on a network you don't trust.

## Adding files

A file copied into the tree over S3 or WebDAV is imported, as an upload in the web app is: the same checks of its type
A file copied into the tree over S3, WebDAV or [SMB](smb.md) is imported, as an upload in the web app is: the same checks of its type
and size (Settings, Uploads), the namespace's pipeline, and the same file twice is added once. Where it's put decides
where it goes:

Expand All @@ -143,8 +143,8 @@ after they last change, seen only by you, and never imported; renaming one to an
kept on the uploads disk, in memory per worker, so a restart forgets them.

Adding files is on whenever the shares are; turn it off in Settings, File shares (**Import files people add**,
`shares.add_files`) and they're read-only again. `GET /api/v1/shares/s3` says whether it's on (`add_files`). The
[NFS](file-shares.md) and [SMB](smb.md) shares stay read-only for now.
`shares.add_files`) and they're read-only again. `GET /api/v1/shares/s3` says whether it's on (`add_files`). The SMB
share has its own switch (`smb.add_files`, on). The [NFS](file-shares.md) share stays read-only for now.

## At the web app's address

Expand Down
2 changes: 1 addition & 1 deletion docs/requirements.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ request, to another check that proves the same thing; loosening a requirement or
| API keys that open one namespace and nothing else | [test_namespace_keys.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_namespace_keys.py) `test_a_key_for_one_namespace_opens_its_servers_alone`, `test_it_has_the_persons_role_there_and_no_more` |
| The graph's folders and files mounted read-only over S3, WebDAV, NFS and SMB | [test_s3.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_s3.py) `test_people_read_their_namespaces_over_s3`, `test_what_s3_refuses`; [test_webdav.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_webdav.py) `test_mount_and_read`, `test_what_webdav_refuses`; [test_nfs_share.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_nfs_share.py) `test_the_runner_follows_the_setting`; [test_smb_share.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_smb_share.py) `test_reads_the_tree_as_its_key_sees_it`, `test_nothing_can_be_written` |
| S3 and WebDAV at the web app's address, through remote access, once an admin allows it | [test_s3.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_s3.py) `test_at_the_web_apps_address_once_an_admin_allows_it`; [test_webdav.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_webdav.py) `test_at_the_web_apps_address_once_an_admin_allows_it`, `test_through_the_web_app_each_visitor_has_its_own_throttle` |
| A file copied in over S3 or WebDAV is imported where it was put, with an upload's checks and permissions | [test_s3_writes.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_s3_writes.py) `test_a_file_put_into_a_collection_is_imported_there`, `test_folders_become_collections`, `test_multipart_uploads`, `test_what_may_not_be_added_is_refused`, `test_files_added_at_the_web_apps_address`; [test_webdav_writes.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_webdav_writes.py) `test_finder_copies_a_file`, `test_windows_copies_a_file`, `test_a_folder_dragged_in_becomes_collections`, `test_what_may_not_be_added_is_refused` |
| A file copied in over S3, WebDAV or SMB is imported where it was put, with an upload's checks and permissions | [test_s3_writes.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_s3_writes.py) `test_a_file_put_into_a_collection_is_imported_there`, `test_folders_become_collections`, `test_multipart_uploads`, `test_what_may_not_be_added_is_refused`, `test_files_added_at_the_web_apps_address`; [test_webdav_writes.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_webdav_writes.py) `test_finder_copies_a_file`, `test_windows_copies_a_file`, `test_a_folder_dragged_in_becomes_collections`, `test_what_may_not_be_added_is_refused`; [test_smb_writes.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_smb_writes.py) `test_a_file_copied_into_a_collection_is_imported_there`, `test_new_folders_under_collections_are_new_collections`, `test_what_may_not_be_added_is_refused`, `test_the_share_is_read_only_while_add_files_is_off` |
| The web app runs on the same REST API, through a client generated from its OpenAPI schema | [ci.yml](https://github.com/adeelahmad/lense/blob/main/.github/workflows/ci.yml) `OpenAPI schema is up to date` |

## Ask, and get cited answers
Expand Down
37 changes: 30 additions & 7 deletions docs/smb.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,10 @@

Lens can serve [the file tree](file-tree.md) over SMB, the protocol Finder, Windows Explorer and Linux file managers
use for network drives. Namespaces, collections, topics and entities are folders; assets are files inside them, with
their metadata and text beside them. Everyone signs in as themselves and sees exactly what the web app shows them.
their metadata and text beside them. Everyone signs in as themselves and sees exactly what the web app shows them, and
a file they copy into a collection's folder is imported there ([Adding files](file-tree.md#adding-files)).

*Status: built. Read-only.*
*Status: built.*

## Turn it on

Expand All @@ -19,12 +20,13 @@ within 30 seconds.
| Where | How |
| --- | --- |
| macOS | Finder → Go → Connect to Server → `smb://<lens-machine>:1445/LENS` |
| Linux | `sudo mount -t cifs -o port=1445,username=<key id>,ro,vers=2.0 //<lens-machine>/LENS /mnt/lens`, or `smb://<lens-machine>:1445/LENS` in Files |
| Linux | `sudo mount -t cifs -o port=1445,username=<key id>,vers=2.0 //<lens-machine>/LENS /mnt/lens` (add `ro` to only read), or `smb://<lens-machine>:1445/LENS` in Files |
| Windows | Explorer only connects to port 445: install with `LENS_SMB_PORT=445` in front of the install line (kept for later upgrades), then open `\\<lens-machine>\LENS` |

| Setting | Default | What it does |
| --- | --- | --- |
| `smb.enabled` | off | Serve the share |
| `smb.add_files` | on | Import the files people copy in (**Import files people add**); off, the share is read-only. Turning it on or off restarts the share, which ends open sessions |
| `smb.allowed_networks` | private networks (`10.0.0.0/8`, `172.16.0.0/12`, `192.168.0.0/16`, `127.0.0.0/8`, `100.64.0.0/10` for Tailscale, `fc00::/7`, `::1/128`) | Machines anywhere else are turned away before they can sign in; changes apply without a restart |
| `smb.bind`, `smb.port` | `127.0.0.1`, `4445` | Startup settings only: where the worker listens (`LENS_SMB_LISTEN`, `LENS_SMB_LISTEN_PORT`). In Docker it listens on `0.0.0.0:4445` inside the worker container |

Expand All @@ -36,10 +38,29 @@ shares on their own while the web app stays local.
Agents read the share's state at `GET /api/v1/settings/smb/status` and turn it on or off with
`PUT /api/v1/settings/smb` (`{"enabled": true}`), like every other setting.

## Adding files

Copy files into a collection's folder (with a file manager, `cp` onto a mount, or `smbclient`'s `put`) and each is
imported when it's closed, as an upload is: the same checks of its type and size, the namespace's pipeline,
and the same file twice is added once. A new folder under `Collections` is a new collection; a file anywhere else in a
namespace goes to its default collection. It takes the import permission (or an editor's role on the collection), as
the web app does; [Adding files](file-tree.md#adding-files) has the rest.

A file is refused as it's made when its type can't be uploaded, and cut off as it's written once it's larger than
uploads take (`uploads.max_mb`); a file that was cut off, deleted as it was closed, or still open when its client went
away isn't imported. Files the system writes for itself (`.DS_Store`, `._` files, `Thumbs.db`, `desktop.ini`, Office's
`~$` files) and files saved under a temporary name are kept aside for a while, seen only by you; renaming one to an
ordinary name imports it, which is how apps that save to a temporary file and rename it work. What Lens made can't be
written over, renamed or deleted.

## What keeps it safe

- **Off by default**, and nothing listens until an admin turns it on.
- **Read-only.** Every call that would write, rename, delete or make a folder is refused before it reaches the tree.
- **Adds, never changes.** A file is imported only where its owner may import, checked as an upload is. Nothing Lens
made can be written over, renamed or deleted, and namespaces can't be made. With `smb.add_files` off, every call that
would write, rename, delete or make a folder is refused before it reaches the tree.
- **Files arrive on Lens's disk, not in memory.** A file being written goes to the uploads folder, cut off at the
upload size limit, and is imported or dropped when it's closed.
- **Per-person views.** A key opens its owner's view of the tree: the namespaces and collections they have a role in,
nothing else. A disabled account or a revoked key stops working within 30 seconds, mid-session too.
- **No guests.** Anonymous and guest sessions get nothing; a share is only handed to a signed-in key.
Expand All @@ -55,8 +76,9 @@ port off the internet.

The worker runs impacket's SMB server (pure Python; impacket's license is a modified Apache 1.1, and it's fetched only
when the share is turned on, as the `smb` extra). impacket serves folders through Python's `os` module; in Lens, its
`os` is a small read-only stand-in (`domain/smb_share.py`) that looks files up in the tree for the signed-in key's
owner, and reads them as they stream (files in vaults are decrypted on the way; files in storage connections are
`os` is a small stand-in (`domain/smb_share.py`) that looks files up in the tree for the signed-in key's
owner, writes files people add to the uploads folder and hands them to the importer (`domain/share_import.py`) as
they're closed, and reads files as they stream (files in vaults are decrypted on the way; files in storage connections are
fetched as they're read). One person's view of the tree is reused for a few seconds, so the many look-ups a file
manager makes after listing a folder don't each ask the database again.

Expand All @@ -65,4 +87,5 @@ manager makes after listing a folder don't each ask the database again.
- SMB 3 with encryption on the wire (impacket speaks SMB 2.0.2).
- Finder's Bonjour discovery, so the share shows up on its own.
- Packaging for the Synology and QNAP compose files, which don't publish the port yet.
- Writing: dropping a file into a collection's folder to import it.
- Delete, rename and move assets through the share (only files just added can be today).
- Free space as the uploads disk has it (impacket reports a fixed size).
6 changes: 3 additions & 3 deletions fastapi_backend/app/domain/settings.py
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@
"nfs": ("enabled", "port", "allowed_networks", "account"),
# bind is a startup setting only
# the SMB share of the file tree (smb_share.py): bind and port are startup settings only (Docker publishes the port)
"smb": ("enabled", "allowed_networks"),
"smb": ("enabled", "allowed_networks", "add_files"),
"sensors": (
"enabled",
"mqtt",
Expand Down Expand Up @@ -1306,9 +1306,9 @@ def _nfs_setting(key, value):


def _smb_setting(key, value):
if key == "enabled":
if key in ("enabled", "add_files"):
if not isinstance(value, bool):
raise ValueError("smb.enabled is true or false")
raise ValueError(f"smb.{key} is true or false")
return value
if not isinstance(value, list) or len(value) > 100:
raise ValueError("smb.allowed_networks is a list of networks like 192.168.1.0/24")
Expand Down
Loading
Loading
0