8000 feat(shares): import files added over S3 and WebDAV by adeelahmad · Pull Request #408 · adeelahmad/lense · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
ff828f9
feat(shares): import files added over S3 and WebDAV
claude Oct 15, 2026
46a7aa1
Merge remote-tracking branch 'origin/main' into claude/project-thread…
claude Oct 15, 2026
ab2f225
Merge remote-tracking branch 'origin/main' into claude/project-thread…
claude Oct 15, 2026
c786ffe
Merge remote-tracking branch 'origin/main' into claude/project-thread…
claude Oct 15, 2026
2c4e0ef
fix(shares): let a refused upload's client read the answer
claude Oct 15, 2026
f8c6bf2
Merge remote-tracking branch 'origin/main' into claude/project-thread…
claude Oct 15, 2026
70131ae
Merge remote-tracking branch 'origin/main' into claude/project-thread…
claude Oct 17, 2026
19f3804
Merge remote-tracking branch 'origin/main' into claude/project-thread…
claude Oct 17, 2026
6ed05d7
Merge remote-tracking branch 'origin/main' into claude/project-thread…
claude Oct 17, 2026
8abf36c
Merge remote-tracking branch 'origin/main' into claude/project-thread…
claude Oct 17, 2026
ec6ac0a
Merge remote-tracking branch 'origin/main' into claude/project-thread…
claude Oct 17, 2026
c0790b3
docs(readme): files added over S3 and WebDAV are built, with their ch…
claude Oct 17, 2026
35accca
Merge remote-tracking branch 'origin/main' into claude/project-thread…
claude Oct 17, 2026
9c46640
fix(ci): the upgrade check counts entities of every type
claude Oct 17, 2026
c6087cf
Merge remote-tracking branch 'origin/main' into claude/project-thread…
claude Oct 17, 2026
5c4a97c
Merge remote-tracking branch 'origin/main' into claude/project-thread…
claude Oct 17, 2026
f795592
Merge remote-tracking branch 'origin/main' into claude/project-thread…
claude Oct 17, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,14 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version
shares; then a new access key's dialog gives those addresses. Signing in is the same: a person's own access key,
read-only. MQTT, syslog, SMB and NFS aren't HTTP and stay on their own ports
([The graph as files](docs/file-tree.md#at-the-web-apps-address), [Remote access](docs/remote-access.md#what-it-reaches)).
- **Files added over S3 and WebDAV are imported.** Copy a file into a collection's folder (`rclone copy`, Finder,
Windows Explorer, any S3 client) and it's imported there as an upload is, with the same checks and the namespace's
pipeline; a new folder under `Collections` is a new collection, and a file anywhere else goes to the namespace's
default collection. It takes the import permission (or an editor's role on the collection), files the system writes
for itself (`.DS_Store`, `Thumbs.db`, half-written files) are never imported, and a re-sync copies nothing twice.
The S3 server now takes streamed and multipart uploads; WebDAV takes PUT, MKCOL and Finder's and Explorer's ways of
copying. This is on whenever the shares are; **Import files people add** in Settings, File shares
(`shares.add_files`) turns it off and makes them read-only again ([Adding files](docs/file-tree.md#adding-files)).
- **The web clipper takes your API token, or just signs in.** It used to take only an import webhook's token, so an API
token was refused with "Lens didn't recognise this token: make a new import webhook and paste its token". Now its
options have **Sign in**: Lens asks you to allow the clipper, nothing to copy, and clips go into the namespace you
Expand Down
9 changes: 4 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,9 +68,10 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p
podcasts. They sign in with OAuth and see only what their person may read; the account menu copies the MCP address in
one click. A namespace can also have an MCP server of its own (`/mcp/<namespace>`) and answer OpenAI's chat
completions API for any app, cited from that namespace alone, each off until its owner turns it on, with API keys that
open that one namespace and nothing else. The same folders and files can be mounted read-only over S3, WebDAV, NFS or
SMB, so file managers and scripts read Lens too. The web app runs on the same REST API, so an agent can do what a
person can.
open that one namespace and nothing else. The same folders and files can be mounted over S3, WebDAV, NFS or SMB (S3
and WebDAV also at the web app's address, through remote access), so file managers and scripts read Lens too, and a
file copied in over S3 or WebDAV is imported where it was put. The web app runs on the same REST API, so an agent can
do what a person can.
([MCP](docs/mcp.md), [one namespace's MCP server](docs/mcp.md#one-namespaces-mcp-server),
[chat completions](docs/api.md#chat-completions-for-any-app), [API](docs/api.md),
[the graph as files](docs/file-tree.md), [authentication](docs/authentication.md);
Expand Down Expand Up @@ -212,8 +213,6 @@ These are open pull requests, not yet part of Lens:
- **Web clipper:** signing in, or using your API token, not only an import webhook's
([#406](https://github.com/adeelahmad/lense/pull/406)).
- **Automations:** more operators for conditions ([#434](https://github.com/adeelahmad/lense/pull/434)).
- **Shares:** S3 and WebDAV reached through remote access ([#402](https://github.com/adeelahmad/lense/pull/402)),
and files added over them imported ([#408](https://github.com/adeelahmad/lense/pull/408)).

## Planned

Expand Down
70 changes: 57 additions & 13 deletions docs/file-tree.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,13 @@
Lens shows its graph as an ordinary tree of folders and files: graph nodes (collections, topics, entities) are
folders, nested along the graph's links, and every asset (a recording, document or image) is a few files inside the
folders it belongs to, its metadata in a sidecar beside it. Agents and tools that already understand files can read
Lens without learning its API.
Lens without learning its API, and add to it the same way: a file copied into a collection's folder over S3 or WebDAV
is imported there ([Adding files](#adding-files)).

| Where | Status |
| --- | --- |
| The API (`GET /api/v1/tree`, `GET /api/v1/tree/file`) and the MCP tools `list_files` and `read_file` | built |
| An [S3 server](#s3) and a [WebDAV server](#webdav) (read-only, off until turned on) | built |
| An [S3 server](#s3) and a [WebDAV server](#webdav) (off until turned on); files added through them are imported | built |
| An [NFS share](file-shares.md) (read-only, off until turned on, allowed networks only) | built |
| An [SMB share](smb.md) (read-only, off until turned on) | built |

Expand Down Expand Up @@ -45,7 +46,8 @@ Names are made safe for every file system: `/ \ : * ? " < > |` and control chara

Exactly what the web app shows them ([Access](access.md)): an admin sees every namespace; others see the namespaces
they have a role in, and in a namespace where they only have roles on some collections, those collections and the
topics and entities of the assets in them (with the topics broader than those). Anything else answers "not found". The tree only reads.
topics and entities of the assets in them (with the topics broader than those). Anything else answers "not found". The API only
reads; files are added through the [S3](#s3) and [WebDAV](#webdav) servers.

## The API

Expand All @@ -59,7 +61,7 @@ The MCP server has `list_files` and `read_file` ([MCP server](mcp.md)): `read_fi
## S3

Off until an admin turns it on, in Settings, File shares (`shares.enabled`). Then Lens's worker serves the tree
read-only over S3 on port 9555 (`shares.s3_port`; with Docker, `LENS_S3_PORT` publishes it on another port of the
over S3 on port 9555 (`shares.s3_port`; with Docker, `LENS_S3_PORT` publishes it on another port of the
host), where the web app is: this machine only unless Lens was opened to the network (`LENS_BIND`;
`LENS_SHARE_BIND` sets it apart). Each namespace you can read is a bucket; keys are the paths above.

Expand All @@ -84,28 +86,66 @@ endpoint = http://lens.local:9555
force_path_style = true
```

then `rclone lsd lens:`, `rclone copy "lens:pods/Topics/Gene therapy" ./gene-therapy`, or `rclone mount lens: ~/lens`.
then `rclone lsd lens:`, `rclone copy "lens:pods/Topics/Gene therapy" ./gene-therapy`, `rclone copy ./talks
"lens:pods/Collections/Talks"` ([Adding files](#adding-files)), or `rclone mount lens: ~/lens`.
Supported: listing buckets, listing objects (v1 and v2, by folder or recursively, a page at a time), reading objects whole or
in ranges, and presigned links. Anything that writes answers AccessDenied. Requests must be signed with Signature
Version 4.
in ranges, presigned links, and adding files: PutObject (whole, streamed with `aws-chunked` and signed chunks, or with
a checksum), multipart uploads, a key ending in `/` for a folder, a copy of a file just added onto itself (to set its
time, as rclone does), and DeleteObject(s) for files kept aside. A file's `x-amz-meta-mtime` (rclone's) dates it.
Making or deleting buckets and deleting assets answer AccessDenied; copying other objects, NotImplemented. Requests must be signed with
Signature Version 4.

The server speaks plain HTTP. On a network you don't trust, keep it on this machine or behind TLS (a reverse proxy),
since files travel in clear.

## WebDAV

Off until an admin turns it on, with S3 (Settings, File shares; `shares.webdav`, on by default once shares are).
Lens's worker serves the same tree read-only over WebDAV on port 9556 (`shares.webdav_port`; with Docker,
Lens's worker serves the same tree over WebDAV on port 9556 (`shares.webdav_port`; with Docker,
`LENS_WEBDAV_PORT`), so it mounts as a drive: in Finder, Go, Connect to Server; in Windows Explorer, Map network
drive; in GNOME Files or Dolphin, `dav://lens.local:9556/`; or `rclone` with `type = webdav`.

Sign in with an access key: its id as the user name and its secret as the password. The folders at the top are your
namespaces. Listing (PROPFIND, one level at a time) and reading (GET, with ranges) work; anything that writes answers
403. Eight failed sign-ins from one address within 15 minutes are refused for a while.
namespaces. Listing (PROPFIND, one level at a time) and reading (GET, with ranges) work, and so does adding files
([Adding files](#adding-files)): PUT (with `Expect: 100-continue`, refused before the file is sent when it can't go
there), MKCOL for a folder, DELETE and MOVE for files kept aside, and PROPPATCH (answered, not kept), so dragging
files and folders in from Finder or Windows Explorer works. LOCK and UNLOCK answer as a class 2 server so Finder mounts
it writable, but they lock nothing. Deleting or moving assets answers 403. Eight failed sign-ins from one address within 15
minutes are refused for a while.

Like S3, it speaks plain HTTP, and Basic sign-in sends the secret with every request: keep it on this machine or
behind TLS on a network you don't trust.

## Adding files

A file copied into the tree over S3 or WebDAV is imported, as an upload in the web app is: the same checks of its type
and size (Settings, Uploads), the namespace's pipeline, and the same file twice is added once. Where it's put decides
where it goes:

| Put it in | It goes to |
| --- | --- |
| `<namespace>/Collections/<collection>/…` | that collection |
| `<namespace>/Collections/<new folder>/…` | a new collection, made with any missing ones above it (up to 8 deep), as **New collection** makes one |
| `<namespace>/`, `<namespace>/Collections/`, or a folder under `Topics/` or `Entities/` | the namespace's default collection; topics and entities come from the analysis |

Folder names match collections whatever their case. It must be yours to add to: the import permission in the
namespace, or an editor's or admin's role on the collection (making a collection takes what making one in the web app
takes). Otherwise the file is refused before it's sent where the client allows (S3's and WebDAV's `Expect:
100-continue`). Files Lens made (an asset's own file and its `.json`, `.txt` and `.pdf`) can't be written over. Each
file added is audited (`share.import`, with the protocol and path), and each collection made too.

After a file is imported, the asset shows as `<title> [<id>]` like every other; the name you gave it stays in its folder,
for you, for 10 minutes, so the copying app finds the file it just wrote and a re-sync sees it's there (with its own
time and MD5, so `rclone copy` and `rclone check` agree). Files an operating system or an app writes for itself
(`.DS_Store`, `._` files, `Thumbs.db`, `desktop.ini`, Office's lock files, `.part`, `.tmp` and other hidden or
half-written files) and empty files (Finder and Explorer make one before they write it) are kept aside for 10 minutes
after they last change, seen only by you, and never imported; renaming one to an ordinary name imports it then. They're
kept on the uploads disk, in memory per worker, so a restart forgets them.

Adding files is on whenever the shares are; turn it off in Settings, File shares (**Import files people add**,
`shares.add_files`) and they're read-only again. `GET /api/v1/shares/s3` says whether it's on (`add_files`). The
[NFS](file-shares.md) and [SMB](smb.md) shares stay read-only for now.

## At the web app's address

The web app can relay both shares at its own address: S3 at `/s3` and WebDAV at `/dav`. They then work wherever the web
Expand Down Expand Up @@ -135,12 +175,16 @@ How it works: the web app hands requests under `/s3` and `/dav` to the worker's
files, else this machine's ports 9555 and 9556). The path and `Host` stay as the client sent them, since S3 clients
sign both, and `X-Forwarded-Prefix` tells the share server the request came through the web app. Failed WebDAV
sign-ins are counted per visitor, from `X-Forwarded-For` as the web app and the proxies Lens trusts report it
(Settings, Access & embedding, Trusted proxies). Signing in is the same as at the ports: a person's own access key,
read-only.
(Settings, Access & embedding, Trusted proxies). Signing in is the same as at the ports: a person's own access key, and
files added that way are imported as at the ports ([Adding files](#adding-files)).

## Refine later

- TLS by itself for S3 and WebDAV, folders' real dates, and WebDAV locks for clients that insist on them.
- TLS by itself for S3 and WebDAV, folders' real dates, and WebDAV locks that lock.
- Delete, rename and move assets through the shares (only files just added can be today), and copy objects over S3.
- A read-only access key, for a person who may add files but wants a key that can't.
- Keep the names files were added under past 10 minutes and across restarts; a folder copy that hits a file type Lens
doesn't take stops in Finder.
- Virtual-host S3 addressing at the web app's address, for clients that can't take an endpoint with a path (MinIO's
`mc`).
- Show supplementary files (captions, translations, attachments) beside each asset.
Expand Down
2 changes: 2 additions & 0 deletions docs/requirements.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,8 @@ request, to another check that proves the same thing; loosening a requirement or
| OpenAI's chat completions API per namespace, cited from that namespace alone, off until turned on | [test_completions.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_completions.py) `test_off_until_an_owner_turns_it_on`, `test_answers_from_the_namespace_alone_with_citations`, `test_streams_as_openai_does` |
| API keys that open one namespace and nothing else | [test_namespace_keys.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_namespace_keys.py) `test_a_key_for_one_namespace_opens_its_servers_alone`, `test_it_has_the_persons_role_there_and_no_more` |
| The graph's folders and files mounted read-only over S3, WebDAV, NFS and SMB | [test_s3.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_s3.py) `test_people_read_their_namespaces_over_s3`, `test_what_s3_refuses`; [test_webdav.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_webdav.py) `test_mount_and_read`, `test_what_webdav_refuses`; [test_nfs_share.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_nfs_share.py) `test_the_runner_follows_the_setting`; [test_smb_share.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_smb_share.py) `test_reads_the_tree_as_its_key_sees_it`, `test_nothing_can_be_written` |
| S3 and WebDAV at the web app's address, through remote access, once an admin allows it | [test_s3.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_s3.py) `test_at_the_web_apps_address_once_an_admin_allows_it`; [test_webdav.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_webdav.py) `test_at_the_web_apps_address_once_an_admin_allows_it`, `test_through_the_web_app_each_visitor_has_its_own_throttle` |
| A file copied in over S3 or WebDAV is imported where it was put, with an upload's checks and permissions | [test_s3_writes.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_s3_writes.py) `test_a_file_put_into_a_collection_is_imported_there`, `test_folders_become_collections`, `test_multipart_uploads`, `test_what_may_not_be_added_is_refused`, `test_files_added_at_the_web_apps_address`; [test_webdav_writes.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_webdav_writes.py) `test_finder_copies_a_file`, `test_windows_copies_a_file`, `test_a_folder_dragged_in_becomes_collections`, `test_what_may_not_be_added_is_refused` |
| The web app runs on the same REST API, through a client generated from its OpenAPI schema | [ci.yml](https://github.com/adeelahmad/lense/blob/main/.github/workflows/ci.yml) `OpenAPI schema is up to date` |

## Ask, and get cited answers
Expand Down
Loading
Loading
0