8000 lense/DEVLOG.md at v0.17.0 · adeelahmad/lense · GitHub
[go: up one dir, main page]

Skip to content

Latest commit

 

History

History
729 lines (576 loc) · 53.1 KB

DEVLOG.md

File metadata and controls

729 lines (576 loc) · 53.1 KB

Dev log

Plans and progress for work in flight. Newest first.

2026-10-18 · Secrets from Vaultwarden

Adeel: a secrets manager, Vaultwarden first, with others to come. A secret field in Settings can hold a reference (vaultwarden://OpenAI/password) instead of the secret; secret_providers.py reads it when the settings are built, keeps it in memory for secrets.cache_minutes and reads it again in the background, and anything that shows settings shows the reference, never the value. The built-in sealed store stays the default. Vaultwarden is read with python-vaultwarden (Apache-2.0), fetched as a component once an address is set: the Bitwarden CLI is GPL and needs Node, and Bitwarden's SDK isn't permissively licensed and reads only its Secrets Manager product. Lens reads one folder of its own vault account. Tested against a fake Vaultwarden that encrypts with the library's own crypto.

Next: Settings › Secrets in the web app, with a picker for the vault's items beside every secret field.

Refine later: references in sources' credentials, MCP servers' environments and sign-in providers; checking a value read from the vault the way a typed one is checked; a custom CA for a self-hosted vault; an environment variable for the master password; more providers (1Password, HashiCorp Vault, Infisical).

2026-10-17 · Kokoro reading aloud on this machine

Adeel: Kokoro v1.0's model and voices for spoken answers and podcasts, so Lens reads aloud fully on its own machine. kokoro.py runs it through kokoro-onnx on ONNX Runtime, in-process (no service to start, unlike StyleTTS2), as voice.tts_provider kokoro; components.py fetches the kokoro extra and the pinned model and voices files once it's picked, and podcast jobs wait for them. The voice list is fixed (54 voices), so the picker works before the fetch. espeak-ng and phonemizer are GPL-3.0, so the extra is opt-in only (AGENTS.md 13). Measured here on a 4-core x64 container, one 5-second sentence: fp32 2.8 s, fp16 3.0 s, int8 4.6 s, so fp32 is the default; on GitHub's arm64 runner (kokoro.yml) fp32 read the same sentence in 3.2 s. kokoro-onnx's fp16 export isn't offered: on arm64 ONNX Runtime's CPU provider returned an empty waveform for it (kokoro-onnx 0.6.1's pause insertion then fails on a zero-size array), and on x64 it bought nothing over fp32; int8 is the small-disk option.

Refine later: misaki's G2P instead of espeak-ng for English (better names and numbers, and no GPL in the path); speaking speed (speed 0.5 to 2); voice blends (af_heart:60,am_michael:40); stream long answers sentence by sentence; pick the hosts' voices in the podcast dialog; Japanese and Mandarin through misaki's own G2P.

2026-10-17 · Read any chat message aloud

Adeel: play any message in the chat through the TTS model. A speaker button on every question and answer reads it by POST /voice/speak (so the provider layer, the voice picker's voice and the activity ledger are voice mode's), one message at a time, in pieces of at most 1,500 characters (Deepgram's limit is 2,000; speech.speak cuts at 4,000), with the next piece asked for while one plays. Audio is kept in the page by text, the 32 most recent pieces.

Refine later: keep audio across reloads (encrypted on the server, or the browser's Cache Storage); a shorter first piece so long answers start sooner; voice mode reading answers through the same pieces; pause and resume rather than stop; the speaker button in an asset's Chat tab; drop kept audio when the voice is changed.

2026-10-17 · Dates read as days

Adeel: dates extraction. dates.py reads the dates each line gives (written out, numeric, and relative to the day) into date_said as first and last days, worked out from the asset's recorded_at, when Settings → Analysis → Read dates is on. GET /api/v1/dates, MCP list_dates and the assistant's find_dates find them by day or range with their lines. English rules in Lens's own code rather than dateparser: dateparser couldn't read "next Tuesday" or "this Friday", and its own search over a line finds dates in ordinary words.

Refine later: DATE entities named by their day, so "next Tuesday" and "20 October" in two assets meet in the graph; other languages' month and day names; a Dates view in the web app (a calendar of what was said); times of day.

2026-10-17 · Entities found by spaCy

Adeel: NLP libraries for named entities. Settings → Analysis → Entity extraction → spaCy existed but needed spaCy installed by hand; it's now a component fetched the first time an asset needs it (the spacy extra and a pinned, checksummed model wheel: English, German or the multi-language one, all MIT). The defined entities come first, spaCy's names fill in around them, and the rules stand in until spaCy is there. Workflows' rules node uses the same extractor.

Refine later: pick the model by the transcript's language; spaCy's medium and large English models (MIT too, 40 to 560 MB) as options; run spaCy over a recording's lines in one batch.

2026-10-17 · Cleaning up transcripts before analysis

Adeel: a text cleaning step before entity extraction and transcript processing, from a punctuation and grammar script he pasted. The clean step (cleanup.py) runs before analyze in the standard pipeline, off until Settings → Analysis turns it on: rules take out fillers and stutters and tidy spacing and capitals, each change kept in the edit history with Revert. The script's alternating "Speaker 1 / Speaker 2" labels aren't carried over: Lens tells voices apart by diarization, or keeps the labels a transcript came with.

Next: punctuation restored by a local model (the script's deepmultilingualpunctuation, fetched on first use), grammar from a LanguageTool server, spaCy fetched for named entities, and dates found and normalised.

Refine later: revert a whole cleanup at once; cleanup rules for other languages' fillers; per-namespace switch.

Grammar: the clean step takes a LanguageTool server's GRAMMAR, PUNCTUATION and TYPOGRAPHY fixes, and EN_A_VS_AN (filed under MISC) (analysis.cleanup_grammar_url, the compose files' languagetool profile), never spelling, style or a word with a capital. language_tool_python from the script is GPL-3.0 and starts Java itself, so Lens talks to a server instead. Refine later: start the profile from Settings through the machine agent; per-namespace grammar; LanguageTool's other categories as opt-in switches.

2026-10-17 · Describing pictures with Moondream on this machine

Moondream is the third describe engine (llm.describe_engine moondream), opt-in, through Hugging Face transformers (Apache-2.0) and PyTorch (BSD), the moondream extra fetched on first use. Both models' code comes with their weights (trust_remote_code), pinned to a revision; both caption with the same call (caption(image, length="normal")). llm.moondream_model picks Moondream 3 Preview (the default when Moondream is chosen, since that's what was asked for) or Moondream 2. moondream.cannot_run keeps 19 GB from being fetched where Moondream 3 can't run (no NVIDIA GPU with 20 GB, no Mac with 32 GB) and says so in the step's log and in picture_descriptions.

Checked here: Moondream 2 at the pinned revision loads with the locked transformers and describes a photo well ("A vintage light blue Volkswagen Beetle is parked parallel to a yellow building with a wooden door …"), but took 3 to 6 minutes a picture on four shared CPU cores in float32: Florence-2 base took 3 s. Moondream 3 Preview wasn't run here (no GPU); its code at the pinned revision was read: the same caption call, with flex attention. The opt-in test (LENS_TEST_MOONDREAM=1) fetches PyTorch's CPU build, transformers and Moondream 2 through the component and describes a red dot. It first wrote "and, (, (, (…": with OMP_THREAD_LIMIT below PyTorch's thread count (the test suite sets it to 1), PyTorch's CPU kernels write nonsense, so _load now caps PyTorch's threads at that limit.

Refine later: Moondream's query skill with what the picture is ("page 3 of …"), as the provider's model is told; its compile() on a GPU; bfloat16 on CPUs that have it; Moondream 3 on a GPU in CI or on a maintainer's machine; a quantised Moondream 2 for small machines.

2026-10-17 · Describing pictures with Florence-2 on this machine

Adeel: use Moondream 3 and/or Microsoft Florence for visual descriptions. Florence-2 (MIT) is now a describe engine (llm.describe_engine florence-2): its 8-bit ONNX export runs on ONNX Runtime on the CPU, fetched on first use, pinned and hashed, base or large. It runs a Raspberry Pi's way (no PyTorch, about 280 MB for base) and needs no model provider. Checked on two photos here: base takes about 3 s a picture on four CPU cores, large about 9 s, and both say plainly what each shows.

Moondream 3 comes next as an opt-in engine. Its licences aren't open: Moondream 3 Preview's weights are BSL 1.1 (with a production use grant; Apache-2.0 two years after release) and Moondream 3.1's are the Moondream Model License; 3.1 runs only on Photon, whose kernels (kestrel-kernels) are proprietary and licensed only under a separate agreement, so Lens won't install Photon. Moondream 3 Preview runs through Hugging Face transformers (permissive) on an NVIDIA GPU or a Mac with enough memory (about 19 GB of weights): not on a Pi.

Refine later: beam search (the model's own default is three beams; greedy with no repeated trigrams matches it on the photos tried); Florence-2's OCR task beside the OCR step; a GPU (CUDA or CoreML execution provider) when there is one; the Q4 exports for a smaller download; describing only the pictures NSFW classification didn't flag.

2026-10-15 · Entity types beyond Term

Adeel: "every entity comes as term? why!!". The default rules extractor typed a capitalised name PERSON only after a title, ORG only before Inc/Corp-like words and PRODUCT with a digit; everything else (people, places, most organisations) was TERM. name_types.py now types names from built-in lists (given names, places, well-known organisations and products, holidays), the words a name starts or ends with, roles and "called" phrases before it, verbs of saying after it and the namespace's named speakers; a recording gives each name one type. Upgrade step entity-types retypes old analysis-made Terms, skipping types people or agents set (graph history and audit log).

Then the decision model: analysis.typing (auto by default: Jev or Laya when set up, never the language model; model lets the language model answer; rules none) asks about a recording's new Terms in one decide.choose_all call, with two lines each, and keeps answers above decisions.act_above.

Refine later: an action that asks the model about existing Terms (API, MCP, a button on the Entities page; next PR); offer the namespace's own types to the model too; ask about names in the hybrid mode's new entities; place, name and brand lists for more languages and regions (Custom vocabulary in Settings covers one install's own names); type existing entities again when a later recording gives better evidence (needs a marker for types people set, kept out of the versioned rows); "Google Cloud"-like names (a known company, then a word) as products.

2026-10-15 · The assistant knows Lens itself

Adeel: the assistant had no clue about the platform, its capabilities or general asks like "what is the MCP address"; ground it in the docs through an internal tool. lens_help (app/domain/manual.py) answers from two things: this server's facts (version, the web app, MCP and API addresses as the person uses them, what is turned on, their access; admins also get each setting's page) and Lens's docs, cut at their headings into an in-memory SQLite FTS5 index (the word index's engine), found by words, with the docs' contents. Both are cited ([n]: "This Lens server", "Docs · MCP server › Connecting", opening the heading on GitHub). The same tool is on the MCP server, and chat rooms list its sources. The Docker Compose files mount the install's docs/ read-only at /docs; Cloudron's image copies them.

Refine later: the docs inside the NAS packages' images (Synology, QNAP build from fastapi_backend/ only, so they have the server's facts and the online link); an in-app docs page so citations open the installed version instead of GitHub's main branch; docs found by meaning too when search by meaning is on; docs passages for models that can't call tools (search-and-answer only reads the archive); the README's overview in containers (its symlink points outside docs/).

Found while landing it (the clicks survey hung after its first chat answer, one run in six): a reader leaving a streamed answer before it ends leaves its generator to the garbage collector, which closes it on whatever thread is running, mid-query, and the answer's cleanup (on.end()) then asked the database for the one embedded connection that same thread held: a self-deadlock that froze the whole server. store.DB.conn now gives a thread the connection it already holds. Refine later: the cleanup of a streamed answer (and, in a long conversation, its summary's model call) still runs on the thread that happens to collect it, the event loop's included; hand it to a worker instead.

2026-10-14 · Long recordings in pieces a speech provider takes

Adeel: large audio files are segmented automatically to each transcription provider's limits. speech.Engine cuts a recording longer than speech.{provider}_max_minutes at the quietest tenth of a second in the half minute before the limit, halves a piece whose encoded file is over speech.{provider}_max_mb (again at a pause), sends each piece and joins the segments on the recording's timeline. Defaults follow each provider's published limits (docs/speech-providers.md, Long recordings); they are in Settings → Speech providers. This replaces the ten-minute parts only OpenAI-compatible servers had.

Refine later: link a new voice heard in two pieces into one speaker by voiceprint (today it shows as two until merged); decode and send a very long recording a piece at a time instead of holding all its samples in memory; send pieces in parallel for providers that allow it; overlap pieces a little and drop the words heard twice, for a cut with no pause near it.

2026-10-14 · MCP servers that run as a program

Admins add a stdio MCP server (command, args, env, as in Claude's .mcp.json) as an mcp extension, and a plugin's stdio servers come along when an admin installs it. They run in the MCP runner (mcp-runner/, the opt-in mcp Compose profile), a container with none of Lens's data or keys, reached over HTTP with a token on a volume only it and the backend share. Recorded in the threat model.

Refine later: turn the runner on from Settings through the machine agent (done, the services switch); the runner in a microsandbox (done 2026-10-18: a process as the mcp user inside the sandbox, microsandbox/mcp.sh); the runner on Cloudron and NAS packages; in a microsandbox, keeping the servers' packages across an update (they're on the sandbox's disk, fetched again after it's replaced) and limits on the runner's memory and processes (the container has them; a cgroup for the process would); plugin servers that run a program from the plugin's own folder; an egress allow-list for the runner's network.

2026-10-10 · Pasted transcripts fetch no speech model

Found by the clicks survey: on a fresh install, importing a pasted transcript queued the standard pipeline from transcribe, so the worker fetched PyTorch and SenseVoice (about 1 GB) before analysing it, only for transcribe to skip. jobs.for_source now leaves transcribe and diarize out for a transcript source, as it does for documents and images.

Refine later: transcript jobs queued before the upgrade still wait for that fetch once (a migration step would back up the whole database for a queue that drains on its own); a run of only transcribe or diarize on a transcript still queues them, to say why they skip, and may fetch the model; a batch of only diarize over documents fails its enqueue.

2026-10-10 · The assistant in your browser

Goal (Adeel): chat with the assistant from the browser extension, and let it do things by using the browser, like the Claude browser extension: "check my email and import the important attachments".

Model (docs/browser-assistant.md): the clipper extension (extensions/browser) gets a side panel. It signs in with OAuth (PKCE, identity.launchWebAuthFlow, nothing to copy), chats through /chats like the web app, and connects a browser session (fastapi_backend/app/domain/browser.py). While it's connected, a question from it gets the browser_ tools; each action is a browser_action row the side panel takes from its event stream, carries out and answers, and the tool waits for it in the database (any process, like stopping an answer). The extension guards what the browser does: sites the person allowed, a yes before anything consequential, no passwords or card numbers, an action log. Page text reaches the model as tool results (third-party text, untrusted.py once #302 is in).

PRs:

  • Server: sessions, actions, the browser tools, ai.browser and ai.browser_steps, the API for agents
  • Extension: side panel chat, OAuth sign-in (the Edge smoke test signs in through a stand-in consent page)
  • Extension: browser control (read with refs, click, type, keys, download into the conversation), site permissions, asking first, the action log (the Edge smoke test drives each against a stand-in Lens)
  • After reading outside text (#309's wary()): addresses to open or download and text to type that the person didn't write go with confirm, and the side panel asks first, on allowed sites too

Refine later: the browser from WhatsApp/Slack while the side panel is open (a setting, off); keep-alives in the chat stream while an action waits; browser tools over MCP; real input events (Chrome's debugger) for sites that ignore synthetic ones; asking about the page or selection from the right-click menu; the model and namespaces picker in the side panel; webmail attachments behind a download button, and files redirected to another site; frames; Safari's default tab; Lens checking that the extension asked (an extension older than confirm doesn't).

2026-10-10 · Similar content

Goal (Adeel): a Similar tab on every resource, and selecting text anywhere (a resource's text or title, notes, topics, entities, the graph canvas) shows similar content in a popover, with a way to find more.

Model (fastapi_backend/app/domain/similar.py, docs/processing.md#similar-content): one list for a text, a resource, a topic or an entity, fused by reciprocal rank from four: meaning (passages near the text, or near the centre of a resource's passages: no model call for a resource), key words (the text's words the term table says are rare, or analyze.keywords, OR'ed through the word index), shared topics and shared entities (rarer ones weigh more). Each item cites up to three moments. GET /similar, assistant and MCP find_similar. Without embeddings (the default) the other three answer, with no model call.

PRs: #296 the API, assistant and MCP tool; then the web app (the Similar tab, the selection popover and the full Similar page at /similar, and similar content for topics, entities and graph nodes).

Refine later:

  • key words of a text are single words and two-word phrases; stems beyond plurals (designing/design) don't match;
  • a resource not indexed for meaning is compared by its title and keywords (one embedding request), not its text;
  • the centre of a long resource's passages blurs what its parts are about; comparing a few of its passages may be sharper;
  • entities in another namespace with the same name count only through the words, not as the same entity;
  • no "not similar" feedback yet, and nothing is cached between requests beyond the query vectors.
  • selecting text in an editor (notes, the document editor) opens nothing, since selecting there is for editing; a "Similar" item in the editor's own menu would cover it;

2026-10-10 · More voices for spoken answers

Goal (Adeel): read answers and podcasts aloud with ElevenLabs, macOS say, Google Gemini TTS or StyleTTS2, with the voice picked in Settings. ElevenLabs and Deepgram were already there (#116); the providers stay shared with podcasts through speech.speak().

Todo:

  • Gemini TTS and macOS say as text-to-speech providers; GET /voice/voices lists each provider's voices; POST /voice/speak takes a voice to try
  • StyleTTS2 as an opt-in local service (styletts2/, Compose profile styletts2; voices are clips in a volume)
  • Settings → AI assistant → Voice: pick a voice from the list and hear it; Gemini in Speech providers

Refine later: pick a second say voice for podcast hosts from what the Mac has; Gemini's multi-speaker mode for a whole podcast in one call; ElevenLabs voice settings (stability, style); cache the voice lists; upload StyleTTS2 voice clips from Settings (or use a speaker's own voiceprint clips); start the StyleTTS2 service from Settings; GPU build.

2026-10-10 · Podcasts from picked knowledge

Goal (Adeel): pick documents, search results, topics, notes or graph concepts and get a two-host, Recast-style learning episode about them, every claim cited to a real source, connected to the rest of the library.

Model (fastapi_backend/app/domain/podcasts.py, docs/podcasts.md): an episode is a recording (its script lines are segments spoken by the hosts) plus podcast:<rid> for the request, excerpts, outline, cited lines and fact-check log. One job on that recording runs podcast steps (gather, plan, write, check, publish) and then analyze, embed, summarize. Citations are segment runs {recording, idx0, idx1}, which exist without embeddings. Prompts are prompt templates with role podcast..

Rulings (Adeel, 2026-10-10, recommended defaults accepted): episodes go in a podcasts namespace only when its readers can read every source namespace and no source is in a vault, else in the sources' namespace; OpenAI-compatible TTS by default, ElevenLabs opt-in, script-only without TTS; 5/10/20 min presets, 30 max; rewrite an unsupported line once, then drop; episodes are never cited as evidence by later episodes (memory only); MCP may queue an episode; on by default; stale citations are marked, never auto-regenerated.

Todo:

  • Backend script pipeline: selection (recordings, passages), plan, write, fact-check, publish as a transcript
  • Audio: per-line TTS with two voices, stitch with pauses, loudness, store as the recording's media
  • REST API and MCP create_podcast / get_podcast
  • Web: Create podcast dialog (Library selection, resource menu), /podcasts list, episode page with progress, player, citations, sources and fact-check log
  • Phase 2: cross-page selection tray (search hits, topics, notes, graph), context expansion, connections
  • Phase 3: episode memory, script editing and re-render, episode→source graph edges, stale citations

Watch out: an episode's speakers are named after the hosts, so they are the same speakers across episodes in a namespace.

2026-10-05 · Threat model

Goal (Adeel): write down Lens's security contract, meaning what it assumes, guarantees and leaves to whoever runs it, so a security report can be routed against something written. Modeled at 5f799f2; produced with the threat-model plugin (1.2.1).

Rulings from Adeel (2026-10), recorded in the model as maintainer claims:

  • Exposure to the internet behind HTTPS is supported. Only the instance admin is trusted; an admin is equivalent to code on the host. Owners, editors, viewers, tokens, MCP agents and the assistant acting for them are not.
  • URLs only an admin can set (model server, sources, IIIF import, Fedora, telemetry) are trusted input. SSRF is in scope for URLs a non-admin sets (web import, extension http tools, notification targets), and for the guards Lens documents on admin URLs (iCal, the telemetry endpoint).
  • Shipped defaults (plain HTTP, SurrealDB root/root) are supported on one machine only. docker-compose.yml and make dev are dev-only.
  • Downloaded binaries and models are trusted to HTTPS and their publisher, with no further integrity claim.

Todo:

  • Recon, surface pass over 4 families, waves 1–2 of questions
  • Draft threat-model.md; backtest against 36 real security commits plus 81 synthesized reports (no fixed item closes); revise
  • threat-model.yaml and threat-model.json; link from SECURITY.md
  • Waves 3–11 of §1.18 (47 open questions), then promote answers to maintainer claims
  • Align SECURITY.md's SSRF example ("a model server") with the admin-URL ruling (YAML hash refreshed; the plugin's validator was not available in the session that made this edit, so re-run it)

Watch out: the prose is canonical. After any edit to threat-model.md, regenerate the YAML and JSON and run ci/validate_threat_model.py --base origin/main (CI runs it on every change to these files and fails a prose change that changes neither companion). The YAML no longer holds a hash of the prose, which made every pair of threat-model pull requests conflict (#507).

2026-10-04 · Graph history: versions, replay, rollback

Goal (Adeel): make the graph replayable and versioned.

Model (fastapi_backend/app/domain/graph_history.py, docs/graph-history.md):

  • Every change to the curated entity graph is one append-only graph_event: its number is the graph's version. It says when, who (a person, a routine, the assistant, analysis), through what (web app, API token, OAuth app, assistant, MCP, routine, analysis), why, and for every record it touched (entity, alias, cross-namespace link, "not the same" pair) the row before and after.
  • Built on what is there: graph changes (organize.py: proposed, applied, dismissed, undone) and merges (entity_merge) are recorded through the same entity functions, and their events point back at them.
  • As of: the graph at any version, rebuilt by walking back from today's rows. Diff between two versions. An entity's history. Versions can be named ("before the cleanup").
  • Rollback to a version: a preview first, then later changes are undone as new events, so a rollback can itself be undone. Merges are undone with their mentions.
  • Replay: checkpoints (the graph at a version) plus the events after them rebuild the graph; verify compares the replay with the live rows and records any difference as an event, so nothing stays unrecorded.
  • Not versioned here: mentions (analysis makes them from transcripts, whose edits have their own history). As-of views use today's mention counts.

Todo:

  • Event log with who, through what and why; every curation path records (entity page, assistant, MCP, routines and graph changes, analysis, the entity map)
  • Versions API: list, the graph as of a version, diff, an entity's history, named versions
  • Rollback to a version, with a preview
  • Checkpoints, replay and verify (lens graph verify, nightly checkpoint)
  • The graph API and Cypher as of a version (as_of); the graph cache is keyed on the graph's version
  • An as-of picker in the explorer (with the graph explorer thread)
  • Web app: a history timeline with diffs and rollback (Routines › History; an entity's History button)
  • Graph events in each resource's activity history (an entity's, a namespace's, a recording's analysis)

Refine later: speakers (same-person links, names) in the history; a mention-level history.

2026-10-04 · Activity history and budgets