Repository navigation
perf(search): keep word search in SQLite, with OpenSearch as an option - #166
Merged
Merged
Conversation
SurrealDB's full-text index cost about six times the records' memory (10,000 transcript lines: 380 MB of server memory with it, 64 MB with a SQLite FTS5 file instead). search.engine "sqlite" is the default; "surrealdb" keeps today's index. Events note changed rows in text_change and the index catches up before each search. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QtzGWLtX99AFMsL4hYpoRe
search.engine "opensearch" keeps the words in an OpenSearch (or Elasticsearch-compatible) cluster at search.opensearch_url, kept in step by the same text_change events as the SQLite index. When it can't be reached, searches read the tables instead. A compose profile runs one next to Lens. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QtzGWLtX99AFMsL4hYpoRe
This was referenced Oct 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Requested by Adeel · project thread
Description
Before: word search used SurrealDB's own full-text index (BM25 + highlights), which cost about six times the records' memory. On 10,000 transcript lines with production
surrealkvstorage, the SurrealDB server used 380 MB, against 47 MB without the index.After:
search.enginepicks where the words are indexed.sqlite, the default, uses a SQLite FTS5 file per database in<data_dir>/search/. On the same 10,000 lines the server used 64 MB, and the index file was 4 MB.surrealdbkeeps today's index.opensearchuses a cluster atsearch.opensearch_url. There's anopensearchcompose profile.Search, the assistant's retrieval and IIIF content search return the same results. An existing install builds the new index on first start and frees SurrealDB's index.
How:
app/domain/textindex.pyholds the engines.DEFINE EVENTon each text table records created, deleted or re-texted rows intext_change. The index catches up from there before each search, so writers change nothing.text_change_state:indexdetects this.lens reindex(or Settings › Search › Reindex now) applies a change of engine or stemming.Motivation and Context
Adeel found the database was the weight, and approved this: "yes but also allow opensearch". Plan:
/mnt/project-files/lens-lite/plan.md.Types of changes
Checklist:
tests/domain/test_search_index.pycover both engines across a reopen, edits and deletes, switching engines, a swapped database, filters, OpenSearch (withLENS_TEST_OPENSEARCH_URL), an unreachable OpenSearch, and settings validation.docs/database.md,archive.example.yaml) and CHANGELOG updated.🤖 Generated with Claude Code
https://claude.ai/code/session_01QtzGWLtX99AFMsL4hYpoRe
Generated by Claude Code