Repository navigation
Recall by meaning — a text-derived vector layer over tags and recognized content #32
Description
Problem
Recall comes in two shapes. "The one from earlier today" is a time
question, and the grid already answers it. "That one with the …" is a
content question, and the store answers it only when the words in the
query happen to match words that got attached to the asset. Full-text
search (asset_fts) and channel tags are exact instruments: they find
what was written down, and nothing else.
The pain at volume: you remember an image by what it looks like, not by
its checkpoint name or the keywords the auto-tagger chose. When the
remembered phrase never made it into a tag or a prompt, the current
answer is scrolling thumbnails — workable at hundreds, not at tens of
thousands. Picked up from #13's Open section (tag-based
vectorisation).
Approach to explore
Vectorise text derived from the asset — its tags, captions, and
recognized content — rather than embedding raw pixels.
- The
auto_tagjob already extracts keywords into channel tags; this
layer extends that pipeline rather than adding a parallel one. - The embedded text is inspectable. When a query misses, the text is
the diagnosis: either the recognizer never wrote the concept down,
or the embedding failed to bridge the words. A pixel embedding
offers no such seam. - Vectors propose, metadata disposes. Semantic candidates are ranked
suggestions; structured metadata and FTS remain the instruments
that filter what is actually usable. The vector layer must not
replace them, and search must degrade cleanly to FTS when the index
is absent.
Working constraint: the index should live comfortably beside the
SQLite store on a personal machine — the working hypothesis is that a
text-derived index stays small (target: on the order of 1 GB at
expected scale). If it cannot, that is a finding, not a tuning
problem.
The blocking decision is model choice. Changing the embedding
model invalidates every stored vector and forces a full re-index, so
the model (and its local-inference cost) has to be chosen before
indexing at scale, not after.
Desired property
- A query in natural words returns ranked candidates whose derivation
is inspectable (the text that matched is showable). - Fully local: indexing and query embedding run offline.
- Incremental: new and re-tagged assets enter the index through the
existing job pipeline, not through periodic full rebuilds. - Honest failure: a query that finds nothing says so, instead of
padding with weak matches.
Verification
Write the recall queries first. Before any index exists, collect ~20
real content-recall queries from actual use ("the one with …"), each
with its known intended answer in the store. Build the index, run
them, count the misses. That count is the go/no-go for the approach,
and the query set becomes the regression suite afterward — a model or
pipeline change reruns the same 20 and reports the same number.
Open
- Embedding model choice under the local-inference constraint — the
decision everything downstream waits on. - What "recognized content" includes in v1: captions only, or OCR for
text-bearing assets as well. Each recognizer widens recall and adds
a runtime dependency. - Storage: inside SQLite (extension) or a sidecar index file beside
the profile. Backup and the profile marker should keep working
either way. - Where query understanding stops: v1 embeds the query text as-is;
anything smarter (query rewriting, filters parsed from prose) needs
its own justification. - Video and audio: whether their text derivation (transcripts, frame
captions) is v1 or a follow-up.
Not in scope
- Raw-pixel visual similarity. It answers a different question
("looks like this image"), needs a reference image in hand, and
carries a different index cost. Nothing here precludes adding it
later as its own layer. - Hosted inference as a requirement. A hosted model may be an option;
the local path is the contract. - Replacing FTS, tags, or any structured search surface.
Activity
ynishi commentedon Aug 15, 2026
Investigation: the gap is wiring, not embeddings
Before choosing a model, I read what the store already does. The finding
changes the shape of this issue: every surface that could hold words
about an asset already exists, and none of them reaches the search
index.
What is already built (measured)
The lexical side is further along than the issue body assumes.
mixed_bodytokenizer: Lindera (IPADIC,Normal) morphological segmentation for Japanese → LowerCaser →
AsciiFoldingFilter → Porter stemmer → 40-char cap
(
crates/asterism-infra/src/search/tokenizer.rs:32-48).TermQuery OR FuzzyTermQuery(distance=1), gated at 4+ characters(
crates/asterism-infra/src/search/tantivy_index.rs:61-67,204).trigramtokenizer for the exhaustive substringpredicate; 3+ characters seek the index, 1-2 fall back to
LIKEoverasset_body(migrations.rs:3402-3422,3452-3457).(
migrations.rs:3396-3400).So "add fuzzy matching", "add trigram", "segment Japanese", "rank by
BM25" are all done. That is not where the recall is lost.
Where the recall is actually lost
There are five places an asset can carry words. Not one of them is
indexed.
material.meta_kv— PNGtEXt, i.e. the generation promptmaterial_hashjob via the PNG probematerial.meta_raw— raw bytes of all five metadata chunksImage.alt/Image.labelson the importer footprintasset_declare_metaover MCP)asset_commentOnly
asset_bodyreaches the index, andindex_rebuildreturns"is not text, skipped"the momentTextLocator::newdeclines anon-text payload (
crates/asterism-infra/src/jobs/handlers.rs:2687-2689).Images therefore enter neither
asset_ftsnor Tantivy at all.auto_tagdoes not close this either. It reads labels, the registernote, the filename split on
-_., and the first three#headinglines of a text file — capped at 8 keywords
(
handlers.rs:269-329). It never looks at what the image contains,so the issue body's premise that it "already extracts keywords" holds
only for text assets.
Two consequences worth stating separately:
unreadable to search this whole time.
meta_rawwas addedspecifically so a later build could decide how to expand metadata
"without opening the file"
(
crates/asterism-infra/src/probes/png.rs:96-111). That is exactlythe affordance this issue needs, already paid for.
meta_kvis lossy in two known ways:tEXtis Latin-1 by spec andgoes through
from_utf8_lossy, andzTXt/iTXtare read byneither axis — a stated gap (
migrations.rs:5128-5132). This is notacademic: ComfyUI workflow JSON is large enough to be compressed
into
zTXt, and prompts containing Japanese neediTXtto beUTF-8. The bytes survive in
meta_raw; nothing reads them.What the external research says
Two independent literature/implementation reviews were run. They
converged.
Document expansion beats query expansion. Enriching the document
with generated terms moved BM25 on MS MARCO from MRR@10 0.184 → 0.277
and Recall@1000 0.853 → 0.947
(docTTTTTquery).
The paper's own conclusion is that expanding documents is more
effective than expanding queries. Translated here: put the words in at
ingest, do not try to be clever at query time.
The vocabulary problem is the actual adversary. Furnas et al. 1987
measured that two people pick the same term for the same object with
probability below 0.20, so single-term access fails 80–90% of the time;
the prescription is "unlimited aliasing"
(CACM 30(11)). Tags
written by whoever generated the asset are exactly the single-term case.
Lexical degradation is a cliff, not a slope. Stratified by
query/document term overlap, BM25 holds Hits@10 ≈ 0.94 in the 0.8–1.0
band and collapses toward 0 as overlap thins, while an embedding model
holds (arXiv:2201.00118). What
lexical cannot reach is the phrasing that is not derivable from any
word present — mood, abstraction, relation ("that lonely-looking one").
Vectors, in this product category, are what you reach for when there
are no words. Of the comparable OSS tools, only Immich ships vector
search (CLIP, 512-dim), and its stated motivation is finding untagged
photos. PhotoPrism discretises via a classifier into labels, digiKam
uses perceptual fingerprints, Hydrus uses tag set intersection with
sibling/parent relations — none of them embed. That is the same
conclusion from the other direction: the embedding layer earns its
place where the words are absent.
Nobody's numbers can settle this for us. There is no public
benchmark for tag-shaped retrieval; BEIR and MTEB measure sentences and
passages, and BEIR itself reverses per domain (DPR at roughly a quarter
of BM25 on BioASQ; ANCE above BM25 on Quora). The verification section
of this issue — collect real recall queries first, count the misses —
is the only thing that can decide the question. Both reviews arrived at
that independently.
Two smaller findings worth keeping:
FUZZY_MIN_LEN = 4counts characters, and Lindera's Japanesemorphemes are mostly 1–3, so fuzzy never fires on Japanese. This
is correct and should not be "fixed": Meilisearch and Typesense both
refuse typo tolerance below a similar length, because edit distance 1
over short CJK tokens changes the word. The right instrument for 6948
Japanese is a reading (yomi) field, not a smaller threshold.
spellfix1is not a candidate. Its matching is ASCIItransliteration plus a phonetic hash, and
spellfix1_scriptcodeunderstands Latin / Cyrillic / Greek only.
Plan
Ordered so that each step is independently useful and the expensive
decision comes last, with numbers in hand.
meta_raw/meta_kv, recoverzTXtandiTXt(currently astated gap) and the Latin-1 bytes
from_utf8_lossydropped, andfeed the result into the existing indexing path. The narrow change
is
index_rebuild's gate: from "are these bytes text" to "is theretext derivable from this asset". Rows ingested before
meta_rawexisted carry the not-captured marker and need one re-read. No new
dependency, no model, no new index — the existing BM25 + Lindera +
Porter + fuzzy stack starts answering for images.
Latin only, so full/half width, katakana/hiragana, long-vowel
variants (
サーバ/サーバー) and inflected forms pass throughunfolded. Adding NFKC + width folding + base-form normalisation, and
a reading field, is a few filters in the chain. These are
deterministic wins: queries that returned nothing start returning
the right row.
alias and implication graphs, which fits this corpus without hand-
authoring a thesaurus. Apply at query time, not index time, and
place it before the stemmer. Do not generate a thesaurus from
co-occurrence — synonyms do not co-occur, and the classical result
is that it does not help (Peat & Willett 1991).
with …" queries with their known answers, run against the store
after 1–3. The miss count is the go/no-go, and the set becomes the
regression suite.
residual misses cluster on mood/abstraction/relation, that is the
signal an embedding layer addresses something lexical cannot. If
they cluster on missing words, the answer is upstream instead.
Sizing is not the constraint either way: 50k × 512-dim float32 is
~102 MB, an order of magnitude inside the 1 GB working budget.
Steps 1 and 2 are independent of each other and of everything above
them.
Deliberately not doing
Batch captioning through an external vision API, i.e. tiling N
thumbnails into one image and asking a cheap model to name what is in
them. It was considered and dropped. It adds a moving part to the
pipeline, and it produces a weaker version of something the operator
already has: an agent driving an import can call its own model and
write the result through the existing surfaces —
Image.alt,Image.labels, orasset_declare_metaafter the fact. Owning adegraded copy of a capability that is available everywhere is not worth
the construction.
That decision has a prerequisite this investigation confirmed: those
write surfaces do exist.
Image.alt("alt text / caption, becomes thecover hint") and
Image.labels("additional free-form labels") are onthe importer footprint,
asset_declare_metarecords an after-the-factstatement and attributes it to the agent that made it, and there is a
sidecar path (
<name>.meta.json) for metadata written beside the file.So the ingest side is open for an agent to write into.
What is not open is the same thing step 1 fixes: none of those
surfaces is indexed. Whatever an agent writes into
alt,labels,AlbumMeta or a comment today is as invisible to search as the prompt
is. Step 1 is therefore also what makes agent-written descriptions
worth writing — and it may be worth widening its scope to cover all
five surfaces rather than the PNG chunks alone.
Raw-pixel similarity remains out of scope as the issue states.