A self-hosted dataset addon for Den. It holds the shared feature store — derived labels (genre / subgenre / mood) + quantized semantic vectors for the whole catalog — and answers queries against it: similar titles, category and facet rows, search, the billboard.
The store stays here. The app used to download it (labels-*.json, vectors-*.bin, a metadata sidecar,
a premise index and a facet blob), verify each blob's sha256 and run the nearest-neighbour search
on-device; since #113 it asks instead, and atlas serves no files at all — only JSON answers.
Den (Apple TV) ──GET /manifest.json──► atlas { resources: ["dataset"] }
──GET /dataset.json───► { datasetVersion, taxonomyVersion, embeddingModel,
dims, count, quantization, embed, queries }
──GET /index/similar/…─► { ids } ← ranked here, off the mmap'd store
──GET /index/query.json?q=─► { parse, hits }
It implements the Den dataset resource (the Stremio superset — see the Den repo's
tickets/EPIC-feature-provider-addon.md, FP-1). A plain Stremio client ignores the unknown resource, so
installing Atlas there is harmless; only Den acts on it.
Atlas answers from derived data — labels, vectors, and the titles / poster paths / years the store
carries so a client draws a card without a per-result TMDB call. No overviews and no raw text, and
nothing personal. There is no per-user state and no token; /configure only picks the catalog region
and services, carried in plaintext in the install URL. Personalisation (your taste vector) never leaves
your device.
No file leaves atlas, so nothing it serves is a verbatim copy of an upstream artifact: every TMDB-derived
value it emits is assembled by a route and audited for prohibited prose at the last HTTP boundary
(src/tos.rs).
Verified live 2026-09-05 by fetching /dataset.json from the deployed addon:
| field | value |
|---|---|
datasetVersion |
ebe9ab936444 (a content hash, not a semver — it moves when content moves, and that is what triggers client re-sync) |
taxonomyVersion |
t02 |
embeddingModel / dims |
bge-m3 / 1024 |
count |
37,533 |
quantization |
int8-symmetric-x127 |
This matches den-dataset/out-t02 exactly, so that directory is what is being served.
Three things about the data that keep being got wrong — see den-dataset/README.md for the measurements:
- 37,533 is not the catalogue. den-dataset enriches 57,715 titles; the 20,182 without a Wikipedia plot are dropped by a ToS rule, and they have no labels and no vectors — the two cover the identical ids. Consumers must degrade for 35% of the catalogue, not treat it as an edge case.
- There are two vector spaces. The plot index (
vectors-bge-m3.bin) and the premise index (vectors-premise.bin, Sonnet-generated premise tags embedded with the same model) are complements, not duplicates — measured mean |cos| 0.43. The app's "More Like This" leads with premise. The dataset publishes the plot vectors with their plot-length direction already projected out, so short plots stop pulling short plots (plotVectorTransformin the manifest; oxyc/den-dataset#109, #129). More Like This,/index/neighboursand search all read them as published; only a raw outside query vector entering the plot space is projected the same way. - A bare TMDB id is ambiguous: movie and TV namespaces overlap (tv 95 is Buffy, movie 95 is
Armageddon). Key every per-title map on
(mediaType, id).
(This section previously described labels-t01.json + 384-dim vectors-e02.bin and called bge-m3 a
future upgrade. That was two taxonomy generations out of date.)
datasetVersion/embeddingModel/taxonomyVersion still say when the dataset moved — a client caching
answers keys on them, and a change of embeddingModel invalidates every query vector with it.
Alongside the dataset, Atlas serves Stremio catalog rows of "most popular" titles per streaming
service — Popular on Netflix, Max, Prime Video, Disney+, Apple TV+ — plus a headline
Trending Everywhere row that unions the services and re-ranks by inverse-rank-sum (a title trending
on several services floats up). Data is the unofficial JustWatch GraphQL API (public, tokenless),
fetched server-side and cached in-process (~6h, serve-stale-on-error). Each meta carries the IMDb id
(a plain Stremio client + Cinemeta resolve the detail page from it) plus JustWatch's TMDB id as
moviedb_id — the key the Den app maps rows through (it bridges everything via TMDB). No TMDB API calls,
no meta resource. The module is fully isolated: if JustWatch breaks, catalog rows go empty and the
dataset resource is unaffected; with no outbound internet the rows are simply empty. Tunables:
JW_COUNTRY, JW_PROVIDERS, JW_CACHE_TTL_SECS. Catalog data from JustWatch.
Each manifest catalog keeps Stremio's required canonical-English name and also publishes an additive
heading: {id, args, fallback}. The id is locale-neutral, args such as a service name are verbatim,
and fallback is exactly the legacy name, so older and non-Den clients remain compatible.
A small Rust (axum + tokio) server — a ~0.8 MB static musl binary, ~2–4 MB RSS whatever the dataset
size. The store is mmap'd, never read into RAM; its header hash is verified once at load, so the row
count served as count is the store's own rather than a number the manifest claimed. (The original
TypeScript server is preserved at the legacy-ts git tag.)
Every response is cache-friendly (src/http.rs): a strong ETag (the body's 64-bit FNV-1a plus its
length) honoring If-None-Match (→ 304), plus HEAD. Range-capable representations advertise and
honor byte ranges (Accept-Ranges / 206 / 416); JSON answers are whole-only, do not advertise range
support, and ignore Range, since the prose guard can only judge a whole body.
dataset.json carries no Last-Modified: its body moves with the embed/index flags under an unchanged
dataset date, so only its ETag can say it changed. Nothing varies on a request header. Sit a CDN in front
and it caches everything by URL with correct revalidation.
| Response | Cache-Control |
|---|---|
manifest.json |
max-age=3600, stale-while-revalidate=600, stale-if-error=86400 |
dataset.json |
max-age=300, stale-while-revalidate=3600, stale-if-error=86400 |
/index/… GET, title search |
max-age=300, s-maxage=2592000, stale-while-revalidate=86400 (answers move only with the dataset, and a deploy purges them); max-age=300 when a search should have been ranked through den-embed and wasn't; max-age=300, s-maxage=300, stale-while-revalidate=300 when runtime maturity providers shaped it |
| a JustWatch row | max-age=3600, stale-while-revalidate=86400, stale-if-error=86400; max-age=60 for a stale or empty fallback |
A public answer with an s-maxage also carries CDN-Cache-Control: max-age=<s-maxage>[, stale-while-revalidate=…], stale-if-error=604800, so Cloudflare (which reads it in place of Cache-Control) may hand out its copy for a week
while atlas cannot answer; s-maxage itself forbids that. A runtime-provider answer gets none.
A search query's vector is remembered in memory (1,000 texts, least recently used first, 24 h), keyed by the dataset's embedding model and width, so a repeated search does not call den-embed again.
| Route | Returns |
|---|---|
GET /, GET /configure |
landing/configure page: pick region + services, get the install URL |
GET /health |
always 200: {"status":"ok"}, or {"status":"degraded","reason":…,"detail":…} with reason dataset_unavailable, embed_unavailable (the configured local engine failed identity validation or its real inference canary), stale_catalog (last JustWatch refresh failed), catalog_schema_suspect (a served chart came back mostly empty) or facts_unusable (the dataset declares a facts file the last index load couldn't read; /recommend and search run without facts) |
GET /manifest.json |
the dataset + catalog manifest (also under a /<region>_<codes>/ install prefix) |
GET /dataset.json |
the descriptor — what the dataset IS (datasetVersion, taxonomyVersion, embeddingModel, dims, count, quantization, signature, the embed/queries capability flags). It names no files to download; 503 when the dataset did not load |
GET /catalog/<type>/<id>[/<extra>].json |
a "most popular" row of {id,type,name,poster} metas |
GET /catalog/<movie|series>/den-titles/search=<q>.json |
with TITLE_SEARCH on: fuzzy, typo-tolerant title search, {id:"tmdb:<id>",type,name,moviedb_id} metas, best 30 |
GET /index/taxonomy.json |
with INDEX_QUERIES on: {schema,taxonomyVersion,subgenres,moods}, each list most-populated first. Label names only, kept for the TV app's browse rows; schema points at /index/schema.json, which describes the dataset |
GET /index/schema.json |
with INDEX_QUERIES on: the self-describing query fields, types, units and formats, value counts and per-field coverage; every count names both its known-field and full-corpus denominators (a series-only field, broadcaster, is out of the series). semantics says what each kind of count is out of and how a search applies a constraint (filter, discount, boost, require); routes lists every public route with its parameters, the field each reads, defaults, limits and a working example, so a client can use the API without reading the code |
GET /index/rows/<movie|series>/<subgenre|mood>/<label>.json?skip=&limit= |
with INDEX_QUERIES on: {ids,total,coverage} carrying the label, most confident first (≥ 0.55), 24 a page, at most 100; coverage names the full corpus, selected-type denominator and known-field population |
GET /index/similar/<movie|series>/<tmdbId>.json |
with INDEX_QUERIES on: {ids,total} for More Like This — premise neighbours gated by animation, genre and plot agreement, else plot neighbours — and beside them {mixed:[{type,id}],mixedTotal}, the same row with films and series together (skip/limit page both). Members of the seed's curated primary franchise are reserved for /index/franchise, and its other versions (with their franchises, see /index/versions) for that row, and all are excluded; a display-only umbrella never excludes anything. The older raw Wikidata P179 series lift is off by default. Where the store carries Jev's More Like This scores (jev_like_*, oxyc/den-dataset#132) the titles they weighed — the row's top ten and a few promotions — lead it in Jev's order, and a top-ten title without a score keeps its place; a seed without scores keeps atlas's row. The playground's jev_mode=0 shows the row without them |
GET /index/suggest/<movie|series>/<tmdbId>.json?skip=&limit= |
with INDEX_QUERIES on: {mixed:[{type,id}],titles,total}, one title's You Might Also Like — the row POST /index/suggest.json answers as its mixed — paged like /index/similar (20 by default), with titles the page's cards (title, posterPath, year, genreIds, primaryGenre, imdbId, originalLanguage) in the same order, so a client draws the row without a metadata request per title. A title with no card is in mixed only |
GET /index/neighbours/<movie|series>/<tmdbId>.json?k= |
with INDEX_QUERIES on: {ids}, the plain plot neighbours (12 by default, at most 50) |
GET /index/studios.json |
with INDEX_QUERIES on: the iconic studios (see Studios below) — {studios:[{id,name,movies,series}]}, each studio's own Q-id and how many films and series with a card credit it, most titles first; the shelf a client draws tappable studios from. Empty for a store without the studio sections |
GET /index/awards.json |
with INDEX_QUERIES on: every award ceremony or awarding body a title here was recognised at — {ceremonies:[{id,name,movies,series,won:{movies,series}}]}, most titles first; each one's row is sel=award:<id>, its winners' sel=won:<id>. Empty for a store without the award sections |
GET /index/studios/<movie|series>/<tmdbId>.json |
with INDEX_QUERIES on: one title's iconic studios — {studios:[{id,name}]}, in the order it credits them, each once; what a detail header links. Empty for a title with none or not in the corpus |
GET /index/title/<movie|series>/<tmdbId>.json |
with INDEX_QUERIES on: one title as the corpus describes it — the card /index/row draws, indexed, the effective minimumAge (0..18, explicit null unknown), and every published maturityRatings observation as {authority,authorityId,rating,normalizedAge,assessed?,runtimeMinutes?,format?}. When an enabled box-local provider has an applicable observation, publicMinimumAge preserves the dataset value and runtimeMaturityRatings names the operational observation, including authority, authority id, jurisdiction, warning names, source URL and fetch time. Display-only official labels remain present with normalizedAge: null; an old store returns maturityRatings: []. The response also carries labels and plotFacets at the display floor, the facts' countries, languages, runtime, basedOn, makers and cast ({id: Q-id, name, tmdbId?}, cast at most 60 with castTotal), and awards ({id: ceremony Q-id, name, won}) when it has any. companies, networks, subjects, places and authors (source authors), when it has any, are {id: Q-id, name, titles}: the id selects it in the filter (company:<id>, network:, subject:, place:, author:), titles how many titles carry it. When curated, franchise is {id,name,confidence,source,era:{id,name,order},umbrella?:{id,name}}. When it has any, otherVersions is [{type,id,kind}] in /index/versions' order. A title with no card is {type, id, indexed: false, minimumAge: null, maturityRatings: []}, and where the store's optional Wikidata names (wd_name_*, CC0) name it, also {title, year?, originalTitle?, source: "wikidata"} — names only, read by key here and nowhere else: search, browse, ranking and coverage never see them |
GET /index/franchise/<movie|series>/<tmdbId>.json |
with INDEX_QUERIES on: the title's curated primary franchise and its mixed movie/series members. The seed's era comes first, then the other eras nearest it (the seed's kind, animated or live action, first, then the nearest in time); within each era the newest member comes first. Each member carries era and franchiseOrder. A title without a curated primary — including every title in an old store — returns franchise:null and members:[] |
GET /index/versions/<movie|series>/<tmdbId>.json |
with INDEX_QUERIES on: the other versions of the title's story (#112) — remakes, and other adaptations of the same source work, grouped by den-dataset from Wikidata P144 and P4969 (recasts of a title character from TMDB's credits can link them too, but that is off: versions::CHARACTER_VERSIONS) — as {seed:{type,id}, versions:[card + kind + group?], total}. kind is source (the same book, play, character, …) or remake (linked through a film or series). A version in a curated franchise brings every member of that franchise, each with group: {id, name, label, era:{id,name}}; label names the row's group, with the country most of its members were made in ("Wallander (Sweden)"). Titles and franchises in release order (a franchise at its first member), a title with no date last, the most popular first on one date; a franchise's members era by era, release order within each. Members of the title's own curated primary franchise are left out, because /index/franchise shows them. More Like This leaves out every version, every member of a version's franchise, and every version of the title's franchise members. Empty for a title with none and for a store without the sections |
GET /index/search.json?q=&type= |
with INDEX_QUERIES on: semantic search in one request — the query embedded by den-embed, then {titles:[{type,id}]}, the 24 nearest; 503 without den-embed |
GET /index/facets.json?q= |
with INDEX_QUERIES on: the facet lane — {facet,titles}, titles matching the query's country/decade/type most-voted first, a leftover theme ranked to the front (best 50) |
GET /index/query.json?q=&type=&skip=&limit= |
with INDEX_QUERIES on: search in one request — {parse,people:[{qid,id,name,credits}],hits:[{type,id,score,title,posterPath,year,genreIds,originalLanguage?,f}],total,semantics,coverage}. total is the retrieved pool, not a corpus count, and semantics says so; coverage names every constraint the query applied, its value, how it was applied and how many titles have that field on record out of the titles of the type asked for (the corpus when none was). The query is read for a country, decade, type, genre, label, plot facet, person (the facts' credited people, by name or alias; id is the TMDB person id) or a place after "set in" / "takes place in" (P840, parse.places). Words after not, no, nothing, without, except or excluding rule out what they name, content warnings included ("without gore", parse.excluded.warnings); a ruled-out phrase that names nothing atlas holds drops nothing and is listed in unknownValues as not:<phrase>. Every candidate (fuzzy title under any of its names, facet, label, plot facet, a named person's titles, plot and premise vectors on the leftover) is scored Φ·[2.0·title + w·max(plotSemantic,premiseSemantic) + 0.25·label + 0.10·plotFacet + 0.8·person + 0.15·popularity], so an exact title always outranks a theme match. Only candidates with a positive internal score are returned; wire scores are rounded to four decimals. 40 a page, at most 100 |
GET /index/row/<movie|series>.json?<axis>=<value>…&mood=&subgenre=&skip=&limit= (also /index/plot/…) |
with INDEX_QUERIES on: a browse row from the store's facet axes (ending, era, chronology, pacing, tone, …) and the labels' moods and subgenres (≥ 0.55), alone or combined. A few thin values also answer as merged display rows — ending=unresolved (open, ambiguous, cyclical), ending=unhappy (tragic, bittersweet), chronology=out-of-order (nonlinear, framed, parallel-strands), listed in /index/schema.json under fields.<axis>.merged; similarity keeps the raw values — {titles:[{type,id,title,posterPath,year,genreIds,originalLanguage?}],total,coverage}, the titles carrying every constraint, most confident then most voted, 24 a page, at most 100. coverage.fields reports every filtered field against the selected movie/series population and also names the full corpus; a missing facet is unknown, never false |
…and the same route's taste tilt: &tilt.liked=m550,t1396&tilt.disliked=m176&tilt.era=<center>,<spread>&tilt.w.embedding=&tilt.w.dislike=&tilt.w.era=&tilt.w.square= |
the WHOLE row reordered for one household before the page is cut, so a title the plain order puts on page 3 can lead page 1. Ids are m/t + TMDB id, at most 500 a list; tilt.era is den-core's fitted curve and its presence is what turns the era term on (omit it for a row already fixed to an era). The four tilt.w.* levers are den-core's tilt::Weights, each independently overridable and defaulting to its shipped value. Reorder only — same total, same titles, so paging state stays valid; the answer adds taste, the fingerprint of the order the page is a slice of. A taste atlas cannot place returns the plain row, never an error. Memoised per (row × taste × weights); a request carrying tilt.* is private-cached rather than public, since its URL names the household's titles |
GET /index/filter/<movie|series|all>/counts.json?sel=… |
with INDEX_QUERIES on: stackable filters' counts (see Filters below) — {total, kinds: {<kind>: {mode, complete, values: {<id>: n}, labels?, selected?, excluded?}}, coverage, ignored, kindsUnavailable?}: for every value of every listed kind, the titles of the type (of both, under all) carrying the whole selection AND that value; zero counts left out, except a selected or excluded id, which always appears. Den Web hides an option at 0 in a complete kind |
GET /index/filter/<movie|series|all>/titles.json?sel=…&skip=&limit= |
with INDEX_QUERIES on: the titles carrying the selection, most voted first (under all, films and series merged by rank within type, each card naming its type; in similarity order when a like is selected), as the cards /index/row returns — {titles, total, order, coverage, ignored, kindsUnavailable?}; 24 a page, at most 100, skip a multiple of limit. order fingerprints the order the page is a slice of (it moves with the rating provider's votes, or is like:<id>) |
GET /index/kids-home.json |
with INDEX_QUERIES on: the reviewed Kid Home row catalogue — {rows:[{id,title,heading,group,source}]}. heading is {id,args,fallback}: locale-neutral identity and verbatim substitution values plus canonical English; legacy title remains unchanged. A source is an Atlas query phrase with its reviewed score floor, an exact franchise seed or an exact filter selection; clients fetch those ordinary policy-bound routes, omit rows too short to display and keep paging the rows that can page |
GET /index/filter/<movie|series|all>/values/<kind>.json?sel=…&q=&limit= |
with INDEX_QUERIES on: one kind's values under the selection, labelled, most titles first — {kind, mode, values: [{id, name, count, tmdbId?}], complete, ignored, kindsUnavailable?}, 10 at most; with q (2 characters or more), only those with a word starting it: the typeahead for people, studios, subjects and places, over names and aliases, the value q names exactly first, then those holding it as whole words, then the rest, most titles first within each. character is search-only: q of 3 or more, a name starting it, 5 at most |
GET /index/filter/<movie|series|all>/people.json?sel=…&traits=…&order=&skip=&limit= |
with INDEX_QUERIES on: the people credited on the titles carrying sel, holding every person trait in traits (see People below), in order: prominence (the default), credits, name, born_asc or born_desc — {people: [{id, name, tmdbId?, credits, roles, gender?, born?, died?, citizenship?, occupation?, birthplace?, birthcountry?, knownFor?}], total, order, orderUnavailable?, labels, coverage, ignored, ignoredTraits?, unknownTraits?, …}; paged as titles.json. labels names every trait id on the page; knownFor is up to 3 of the person's matching titles, [{type, id, title, year}], highest in their type's popularity order first |
GET /index/filter/<movie|series|all>/people/counts.json?sel=…&traits=… |
with INDEX_QUERIES on: for every value of every person trait, the people credited under sel and the other traits holding it — {total, traits: {<trait>: {mode, complete, values, labels?, selected?, excluded?}}, traitCoverage, …}, shaped as counts.json's kinds |
GET /index/filter/<movie|series|all>/people/values/<gender|citizenship|occupation|birthplace|birthcountry>.json?sel=…&traits=…&q=&limit= |
with INDEX_QUERIES on: one person trait's values counted as people/counts.json counts them, but every value rather than the top 30, labelled, most people first — with q, those whose name or an alias has a word starting q — {kind, mode, values: [{id, name, count, iso?}], complete, denominator, …}, shaped as values/<kind>.json; a birth country carries its ISO code as iso |
POST /index/labels.json |
with INDEX_QUERIES on: {titles:[{type,id}]} → {labels}, each title's labels or null |
POST /index/score.json |
with INDEX_QUERIES on: {space?,liked,disliked,candidates} → {space,scores:[{taste,dislike}]}, cosine to each centroid, clamped at 0 |
POST /index/suggest.json |
with INDEX_QUERIES on: {seeds (≤8),exclude?,limit?} → {perSeed:[{seed,ids,mixed}],pooled,pooledMixed}, per seed and pooled in seed order. A seed the store has fan picks for (store-v3's optional fan_picks_*, #121: titles a model named that a fan would also love, matched by den-dataset) gets those, in the model's order, less its franchise and other versions and the first 20 of its More Like This (ids also less the other type); with fewer than 3 left it gets none. FAN_PICKS in src/queries.rs switches this off. Any other seed gets seed-focused structural affinity, with More Like This appended for coverage and as an old-store fallback, then kept to titles of the seed's form (animated or not), no comedy or family title beside a seed that is neither, sharing one of the seed's specific genres (all but Drama, Action and Adventure) where both have any, seen and rated enough (a tenth of the seed's votes up to 150, a score of 6), and — beside a seed from 1990 on — no more than 35 years older unless a classic (7.5, 2,000 votes); mixed and pooledMixed include films and series together, {type,id} |
POST /recommend |
with INDEX_QUERIES on: {surface?,service?,now?,services?,library:[{type,id,weight,hint?:{title?,year?,overview?,…}}],owned,hide?,candidates?,limit?} → {slides:[{type,id,imdbId?,why}],unjudged:[{type,id}],unjudgedCount,libraryUnjudged,pool:{libraryIndexed,libraryUnindexed,libraryEmbedded,…},facts,scorer,datasetVersion}, the titles a featured surface leads with (40 by default, at most 300); an unindexed library title can transiently shape plot fit from its hint (at most 32 new embeds/request, while every cached title is free); no-store |
GET /recommend/<scope>.json?day=YYYY-MM-DD&skip=N |
with INDEX_QUERIES on: the billboard for everyone, no library — <scope> is home, movies, series or service/<id> — ranked from atlas's own lists as of that day (default today, UTC) → {slides:[{type,id,imdbId?,why}],next?,day,scorer,datasetVersion}, 100 slides a page and next the skip of the one after. Ranked once per scope and day and kept in memory, so every page comes from the same ranking. public, max-age=300, s-maxage=<until the day ends>, stale-while-revalidate=86400, tagged atlas; a browser drops what its household owns or hides and re-ranks the rest by its taste |
GET /playground |
with PLAYGROUND and INDEX_QUERIES on: the tuning page. /playground/params.json lists every More Like This knob (den_index::SimilarParams) with its range and production value; /playground/similar/<movie|series>/<tmdbId>.json?<knob>=…&limit= ranks with those overrides and returns each title's score, production rank and per-signal points. Inspection reports primary_franchise when the dedicated franchise row reserved a candidate. The ranges bound one request's cost (pool_k ≤ 1000, max_row ≤ 400, limit ≤ 200). Besides the knobs every playground route reads watched=movie:…,series:… (dropped from each row, and each row ordered by den-core's tilt with them as liked) and filter.<axis>=<value> (only candidates carrying it, as /index/plot reads constraints). /playground/rows.json?seeds=movie:5723,series:1438&suggest=…&<knob>=…&limit= answers the same for up to 12 seeds, at most 50 titles each, plus the separate structural-affinity You Might Also Like (/index/suggest) for up to 6 suggest seeds, in one request (no seeds/suggest: the page's defaults; empty: none), and with judged=1 also judged, the /playground/judged.json answer for the same knobs — what the page asks on each change. The More Like This knobs do not change that separate affinity row. /playground/judged.json?<knob>=… scores those knobs against judged/rail.json (nDCG@10 and companions, den_index::eval), dev and test halves apart, beside production — one tuned row per judged seed. /playground/export.json?<state>&full=&judged= writes the page's state and a read-only snapshot as a versioned file (the shape is documented at playground::FILE_FORMAT); POST /playground/import.json reads one back. /playground/titles.json?q= is the page's title autocomplete: the TITLE_SEARCH fuzzy index over films and series in one answer, kept to titles the store has (
8BD0
{titles:[{key,title,year}]}, at most 8); a 404 without the title index, when the page searches with /index/query.json instead. An unknown knob or out-of-range value is a 400 naming it; no-store; off ⇒ 404 |
GET /airing/tv/<tmdbId>[?imdb=tt…] |
with TVMAZE on (the default): one series' episode release dates from TVmaze — {source:"tvmaze",status:running|ended|tbd|development|unknown,service,next:{season,episode,name,date},previous:{…},upcoming:[{season,episode,date}],seasonPremiere:{season,date}}, next the first episode out today (UTC) or later, upcoming those out within 14 days, seasonPremiere the first episode of next's season (else previous's); absent ones are null. date is a calendar date where the episode is released; a broadcast network's episode also has time and zone (IANA), a streaming one never. TVmaze is asked by the dataset's IMDb id for the series, else the imdb hint. public, max-age=21600 with an ETag; 404 when TVmaze has no show (cacheable) or the series is still being looked up ("pending":true, no-store) |
POST /embed |
a search query ({"text":…}) embedded by den-embed; 503 when EMBED_URL is unset |
GET /metrics |
Prometheus text for Authorization: Bearer $METRICS_TOKEN; 404 when the token is unset or wrong |
HEAD /metrics |
Same status and representation headers without rendering or sending the exposition |
other /metrics methods |
405 with Allow: GET, HEAD when authorized; missing/wrong authorization remains 404 |
Discovery GETs accept maximumAge=0..18, unknownRatings=allow|block, an optional explicit two-letter
region, and optional kidAudience=child|child-or-teen. A ceiling defaults unknown ratings to block; without a
ceiling, explicit unknown policy, or Kid audience band, behavior is unchanged. Age eligibility uses the store's
redistributable minimumAge; Kid relevance independently requires the store's made_for_children >= 0.5, or also
made_for_teens >= 0.5 for child-or-teen. Missing relevance evidence fails closed. Both gates run before ranking,
limiting, and pagination; genres, animation, gentle tone, and low intensity are not relevance evidence by themselves.
region is normalized to uppercase and participates with the policy in memo/ETag identity, but is not inferred
from mutable server or install defaults; the current public baseline is the conservative global age.
POST /recommend accepts the same four fields in its JSON body. Atlas never reads a TMDB certification from
the public store or another exported artifact.
Every response carries Access-Control-Allow-Origin: * and OPTIONS answers the CORS preflight. An
unknown path — and a refused /metrics — is a 404 {"error":"not_found"} with cache-control: no-store.
Catalog rows, /embed and the /index/… answers carry Server-Timing: justwatch;dur=<ms> when the row
was fetched upstream or cache;desc=hit when it came from the cache (plus cache;desc=stale when the
last-good copy was served), embed;dur=<ms> for the den-embed call, and total;dur=<ms>. An answer that
is stale or a fallback carries X-Den-Degraded: <reason> with /health's reason slug — a catalog row
served after a failed JustWatch refresh says stale_catalog. Normal answers carry no X-Den-Degraded.
POST /embed returns an int8 vector ({"vector":…,"dims":1024,"model":"bge-m3"}). By default Atlas
forwards to the internal den-embed service. The same engine now lives in this workspace under embed/, so
an opt-in build can run it in-process without changing the HTTP contract. The compatibility den-embed
binary uses that exact library too, which lets den-dataset keep using the same path while deployment moves.
Only Atlas is exposed.
Title search (TITLE_SEARCH) builds an in-memory index of the 100k most popular movies and series in
TMDB's daily ID exports — downloaded once a day, gunzipped straight into the scanner, nothing written to
disk — and ranks by the share of the query's trigrams a title contains, blended with popularity, as the Den
TV app's on-device index does. The index lives in the den-titlesearch crate (crates/), which has no
async runtime, IO or global state, so it also compiles for Wasm and tvOS. Until the first build lands a
search answers empty with X-Den-Degraded: title_index_building. The request log shows the query as
<query>.
Index queries (INDEX_QUERIES) answer from the dataset's plot and premise indexes through the
den-index crate (crates/, portable like den-titlesearch), the same way the Den TV app's on-device
index does. They return TMDB ids only; clients hydrate titles themselves. The indexes load as atlas starts
and stay for the life of the process; a query that arrives before the load has finished waits for it. The
descriptor carries "queries":true when they're on.
Semantic search and the facet lane's theme ranking embed the query through the configured embedding engine
(EMBED_URL, or the opt-in local engine). Unified
query search scans both plot and premise vectors when the latter exist, normalises each scan against its own
distribution, and lets the stronger one spend the single semantic weight; a missing premise index preserves
plot-only behaviour. The facet lane reads the store's country, language, year and vote columns — it used to
read a facets.bin sidecar, which covered 9,086 fewer titles. Taste weights stay with the client:
score returns the raw boosts.
Vote counts — what every browse row is ORDERED by — are TMDB's vote_count, with its vote_average the score
/recommend rates a title with where no upstream list scored it, and TMDB's credits give the list of titles
that share a character and the billing people's prominence weighs a cast credit by. None of it is in the
published dataset: atlas asks den-edge's TMDB proxy for it (TMDB_PROXY) and keeps it in CACHE_DIR on the
box — vote counts in bulk from /discover, sliced by date, about once a month; credits a share of the corpus a
day, and at once for a title TMDB's changes feed names.
Every kept value is dropped six months after TMDB gave it. src/tmdb.rs opens with the rules on what these
numbers may be used for: sort keys, filters, floors, merit and popularity terms and character-link evidence at
runtime — never an embedding, a model or a file that leaves the box. A first boot reads a seed made from
den-dataset's own TMDB cache (scripts/tmdb-seed.py). The store's own votes column, where an older store
carries one, is the fallback for a row nothing is kept for — stated in the load line (row order: …) rather
than left to be inferred. When NEITHER source has a count, /health reports votes_unusable: rows still come
back full, in tmdb-id order, which is the one degradation here that looks like a working addon.
This product uses the TMDB API but is not endorsed or certified by TMDB.
Box-local maturity providers are a separate runtime tier. MATURITY_PROVIDERS=kavi,imda walks the store's
movie and series identities against official classifications. KAVI's public IKLU search covers series. IMDA's
open dataset is fetched in a few paged requests and covers whole films plus TV submissions that explicitly name an
episode or episode range; an episode title is never promoted to its parent series. kmrb adds Korean
self-classifications for streaming movies and series when KMRB_SERVICE_KEY holds a data.go.kr service key.
All accept only an exact title (including Wikidata aliases), a compatible year where the authority supplies
one, and a title identity unique in the local corpus. Both matches and misses are refreshed after 30 days. The
shared provider coordinator limits and paces requests, writes
maturity-providers.v2.json durably and atomically under CACHE_DIR only when state actually changes, honors
authority retry/rate-limit headers with bounded backoff, and keeps only normalized observations plus check times —
never raw API responses or source titles. The observation keeps KAVI's Finnish jurisdiction as
provenance (and likewise SG/KR for IMDA/KMRB), but age guidance applies regardless of the viewer's region: a US
viewer watching Pippi benefits from the same official evidence. Only answers actually affected by runtime ratings
use a five-minute shared-cache TTL because provider data can change without a dataset release or its cache purge;
unaffected routes retain their normal dataset TTL. KAVI's HTTP response says no-store; this operator cache is
therefore an explicit exception, is identified in metrics, and is never copied into den-dataset, an atlas download,
or a release artifact.
/index/filter/<type>/… is Den Web's stackable Search: a selection of values, AND-ed (an item may OR several
values of its kind), and three questions
about it (counts, titles, one kind's values). /index/schema.json's filter object lists every kind with its
mode, id format, listing and floors, the axis aliases and the canonical rules; tests/fixtures/facets-canonical.json
pins url → canonical url pairs a client can test against.
selis[-]<kind>:<id>items joined by,. Kinds:genre(TMDB id; a series also under 10759/10765/10768),language,country(every one a title lists),region(one pick: any of its countries; the table iscrates/den-index/src/regions.rs, published as the schema'sregions),decade(first year; a series' first air date),mood,subgenre,primary(the labels' primary genre),animated(yes/no),runtime(under-90,90-120,120-150,over-150; a series per episode),source(adapted from),rating(the rating provider's average ≥ 6, 7 or 8 out of 10, on 10+ votes),technique(≥ 0.4),audience(≥ 0.5),critique(≥ 0.6),warning(depicts, ≥ 0.5), the twelve plot axes (structureresolves by value; the merged rows count their members' union), the entity kindsperson(anyone credited),made,director,writer,creator,cast,author(the author, P50, of a work the title is adapted from, P144:author:Q39829is adapted from Stephen King; not a credit),company,network(series),subject,place,formatandfranchise(Wikidata Q-ids; top 30 listed, studios, subjects, places and formats from 5 titles, formats from a curated list; stores without split credit or source-author sections do not offer those kinds; an unnamed franchise is labelled by its Q-id),studio(an iconic studio by its own Q-id: every item it is credited as; see Studios),award(a ceremony or awarding body by its Q-id, won or nominated there:award:Q19020the Academy Awards) andwon(won at least one award there), from Wikidata's P166/P1411 filed under the ceremony, categories not held — a title with no award on record is not known to lack one, so-won:keeps only titles recognised elsewhere, and a store without the award sections offers neither;character(role names played in 2+ titles, from the character provider; search-only),like:<tmdbId>(the set/index/similaranswers) andfans:<tmdbId>(the title's You Might Also Like, as/index/suggestserves it).allasks every question of films and series together: counts and values over both; titles are the two types' own orders merged by rank within type (position in its type's order ÷ that type's size, ascending, films first on a tie), so series run at their share of the corpus, evenly spread, and the most popular series sit beside the most popular films. A selection merges the filtered lists by the same keys. Every kind answers there; one only a type has (network, a series-only genre such as 10762 kids, a composite) matches only that type's titles, and a film genre id matches the series filed under it too (genre:28is action films and Action & Adventure series).likenames its title's type —like:movie-550,like:series-1396— and answers/index/similar'smixedrow;fansnames its type the same way and answers/index/suggest's row; a bare id is a400there.- Mode.
andkinds hold several values a title (two genres are both genres) and count under the whole selection.singlekinds hold one (decade, runtime, rating, primary, animated, the plot axes, like, fans) and count each value under the selection WITHOUT the kind's own pick, so the other decades read as alternatives. - Exclude with
-kind:id: the titles known for the kind (something on record for it) and not carrying the value.coveragein every answer says how many titles of the type each applied kind is known for. - Or with
|:country:FR|ITis one item, French OR Italian; separate items still AND (country:FR,country:ITis both,country:FR|IT,decade:1990either of the 1990s), and-country:FR|ITis neither (known for the kind, carrying none). Every kind takes a group exceptlikeandfans; a one-pick kind's group is how to ask for several (decade:1980|1990). A value the kind lacks is named inunknownValuesand the rest of its group applies. On a plot axis a group matches both tiers as a lone value does.counts.jsonandvalues/<kind>.jsoncount a kind with a group without the group — each value is what adding it would give — and every other kind under it;selected/excludedlist a group's ids one by one. - Canonical form: items normalised per kind, sorted by kind, then positive before excluded, then id, each
once, ids encoded as
encodeURIComponentdoes,:,-|literal; a group's ids sorted and deduplicated, the group sorting by its ids joined with|, and a one-value group written without|(%7Cis accepted as a second spelling); thenskip/limit(titles) orq/limit(values), each only when not its default. The canonical URL is public for an hour and revalidates by ETag. Any other spelling is ANSWERED, not redirected (den-edge's relay dropsLocation): the same body,private, max-age=60, withContent-Locationnaming the canonical URL. At most 16 values, a group counting each of its values, and a 2,048-byte query; a malformed item, an id its kind cannot read, a group oflikeor of values resolving to different kinds (structure:single-day|nonlinear), askipoff a page boundary or a short prefix is a400. - Unknown, unavailable, not offered. A kind atlas does not know, or cannot answer now, is left out of the
result and named in
ignored; a value its kind does not hold (a typo, a label in the wrong case) matches nothing and is named inunknownValues.kindsUnavailablelists the kinds this atlas should answer and cannot through a failure at runtime — the facts or facet rows did not load, the rating provider has no votes yet — and such an answer carriesX-Den-Degraded: filter_kinds_unavailableand is kept five minutes. A kind the loaded dataset has no data for — a section the store does not carry, a score table no title reaches the floor of — is simply not offered: absent fromkinds, not unavailable, cached as usual. On 5b1c3213b6a1warningis not offered (itsdepictsscores are title-only, the highest 0.28); a store with article-based scores offers it on load.characteris offered once its provider has built the names. - Providers.
ratingand the vote order read whichever provider fills atlas's ratings holder, andcharacterits character links: TMDB's, kept on the box (src/tmdb.rs), as a filter and a sort only. - Cost on that store: 509 values in bitsets plus posting lists for the entity kinds, 11 MB, built at load in
~0.1 s; the name index for
values/…?q=is built on the first search, 7.7 MB. The empty selection's counts are kept (0.1 ms); a three-kind selection answers in under 1 ms, the broadest single genre in ~6 ms, a titles page in ~0.1 ms, a name search in ~7 ms, and a page of people or cast without a prefix in 2–7 ms (only the values that can reach the page are named). - People (
people.json,people/counts.json;src/filter/people.rs) answer who is credited on the titlesselmatches.traitsis a second list insel's grammar and canonical order, placed after it, because a trait is about a person and means nothing to the title routes. Kinds:gender,citizenship,occupation(Wikidata Q-ids of the items the store names: P21, P27, P106),birthplace(the place of birth, P19, by Q-id:birthplace:Q1754is Stockholm),birthcountry(the country, P17, of that place — never read from citizenship — by ISO 3166-1 alpha-2 code or Q-id:birthcountry:SEandbirthcountry:Q34are the same question; canonical as written, the code uppercased; a code matches every country carrying it, and one with none, the Soviet Union or the Netherlands' Q55, is asked by Q-id;people/counts.jsonnames the codes of the countries it lists incodes),born(a range of P569 birth years) androle(cast,director,writer,creator: the credit on a matching title; two roles mean both on one title,role:cast|directoreither,-role:castcredited without it). Every trait takes an OR group asseldoes (citizenship:Q30|Q145is American or British,citizenship:Q30,citizenship:Q145dual citizens);people/counts.jsoncounts a trait with a group without it and the others under it,roleover the credits walked without the group.borntakes a range of birth years, both ends inclusive and either left open —born:1976-1996,born:1976-,born:-1996, each year 1800 to next year — one per request, never in a group, and with no other positivebornbeside it; a reversed range, a year that is not digits or out of that span is a 400. A birth dated only to its decade or century is in a range when its whole span is, out of it when none of it is, and unknown when it straddles an end.gender,birthplace,birthcountryandbornare one pick and count without their own pick: under a range,people/counts.jsonnames the range inselected/excludedand exposes no duplicate birth-decade values. Gender values held by at most three people in the full corpus are counted together asgender:other; the P21 value “gender not disclosed in work” is omitted. Unknown is never a match: a person with no gender on record matches neithergender:nor-gender:— nor a person born in a place in no country anybirthcountry:— andtraitCoveragesays how many credited people each applied trait is on record for (for a range, those it is decidable for). A store without the trait sections answers credits and roles and names the rest inignoredTraits; one with the traits and not the birthplaces (published before them) namesbirthplaceandbirthcountrythere.order(aftertraitsin the canonical URL, left out whenprominence) ranks the people:prominencesums a person's 5 biggest matching titles, each weighing 1 − its rank in its own type's popularity order over that type's size (the share thealltitle order interleaves by), so a few hits outrank a long run of mid-table titles. A cast credit weighs its title by its TMDB billing: in full for the first three billed, 3/position after (¾ fourth, 0.3 tenth), a fifth below the tenth; a credit with no billing kept (no credits for the title, no TMDB id for the person) or on a title the person also directs, writes or created weighs in full. Only prominence reads the billing;creditsis most matching titles, then most titles in the corpus;nameis folded as search folds names;born_asc/born_descput people with no birth on record last. Ties go tocredits, then the Q-id. The score is never published (it comes from TMDB's vote counts and billing). Without a popularity order (no vote counts kept) prominence answers incredits, withorder: "credits"andorderUnavailable: "prominence". With a popularity order each person carriesknownFor: their three heaviest matching titles by the same weight (so a director's are the titles they direct whenrole:directoris asked), named by the corpus cards; without one it is left out rather than picked by TMDB id.people/counts.jsonlists an item trait's top 30 values;people/values/<trait>.jsoncounts every value the same way and finds one by name or alias (citizenshipq=iceland), for a value past the top. Nothing is built at load: each answer walks the matching titles' credit lists in place. On b2c60751c955 (130,116 people credited), male cast of 2020 films answers in ~0.8 ms warm (~39 ms the first time, mapping the pages in), every person unfiltered in ~4.4 ms, and the unfiltered trait counts in ~11 ms.
The iconic studios are the production companies a viewer browses by — a house style (Ghibli, Aardman, Hammer)
or a curation (A24, Searchlight). They are a hand-kept list in den-dataset (data/iconic-studios.json), because
no measure over plots or credits finds the curatorial labels, and reach atlas as the store's optional studio_*
sections (den-spec wire/store-v2.md, oxyc/den#132). One studio is often several Wikidata items (Toho and Toho
Animation, HBO and HBO Films); a title crediting any of them is the studio's, and the studio is named by its own
item.
- The list:
/index/studios.json, for a browse shelf. - A title's studios:
/index/studios/<type>/<tmdbId>.json, for the detail header's link. - The studio row:
/index/filter/<movie|series|all>/titles.json?sel=studio:<id>— paged and ordered as every filter row, most voted first.company:<id>still selects the single item it names and would miss a studio's other arms. The kind combines with every other (sel=decade:1990,studio:Q182950) and is listed whole incounts.json, named inlabels.
A store without the sections loads and serves as before: both studio routes answer empty and studio is not
offered (a selection naming it is ignored). On b2c60751c955 all 48 are credited, from Amblin's 119 titles to
Neon's 5.
total on search is the number of retrieved candidates, not a corpus aggregate. Clients that present corpus
counts must use the field coverage and denominators from /index/schema.json; browse rows and search include
the relevant coverage inline. Group-by is advertised as unavailable until it can preserve that contract. Every
response that is or may be JSON (any +json type, or none) is checked at the final HTTP boundary for
expressive prose fields, and since nothing is served verbatim any more, that boundary is the whole guard. Only
/manifest.json and /dataset.json are exempt, by exact path: both carry atlas's own description. A future
MCP/TMDB hydration route therefore fails closed wherever it is mounted, instead of re-serving an overview,
synopsis, description, tagline, or equivalent prose.
POST /recommend ranks what a featured surface leads with — the Den web app's billboard — so no client
ranks. It is the web app's billboard.ts ported: what is new in the world and new to this library, with
attention (a place in Trending Everywhere, the household's "new on" lists, and the client's own lists) and
quality, multiplied by the library's taste and discounted where the library's own More Like This already
reaches. It describes each title from what atlas holds, all of it out of the one store: the labels, the
country/language/year/vote columns, and the Wikidata facts; a candidate's hint (release
date, genres, countries, popularity, rating and votes) fills only what those leave unknown. Rating hints affect only
that response: Atlas never writes them to catalogs, its dataset or replay fixtures. Existing upstream scores,
including JustWatch's IMDb scores, remain in Atlas's own catalog output. unjudged names the 20
titles atlas knows nothing about that are most worth describing; a client that can describe them asks again
with their hints, since an undescribed title is dropped once enough are judged. library is
[{type,id,weight,at,hint?}],
owned every title the library holds, which never appears; hide is the household's rules
(minYear, genres, languages, anime). Each slide's why gives its terms. Its reason is the strongest
meaningful contribution on the scorer's common scale: similar, profile, people, franchise, arrived,
recent, upcoming, timely, quality or buzz (or null when none is strong enough). Clients can turn that
stable code into short copy. The raw score, fit, similar, profile, people, confidence, fresh, arrived,
quality and buzz terms remain alongside it for diagnostics and richer future explanations; they are deliberately
not directly comparable with one another. Server-Timing carries lists;dur= and rank;dur=. Bodies over 512 KiB
are refused.
surface is home (both types, the default), movies or series. services is the household's
[{id,country?}]: id a provider id from the catalogs' denProviderIds, country ISO 3166-1 alpha-2, else
the install's country; none picked, every service the install carries. A service channel (one service's page)
adds service: {id,country?} in the same shape, and its slides are then only titles atlas can show are on
that service in that country. Atlas holds no per-title availability, so that means titles one of its lists
for that service named — "new on" and "popular on" (JustWatch, filtered by that country's package), the
service's Movie of the Night Top 10 and additions, and Trending Everywhere read over that one service, i.e.
its own trending chart — plus the client's candidates, which a channel sends from its own rows and which
are trusted as the service's catalogue. The personal pool's whole-catalogue titles, other services' lists
and Netflix's US Top 10 don't reach a channel. surface still filters the types, and taste, owned, hide
and scoring are unchanged. A service whose id the install doesn't carry reads no lists and ranks the
client's candidates alone. The response is the same shape.
The invented households in fixtures/billboard/ can be replayed as one deterministic evaluation surface:
CACHE_DIR=<cache> den-atlas billboard-eval <dataset>. It prints the existing slate-shape checks and, beside
them, hand-judged nDCG@10, bad@10, judged coverage (with the number of displayed titles as its denominator),
and holdout recall@40 (with the number of held-out positives as its denominator). judged/billboard.json is
the versioned, validated judgement file. It intentionally contains no relevance labels until a human supplies
them; an empty case prints relevance as unavailable rather than as zero. Set
BILLBOARD_EVAL_UNJUDGED=<file> to export the current candidates in their original rank order for later human
labelling. BILLBOARD_JUDGED=<file> selects a different judgement file. Neither command calls a model.
The curated-franchise corpus has its own stronger, no-network acceptance command:
den-atlas franchise-check <dataset>. It is intentionally separate from the general check, because an old
store without the optional sections is still a valid backward-compatible store. franchise-check reads
judged/franchise.json and refuses the generation unless the exact 46-title Beck group, the Wallander
control, Bond, Spider-Man and Star Wars group correctly; the named adaptation/catalogue controls stay apart,
including separate productions of one source (the British Wallander, the non-Eon Bond films); each row leads with the seed era and lists each era newest first; and More Like This contains none of
the seed's actual primary-franchise members or other versions. Run it on the staged post-rebuild store before
deployment. The ordinary rail-eval also removes those reserved titles from its judged ideals: a sequel owned by
the franchise row, or a remake owned by Other versions, is no longer counted as a recommendation the More Like
This ranker failed to return.
/metrics publishes only what the addon already knows: atlas_build_info{version},
atlas_dataset_loaded, atlas_dataset_info{dataset_version,taxonomy,embedding_model},
atlas_dataset_titles, local admission state and totals as
atlas_admission_{active,queued,refused_total,completed_total,hold_seconds_total,capacity,queue_capacity}{class="cpu|body"},
runtime-provider coverage as
atlas_maturity_provider_titles{authority,media_type,status="catalog|checked|matched"}, and the two catalog
signals behind /health — atlas_catalog_fresh and atlas_catalog_schema_suspect.
Catalog-row and maturity persistence expose successful replacement, bytes-written, error, and last-write-duration
series under atlas_catalog_cache_* and atlas_maturity_cache_*, so write amplification is visible without
tracing host syscalls. Catalog-row replacements wait in a fixed one-second window; their detached coordinator is
process-bound and is not flushed during shutdown, so any stop or restart may discard at most that second of
disposable cache rows. The next access fetches them again.
Every variable is optional; the binary reads the process environment only (no .env loader).
.env.example carries the same list.
| Variable | Default | Purpose |
|---|---|---|
PORT |
8080 |
the port to listen on |
DATA_DIR |
data (the image sets /app/data) |
directory holding dataset.meta.json and the store it declares |
JW_COUNTRY |
US |
catalog country when an auto install forwards none |
JW_PROVIDERS |
all | provider subset for an install with no <region>_<codes> segment |
JW_CACHE_TTL_SECS |
21600 |
in-process freshness of the catalog rows |
CACHE_DIR |
unset | a writable directory the catalog rows are also kept in, so a restart serves them instead of asking JustWatch again; written only when a row is refreshed. TMDB's kept numbers and credits live here too (tmdb-votes.tsv, tmdb-credits.tsv, tmdb-sweep.tsv), and nowhere else. Unset ⇒ memory only |
EMBED_URL |
unset | den-embed base URL for POST /embed; it wins over EMBED_LOCAL during migration |
EMBED_LOCAL |
off | with an embedded-model build, 1 runs the workspace's engine in Atlas after its model/dims/vector-epoch/token-cap exactly match the corpus and a real inference canary succeeds; without a ready engine /embed answers 503 |
EMBED_MODEL_DIR |
/models |
model directory for the local engine; its other settings are the den-embed variables prefixed with EMBED_ |
INDEX_QUERIES |
off | 1 turns on the /index/… routes (taxonomy, label rows, More Like This, neighbours, semantic and facet search, filters, labels, taste scores, suggestions); the indexes load as atlas starts and stay loaded |
ATLAS_CPU_JOBS |
visible CPUs minus one, at most 4 |
simultaneous local full-corpus scans/rankings; its bounded queue holds twice this many requests for 100 ms, then answers 503 with Retry-After |
ATLAS_BODY_JOBS |
4 |
simultaneous POST body aggregations; its bounded queue holds twice this many requests for 100 ms, then answers 503 with Retry-After. An upload that does not finish in 30 seconds answers 408 and releases its slot |
TMDB_PROXY |
unset | den-edge's TMDB proxy base (e.g. http://den-edge:8080/tmdb), which atlas asks for TMDB's vote counts, scores and credits (see above); atlas holds no TMDB key. Only alongside INDEX_QUERIES. Unset ⇒ atlas serves what CACHE_DIR keeps and asks for nothing |
TMDB_DAILY_MAX |
1500 |
questions atlas asks the proxy in a UTC day, two seconds apart. A full vote sweep is ~4,800, so it spreads over a few days; den-edge's own ceiling (TMDB_DAILY_MAX there) is shared with the web app's guests and must leave room for this |
MATURITY_PROVIDERS |
unset | comma-separated box-local maturity authorities: kavi (Finnish series), imda (Singapore films and episode-range TV submissions), and kmrb (Korean streaming movies/series; needs KMRB_SERVICE_KEY). Exact, unambiguous matches apply to viewers in every region, retain authority jurisdiction, and persist normalized observations under CACHE_DIR; no raw provider response enters dataset or release artifacts |
KMRB_SERVICE_KEY |
unset | data.go.kr service key required only when MATURITY_PROVIDERS includes kmrb; it is sent to the gateway but never cached or logged |
MATURITY_DAILY_MAX |
4000 |
upstream questions each maturity provider may ask in one UTC day |
MATURITY_PACE_MS |
250 |
minimum pause between a maturity provider's upstream questions |
MOTN_KEY |
unset | a Movie of the Night (Streaming Availability API) key. Each service's own daily Top 10 and what was added to it, per country, then lead the "Popular on" and "New on" rows and count as attention on /recommend; Netflix's US Top 10 reaches every billboard. Each service also gets "Leaving Soon" and "Coming to " rows (the next 30 days, read every 3 days while those rows are requested). Each list is fetched in the background only for the markets that request it, at most 30 requests per UTC day per atlas process (the free plan allows 1,000 a month), and kept in CACHE_DIR; run one refresher process or add shared quota coordination before using multiple replicas. If configured cache state cannot be written, no paid request is sent. A country the API lacks is read as a neighbour (Uruguay as Argentina). Unset ⇒ JustWatch alone |
TVMAZE |
on | 0 turns off /airing/tv/… and every TVmaze request. On, a series is looked up and fetched once when first asked for, in the background (at most one request at a time, ~17 per 10 s against TVmaze's 20). A lookup's show is fetched next; title pages alternate with whole-following batches, batches rotate after each series, and due maintenance gets a turn after at most eight first answers. Shows are fetched again only when the one daily /updates/shows read says TVmaze changed one, or a month on. A series nobody asks for in a month is dropped. Kept in CACHE_DIR (tvmaze.v1.json); a 429 or 5xx is waited out as Retry-After asks. Credited in the manifest's denAttribution (CC BY-SA 4.0) |
RECOMMEND_FIXTURES |
unset | a writable directory each POST /recommend is kept in as <surface>.json (a service channel's as <surface>-service-<id>[-<country>].json): the body as sent (the household's library included), atlas's lists for it and the moment it was ranked. den-atlas replay <file> ranks one again against DATA_DIR with that binary's scoring and prints every slide with why. Unset ⇒ nothing kept |
TITLE_SEARCH |
off | 1 builds the daily title-search index and declares the den-titles search catalog. Off by default: the Den TV app fuses every addon search catalog into its text search |
METRICS_TOKEN |
unset | bearer token for GET/HEAD /metrics; unset or empty ⇒ 404 |
PLAYGROUND |
off | 1 turns on the /playground tuning routes (only alongside INDEX_QUERIES); computed per request, so enable it only where anyone reaching atlas may spend that. /index/similar never reads an override |
LOG_REQUESTS |
off | 1 writes one stderr line per request, <METHOD> <path> <status> <ms>ms, with a config segment shown as <config> and the query dropped; a degraded answer adds degraded=<reason> (the same reason /health//ready report), so the one request it affected carries why |
LOG_IDENTITY |
on | Gates the title names/ids in /recommend's decision log (one throttled line per answer naming the pool/library counts, slide count, and why the top slides scored what they did). 0/false/off/no (case-insensitive) drops the names and ids; everything else in the line is unaffected |
Completed POST bodies remain charged to the body class across a cold index load until the request acquires its next bounded CPU stage (or finishes a cheap answer), so near-limit buffers cannot accumulate behind one lazy load. Recommendation then holds one CPU admission slot across its local preparation, catalogue-list lookup and final blocking rank. This deliberately limits successful work during overload and may hold a CPU slot while a cold list is fetched, but keeps one simple bound over all per-request recommendation work. Catalogue routes, their single-flight fills, and shared/Edge cache hits do not use this gate. Heavy index route classes enter CPU admission even when an in-process memo could answer them; avoiding a racy cache-specific fast path keeps overload behavior predictable.
scripts/fetch-dataset.sh # prep ./data from the den-dataset `data-latest` release (labels + vectors + gzip + meta)
cargo run # http://localhost:8080 (add /manifest.json in Den → Plugins)
cargo run --features embedded-model # opt-in local-engine build; still needs EMBED_LOCAL=1 and model files
cargo test # the caching layer (ETag / Range / gzip / 304), routes, catalog, shutdown
cargo fmt --all --check # CI gates on this — see below if the command is missing
scripts/release.sh --prepare 0.96.15 # opens a VERSION-only PR when a future release needs a bump
scripts/release.sh 0.96.14 # from exact merged main: waits for CI, tags, and waits for promotionIf cargo fmt reports "no such command", the toolchain has no rustfmt component and there is no rustup
to add one. Run the formatter through nix instead — it reads this repo's rustfmt.toml and matches CI
exactly:
nix run nixpkgs#rustfmt -- --edition 2021 --check $(git ls-files '*.rs')Worth doing before every push. Two releases in a row built nothing because a wrapped expression was left in
a shape rustfmt disagreed with: docker-publish never produced an image, and den-update then correctly
reported the box was already at the previous digest — a failure that reads as "nothing to deploy" rather
than as a broken build. Checking line widths by hand does not substitute: rustfmt also JOINS short wrapped
lines and SPLITS long array literals, neither of which a width check can predict.
fetch-dataset.sh is anonymous (needs curl, cosign,
python3 and shasum; set DEN_COSIGN when cosign is not on PATH). It stages the raw manifest, fetches its
digest-addressed Sigstore bundle, and verifies that exact manifest was signed by
oxyc/den-dataset/.github/workflows/daily.yml@refs/heads/main before parsing it or downloading blobs. During
the proof-name migration only, an absent digest-addressed bundle falls back to the fixed-name bundle; an
invalid, mismatched, timed-out or unavailable proof never does. Every request has a connect timeout and a
total deadline: 60 s for the manifest and proofs (DEN_PROOF_TIMEOUT_SECS) and 300 s per blob
(DEN_BLOB_TIMEOUT_SECS), matching the deployed sync; a timeout fails the fetch with ./data untouched.
Overrides must be positive integers: 0, empty and non-numeric values are refused. It then downloads every file dataset.meta.json
declares and verifies each against the authenticated meta's sha256 before moving them into ./data.
storeFile is the one it REFUSES a release without: the store is the only artifact the server reads, and
a manifest that still names the retired sidecars simply fetches files nothing opens. To pick up a new
release, re-run it and restart the server.
Refreshes stage on the destination filesystem and replace files by rename. A load that overlaps one is
refused rather than binding the new files to a descriptor read before them: Dataset::load stats
dataset.meta.json on both sides of its own read and compares size, modification time and Unix file
identity. The deployment sync stops Atlas after all downloads verify, replaces the data, then restarts
it. An interrupted replacement stays stopped with a recovery marker and no live descriptor until the next
successful refresh.
The dataset is produced by den-dataset (taxonomy-backfill finalize
→ publish-dataset.sh) and published as a GitHub Release — the single source of truth this server and the
Den app both fetch. den-atlas no longer reads the Den repo.
The published image (the main-CI candidate that .github/workflows/docker-publish.yml promotes on a v* tag,
only if the tag equals VERSION) is the server only, so mount the data:
docker run -d --name den-atlas -p 8080:8080 --read-only \
-v "$PWD/data:/app/data:ro" ghcr.io/oxyc/den-atlas:latestdocker build -t den-atlas . bakes whatever is in ./data into the image instead.
The live deployment is the Den homelab box: a Podman Quadlet unit (deploy/quadlet/den-atlas.container)
from the den repo's deploy/, whose deploy/README.md covers provisioning, the health-gated updater and
rollback.
ghcr.io/oxyc/den-atlas, host port 8081 → 8080, 256 MiB memory cap, read-only root filesystem, every capability dropped, uid 65532; environment from/etc/den/env/den-atlas.env.- The dataset is bind-mounted read-only from
/var/lib/den/atlas-dataat/app/data, kept current by theatlas-dataset-synctimer, which restarts atlas only when the release actually changed. - Image updates go through
den-update: a new:latestis proved against/healthand/manifest.jsonin a throwaway container before its digest is pinned, and a failed proof rolls back. EMBED_URL=http://den-embed:8080over the shared Podman network enablesPOST /embed. This remains the production path; seedocs/EMBED-MIGRATION.mdfor the unactivated local path.
Release images. Main CI builds ghcr.io/oxyc/den-atlas:candidate-<sha> for every main commit while its
tests run, starts it and checks /manifest.json reports VERSION, scans it, and records its digest. It also
builds, probes and scans ghcr.io/oxyc/den-atlas-embed, the compatibility service consumed under the
cross-repository gates in docs/EMBED-MIGRATION.md, and the currently unused
ghcr.io/oxyc/den-atlas-integrated image from the same bake. When VERSION needs to change,
scripts/release.sh --prepare X.Y.Z opens the required version-only pull request
and stops without tagging or publishing. After it merges, update the checkout to main and run
scripts/release.sh X.Y.Z. Publishing never writes or pushes directly to protected main; it waits for that exact
merged commit's main CI and only then creates vX.Y.Z. Cargo.toml is never changed, so the cached Rust build is
reused and the image carries the version as DEN_VERSION. The tag's docker-publish run verifies and signs all
three exact digests, adds
their :X.Y.Z, :X.Y, :X and :sha- tags and moves each package's :latest; it does not rebuild or
retest. Publishing either optional package does not enable Atlas's in-process engine.
docker-publish also runs every Monday: the weekly run rebuilds the newest v* tag (never main) from its
digest-pinned bases with refreshed OS packages and no build cache, and publishes it as
:X.Y.Z-patch.<date>.<run> and :latest, so package fixes reach the box between releases through
den-update's probe and rollback like any release. Dependabot updates the pinned toolchain/base digests. The
workflow fails if the newest tag predates the candidate pipeline (no VERSION file); cut one release first.
Trivy scans each image before :latest moves: a CRITICAL with a fix available fails the run (on the
weekly rebuild only in OS packages, the part a rebuild can fix), and fixable HIGH and CRITICAL findings go
to code scanning. A finding that does not apply goes in .trivyignore with a reason. Every image carries
SLSA provenance and an SBOM and is signed keylessly with cosign; verify a digest with:
cosign verify \
--certificate-identity-regexp '^https://github\.com/oxyc/den-atlas/\.github/workflows/docker-publish\.yml@refs/(heads/main|tags/v[0-9]+\.[0-9]+\.[0-9]+)$' \
--certificate-oidc-issuer https://token.actions.githubusercontent.com \
ghcr.io/oxyc/den-atlas@sha256:<digest>Smoke test:
curl -s localhost:8081/health # {"status":"ok"}, or "degraded" with a reason
curl -s localhost:8081/manifest.json | jq .resources # ["dataset","catalog"]
curl -s localhost:8081/dataset.json | jq '.count, .embeddingModel, .dims'
curl -s localhost:8081/catalog/movie/jw-nfx.json | jq '.metas | length' # live JustWatch (needs egress)
curl -s -H "Authorization: Bearer $METRICS_TOKEN" localhost:8081/metrics # only with METRICS_TOKEN setTo install: Den → Settings → Plugins → add http://<den-ip>:8081/manifest.json (the app needs https or a
LAN/private-range host over http). Den's discovery then answers from here.