Repository navigation
Description
Component
Knowledge (documents)
Use case
Agents need to find and cite relevant evidence across SQL and TOML files, PDF and office documents, spreadsheets, CSV and Parquet datasets. Some questions also require calculations over those datasets.
Examples:
- Find the SQL definition and configuration that govern retention, citing statement lines and TOML key paths.
- Explain a policy from a DOCX file and a scanned PDF, citing sections and pages.
- Calculate quarterly totals by region from CSV, Parquet or an XLSX table, then compare the result with a management report.
Quality is the highest priority. Preserve source structure, types, permissions and revisions before optimizing scalability, end-to-end latency and cost. Finding similar passages is sufficient for some questions; a numerical answer needs the complete authorized data required by its calculation.
The existing knowledge plane already stores documents and originals, chunks by headings, and fuses vector and BM25 retrieval. The dashboard extracts PDF and DOCX text in the browser. The ingest CLI extracts PDF text, reads CSV as text, and permissively decodes unknown extensions. Extend these capabilities with one versioned ingestion and publication contract across the dashboard, CLI and SDKs.
Related: #142, transactional authority and ClickHouse retrieval, #25, temporal memory and knowledge retrieval. This proposal targets the Postgres+ClickHouse deployment in #142. It adds document and dataset semantics; it does not replace that issue's authority, readiness, isolation or recovery protocols.
Describe the solution you'd like
Storage and authority
| Store | Owns |
|---|---|
| Postgres | Source identities and revisions, collection membership and access policy, immutable manifests, schema approvals, active serving pointers, jobs, idempotency and audit state |
| Object storage | Original binaries, normalized extraction artifacts, table snapshots and any page/figure artifacts, addressed by immutable version and verified content hash |
| ClickHouse | Search chunks and metadata, all stored embeddings and vector execution, lexical retrieval, and typed dataset serving tables |
Postgres and ClickHouse remain the only query databases. Object storage is an explicit additional content substrate for this deployment, with its own access, durability, retention and erasure obligations. Large binaries and full extraction artifacts do not pass through Postgres rows, CDC or the JSON/base64 request path. Use ClickHouse materialization for datasets whose serving latency requires it; object-store scans are an evaluated option for infrequent or large cold queries, not the default for every interactive request.
Keep one authority for each source revision and active generation. Search documents and dataset rows are rebuildable projections of immutable source artifacts. WalShadow may carry selected catalog/history events under #142; parsing and enrichment do not wait for the CDC mirror, and raw file bytes are not replicated through it.
Reuse knowledge documents, collection access, search, outline, synthesis and download interfaces. Do not call memory.set for every chunk: source evidence does not require a belief-admission verdict per passage. Any later extraction of memory claims keeps exact document dependencies and derived provenance.
Tenant isolation and placement
#142 owns Postgres-provider selection, shared shards, dedicated tenant services, routing and authority epochs. This feature uses that contract; it does not create a database for every document, dataset, namespace or user. A tenant is a customer organization, with an explicitly registered workspace placement where its contract requires one. Users and agents share authorized knowledge through existing namespace/collection grants; personal documents remain private through those grants and row scope.
A dedicated tenant Postgres service does not imply a dedicated ClickHouse service or object bucket, and it does not replace their isolation checks. Bind every source, dataset snapshot, query plan, artifact reference and job to its tenant/environment scope and current placement epoch. Resolve routing from the authenticated scope rather than a caller-supplied database endpoint.
Xata branching is evaluated in #142, not selected as a dependency here. A document/data sandbox must register a consistent branch manifest: authorized read-only baseline artifacts and serving generations, separate writable ClickHouse/object destinations and jobs, and current authorization at disclosure. Forking only Postgres must not let ingestion, query execution or copied delivery jobs modify production. Include branches in erasure, retention, garbage-collection references and restore checks; a fork cannot resurrect erased documents or revoked access.
Source identity and normalized representation
Retain source_uri as the public logical key and bind it to a stable internal source ID within its org, namespace and collection context. Upload filenames and mutable object locations are not the sole identity. A source revision names the original object version and hash. Parsing or model changes create new extraction/serving generations without pretending that the original source changed.
Preserve the full identity tuple, authenticated actor, source class, uploader and retention/erasure ownership on every applicable row. The server derives provenance; extracted source content cannot promote itself to an asserted or verified fact. Record source timestamps, receipt time, extraction time and publication time separately. Full valid-time/knowledge-time queries remain the separate contract in #25.
Each extraction artifact contains ordered blocks with type, text, parent/section context, structural relationships and source locators. Tables also retain schema, headers, units, row/cell addresses and typed values. Link textual descriptions and tables to the same source revision. Generated summaries and figure descriptions carry derivation metadata and are distinguishable from source text.
Every searchable chunk refers to its source revision, extraction generation, block IDs, embedding-input hash, model/configuration and source locator. Locators include page and bounding box, document section, slide, sheet/cell range, immutable dataset row/key or SQL line span/TOML key path as appropriate. A citation identifies the source revision as well as its location; a chunk index alone cannot remain meaningful after replacement.
Extraction by format
| Input | Representation and required behavior |
|---|---|
| SQL | Parse using a declared or detected dialect. Preserve statement boundaries, comments, identifiers, definitions and dependencies with original line spans. Search scripts as evidence; never execute uploaded scripts as ingestion instructions. Unknown dialect or syntax reports lost structural capability. |
| TOML | Parse types and full key paths while retaining sections, arrays, comments and original line references. Combine exact identifier/value lookup with semantic descriptions. |
| Extract reading order, sections, page coordinates and tables; OCR scanned pages when needed. Preserve extraction coverage and page-level failures. A missing table or empty OCR result cannot silently become a fully indexed document. | |
| DOCX and PPTX | Preserve headings, paragraphs, lists, tables, slide order and available notes, with stable locators into the original revision. Advertise figure/chart interpretation only after its own evaluation. |
| XLSX | Preserve sheets, table regions, multirow headers, merged-cell context, formulas, cached values, dates, units and cell errors. Record the hidden-sheet/row inclusion policy. Do not assume one sheet is one rectangular table. |
| CSV and Parquet | Stream typed rows and schema into immutable dataset snapshots. Record encoding, delimiters, null rules, decimals, timestamps/timezones and row counts. Index schema descriptions and useful textual fields without requiring an embedding for every numeric cell or row. |
| Markdown, HTML and plain text | Preserve headings, lists, code blocks and available source offsets through the same normalized representation. |
Use deterministic format parsers for structured files and evaluate a layout-aware document parser as the office/PDF extraction baseline. Select parser versions and OCR/layout models from representative fixtures, not a supported-extension list alone. Retain explicit overrides for ambiguous delimiters, encodings, SQL dialects and spreadsheet table boundaries.
A spreadsheet parser can return a saved formula value without recalculating it. Preserve that distinction: numerical queries use the declared saved-value policy, and requests requiring fresh recalculation report unsupported or unavailable unless a separately qualified calculation service exists. Never execute macros, external workbook links or uploaded SQL. Legacy DOC/XLS conversion, arbitrary spreadsheet recalculation, audio/video and unrestricted chart reasoning are outside the first release.
Asynchronous ingestion and publication
- Authorize upload or a connector selection, enforce limits, and allocate a scoped upload identity. Stream bytes to a private immutable object. Verify size, checksum and object version before finalizing; mutable source URLs are not reproducible snapshots.
- In a short Postgres transaction, register the source revision, expected previous revision, idempotency state, required jobs and audit event. Object upload and database commit are not atomic: leased staging objects and a bounded orphan sweep cover failures between them without deleting referenced or in-flight content.
- Isolated workers parse into normalized artifacts, validate coverage and types, identify tables, and create structural chunks. Split oversized sections with parent context and token bounds; retain table headers with row groups. Do not turn every file into a single vector or split all formats into identical character windows.
- Batch embedding work, reuse only identical embedding inputs under the same model/configuration, and publish versioned ClickHouse chunks and any typed table snapshots. Validate row counts, conversion errors, required lexical statistics and reader-visible readiness.
- Compare the source/authority/erasure generations, then atomically publish the Postgres serving manifest when its required projections and artifacts are available. A failed replacement leaves the earlier published revision intact and reports the newer revision as pending or failed. Strict latest-version dependencies never silently fall back to that older revision.
Use source states such as uploaded, processing, ready, partial, failed, deleting and deleted, with durable stage progress and typed failure reasons. Track retrieval readiness and structured-query readiness separately; a workbook can be searchable while a table needs schema clarification. Partial publication is explicit and opt-in, including omitted pages, tables and rows.
Proposed API contracts, using the existing operation registry and transports:
- A streamed upload/finalization endpoint returns a scoped
upload_idafter byte/hash verification. The server resolves its object reference; callers cannot substitute an arbitrary storage key. knowledge.ingestaccepts that upload or an authorized connector reference,source_uri, expected revision, parser options, required capabilities and an idempotency key. It returns an ingestion ID, source revision and pending state.knowledge.ingest_statusreturns stage progress, coverage, requested and published revisions, warnings and per-capability readiness. Cancellation is an explicit action on that ingestion identity.knowledge.queryexecutes a validated typed plan over pinned dataset snapshots and returns typed results, calculation receipts and completeness. It requires no generative-model call. Extend existing synthesis/ask orchestration to plan and combine evidence where requested.
The API examples and exact wire fields are part of the first workstream; preserve existing text-op responses, expose file capability/status in CLI and SDKs, and keep MCP exposure within its listing budget.
Jobs are resumable and idempotent by source revision, stage and configuration. Fence overlapping claims, reject stale publication, reconcile ambiguous inserts and apply backpressure. Cancellation after source registration has an explicit outcome; it cannot imply that committed source data never existed. A status operation exposes counts, warnings, active and requested revisions, and retry state without leaking another principal's objects.
Tabular semantics and physical layout
Give each extracted table a logical dataset identity, an approved or validated schema and an immutable snapshot manifest. Preserve original and normalized column names, identifier columns, types, units, locale, formula-value basis and source row/cell mapping. Uncertain type conversion or an ambiguous table boundary prevents complete numerical serving until resolved; rejected rows are counted and surfaced, never silently coerced to zero or dropped.
Use stable typed serving tables across compatible dataset versions, keyed by tenant, dataset, snapshot and row identity. Bound the number of physical tables and columns; do not automatically create a table or partition for every upload, sheet or file revision. Pool small compatible schemas where useful and give large datasets dedicated placement. Schema changes create an explicit schema generation and mapping, not an accidental reinterpretation of earlier rows.
Dataset summaries and schema embeddings help discover the right table. They are routing aids, not substitutes for querying all rows needed by an aggregate. Text-bearing rows may also receive semantic chunks where row discovery is part of the use case. Large Parquet imports use bounded reads and projection/filter pushdown where supported; importing and querying cannot require loading the entire dataset into API memory.
Retrieval, structured queries and answers
Provide explicit retrieval and structured-query contracts, with an answer workflow able to combine them:
knowledge.searchremains a retrieval operation without a required generative-model call. Extend it with structural filters and revision-qualified locators. Fuse exact identifier/key matches, lexical evidence and vector candidates; use parent-section/table expansion within a bounded context budget.- A structured-query operation accepts a bounded typed plan over registered logical datasets. The server resolves approved schemas and snapshot IDs, validates the plan and compiles parameterized read-only ClickHouse SQL. Optional natural-language planning proposes this plan; it does not acquire database privileges.
- An answer workflow can retrieve narrative evidence, execute a validated calculation, or combine both. If table/column meaning, joins, units or time boundaries are ambiguous, request clarification or abstain. Do not invent joins from similar column names or answer an aggregate from top-k retrieved rows.
Authorize every dataset and source version before execution or external model dispatch. Apply tenant, collection and any required row/column restrictions inside scans and joins before aggregation. Post-filtering an aggregate cannot undo disclosure. Restrict execution to the registered dataset catalog and approved operators; model output cannot query system tables, credentials, arbitrary SQL functions, external URLs or object-store table functions. Workers may use approved object-store reads under service-controlled credentials; generated user plans cannot supply arbitrary locations. Scan/time limits fail a calculation rather than return a partial aggregate as complete; limited previews have a separate, explicit contract.
For semantic retrieval, scope all candidate legs, validate current source revision and permissions in a Postgres batch, refill rejected candidates within budget, and rerank only authorized evidence. Propagate failed required legs, readiness gaps and exhausted refill/context budgets as incomplete or unavailable outcomes. The answer model cannot describe partial coverage as complete evidence. Follow #142's finite visibility barriers and revalidate access at the required disclosure check points.
Pin the source snapshots and compatible serving generations for each request, including all joined tables and narrative documents. A numerical citation includes source/snapshot IDs, the validated query or typed-plan receipt, parameters, filters, units and relevant source locators. Large aggregates cite their input scope and calculation rather than pretending that one nearby cell proves the total. Store content-bearing receipts only under an explicit retention policy.
Revalidate access before releasing structured-query results, downloads or resolved citations, including when a grant changes during a long calculation. Source content is untrusted model input, not tool instructions; retrieved text cannot authorize query execution or change access policy. Partition caches by authorized scope, source snapshots and processing/model generations, and invalidate them on relevant revocation or erasure.
Scalability, latency and operations
Reuse #142's tenant placement and authority epochs. Keep authoritative metadata, serving queries and workers in the source's allowed region. Separate CPU parsing, OCR/layout inference, embeddings, interactive retrieval, structured execution and rebuild capacity. Use per-tenant quotas, bounded queues, fair scheduling and provider concurrency limits so a large PDF or Parquet import cannot occupy all interactive capacity.
Batch by rows, bytes and bounded deadlines. Publish directly to ClickHouse without a CDC dependency, and avoid accumulating application and async-insert waits. Materialize frequently queried datasets and keep evaluated search working sets warm. The exact-versus-approximate retrieval decision follows #142 and measured filtered quality; no universal ANN index is assumed.
Bound upload size, decompressed size, pages, sheets, rows, columns, parse time, memory, scanned bytes and query/context output. Validate format from content as well as filename. Run parsers without ambien 8BD0 t credentials or unrestricted network access; connector fetches use controlled destinations and redirect policies. Corrupt, encrypted, unsupported or over-limit inputs return actionable states rather than permissive binary-to-text ingestion.
Use one parser artifact contract across CLI, SDK and dashboard. Browser extraction may provide a preview, but it cannot silently determine a different published representation. Existing text submission remains valid with declared client-extracted provenance and coverage; it must not claim that Loom verified an unprovided original.
Retention, erasure and rollout
Retain every original and extraction artifact required by active generations and writer-authorized history. Garbage collection uses durable references and fenced upload/query/build leases. Hash equality is not permission to share content across tenants; preserve ownership references for scoped erasure and keep unqualified cross-principal blob sharing out of the first release.
Deletion immediately suppresses the source through Postgres authority. Erasure fences and drains publishers before removing originals, normalized artifacts, extracted tables, vectors, lexical/graph projections, caches, temporary files and retained job/receipt payloads. Register every content substrate in the erasure inventory and verify residue. Apply deletion evidence before replaying restored jobs or opening restored object versions for serving; document object-version and backup expiry obligations.
Build new parser/model/schema generations alongside the published one, evaluate and reconcile them, then switch the serving manifest. Requests pin generations; retire them only after readers drain and retention permits it. Rollback cannot restore an erased source or a stale permission decision.
For existing originals stored in ClickHouse, copy and hash-verify the retained bytes, record the object reference, validate downloads and citations, then switch each manifest. Keep one selected authority and writer per artifact path during migration. Removing the replaced path requires reconciled references and an explicit deployment/SDK compatibility plan; do not introduce a permanent per-request fallback between content stores.
Delivery and acceptance
Implement in workstreams that can be reviewed separately:
- Source/artifact/locator and capability contracts, parser fixtures, and ingestion/status API examples; document the dependency on Move live memory to Postgres and keep vector search in ClickHouse #142's dedicated deployment.
- Streamed upload, object storage, Postgres manifests/jobs, isolated parsing and full erasure coverage.
- Versioned structural retrieval for SQL, TOML, PDF/OCR, DOCX/PPTX and existing text formats, with citations and knowledge-interface integration.
- XLSX/CSV/Parquet normalization, typed snapshot publication, schema clarification and bounded structured-query execution.
- Combined answers, parser/model qualification, migration and load/failure qualification. Advertise each format and query capability only after its gates pass.
Before qualification, declare numeric quality floors and latency/capacity SLOs for the intended deployment. Compare the current knowledge path with the proposed parsers, chunking and retrieval on the same source revisions and queries. Highest quality is the objective, not a claim established by choosing a database or parser.
| Gate | Required evidence |
|---|---|
| Extraction and grounding | Labeled fixtures cover native and scanned PDFs, columns, rotated pages, tables spanning pages, DOCX headings/tables, slide order, SQL dialects, TOML paths, merged spreadsheet headers, formulas/cached values, CSV quoting/encodings and Parquet nested/decimal/time types. Measure coverage, table/cell accuracy and locator correctness separately by format. |
| Retrieval and answers | Evaluate filtered candidate recall, relevance/ranking, exact identifiers, multilingual and table questions, citation validity, context completeness, numerical correctness and abstention. Set quality floors and comparison margins before testing; an aggregate score cannot hide a failed format or tenant slice. |
| Structured execution | Reference queries verify filters, joins, totals, nulls, units, decimal precision, temporal boundaries and saved-formula semantics. Query receipts identify exact snapshots. Unauthorized rows cannot affect aggregates or counts. |
| Isolation and erasure | No observed cross-tenant or cross-environment disclosure through search, SQL, catalog, downloads, caches, citations or errors, on shared and dedicated placements and sandbox branches. Revocation and erasure racing workers, queries, model dispatch, copied jobs and restore cannot republish or expose prohibited content. Any occurrence blocks release. |
| Publication and recovery | Fault injection covers orphan uploads, duplicate/out-of-order jobs, poisoned batches, source replacement, stale leases, model/schema cutover, provider failures, object-store outages and failover/restore of both databases. Earlier published revisions remain intact; incomplete replacements are explicit. |
| Sustained scale and latency | Mixed small-document, large-file, import, OCR and structured-query traffic meets declared ingestion and interactive p95/p99 SLOs with bounded queues, database connections, object requests, parts/merges and memory. Report provider-inclusive time, errors, rejected work, catch-up capacity and storage/write amplification. |
Update wire models, generated SDKs, MCP exposure, CLI, dashboard progress/citations/downloads, manifests, migrations, settings references, erasure census and architecture/operations docs together for the surfaces each implementation changes. Keep fast retrieval available independently of optional generative answering.
Describe alternatives you've considered
- Embed flattened text from every format. Useful as an explicit text-only capability, but it loses table structure, types and source coordinates and cannot establish complete numerical results.
- Use another vector database. File-format support still requires extraction and query semantics. Keep vectors in ClickHouse under Move live memory to Postgres and keep vector search in ClickHouse #142; evaluate its actual filtered retrieval and table-serving workload before adding a storage system.
- Store large binaries in Postgres or keep every upload in ClickHouse. Existing ClickHouse originals remain migration inputs. The proposed path separates large immutable artifacts from transactional metadata and retrieval working sets, with object storage treated as a real erasure and durability dependency.
- Always query files remotely. Avoids materialization for cold data but adds source/object-store latency and availability to serving. Use pinned snapshots and measure that path; materialize when required by the SLO.
- Let an LLM execute arbitrary SQL or spreadsheet code. A typed, validated query plan over authorized snapshots provides a bounded execution and provenance contract. Full spreadsheet recalculation is a separate feature.
- Make all file processing synchronous in
knowledge.add. Parsing, OCR and large dataset loads need resumable jobs. Preserve text-op semantics and expose the file workflow's pending, partial and failed states explicitly.
Where the need comes from
The need is mine, an AI tool helped shape the proposal
Additional context
This is a feature and architecture proposal. No new parser-quality, retrieval or load benchmarks were run, and no end-to-end implementation was reproduced. Current-path observations were checked against main at 3c2196954a5728b7ff5af3426f51b0f62903d506:
- Knowledge operations and current ingestion contract.
- Browser PDF/DOCX extraction and original handling, CLI format extraction.
- Current chunking and publication, original-file storage.
External implementation references: spreadsheet formula versus saved-value behavior; ClickHouse object-store file queries. These establish available building blocks, not Loom's end-to-end quality or performance.
Activity
An extractor service parses SQL, TOML, Markdown and HTML, and the ser…
An extractor service parses SQL, TOML, Markdown and HTML, and the ser…
The document extractor ships as its own image, beside the server in c…
The document extractor ships as its own image, beside the server in c…
The document extractor ships as its own image, beside the server in c…
The document extractor ships as its own image, beside the server in c…
Uploaded files are ingested: the extractor service extracts them and …
Uploaded files are ingested: the extractor service extracts them and …
Uploaded files are ingested: the extractor service extracts them and …
An ingested file's tables are typed datasets a writer can approve
An ingested file's tables are typed datasets a writer can approve (#266)
The reference extractor reads Parquet as a table, its decimals exact
The reference extractor reads Parquet as a table, its decimals exact
The reference extractor reads Parquet as a table, its decimals exact
23 remaining items
Under Postgres, knowledge ingestion jobs live in loom.knowledge_jobs,…
Under Postgres, knowledge ingestion jobs live in loom.knowledge_jobs,…
Under Postgres, the dataset catalog lives in loom.knowledge_datasets,…
Under Postgres, the dataset catalog lives in loom.knowledge_datasets,…
Under Postgres, the dataset catalog lives in loom.knowledge_datasets,…
Under Postgres, the dataset catalog lives in loom.knowledge_datasets,…
Under Postgres, the dataset catalog lives in loom.knowledge_datasets,…
Under Postgres, the dataset catalog lives in loom.knowledge_datasets,…
Under Postgres, the dataset catalog lives in loom.knowledge_datasets,…
Under Postgres, the dataset catalog lives in loom.knowledge_datasets,…
Under Postgres, the dataset catalog lives in loom.knowledge_datasets,…
Under Postgres, the dataset catalog lives in loom.knowledge_datasets,…
Under Postgres, the dataset catalog lives in loom.knowledge_datasets,…
Under Postgres, the dataset catalog lives in loom.knowledge_datasets,…
Under Postgres, the dataset catalog lives in loom.knowledge_datasets,…