Repository navigation
Export and import memories and documents with verified recovery #159
Description
Component
Memory and search
Use case
Users need to download the memories and documents they are authorized to take with them, upload an existing memory collection into a selected destination, and recover memories and documents after accidental changes or infrastructure failure.
These workflows make sense for customer ownership, migration between deployments, reuse of reviewed agent/project knowledge, and recovery. Their guarantees differ:
| Workflow | Contract |
|---|---|
| Portable export/download | A documented, complete bundle of the selected authorized content and dependencies, usable outside the source deployment |
| Ordinary upload/import | New writes into a destination, with preview, current authorization, provenance, duplicate/conflict handling and normal memory admission |
| Backup and recovery | A privileged, tested restoration of trusted authoritative state and its required artifacts at a declared recovery point |
Quality is the highest priority. An import must not turn uploaded claims into verified local facts, overwrite protected memories, restore old grants, or report success while document originals are missing. A logical export must not be advertised as disaster recovery without a consistent recovery protocol and a tested restore.
The scope is product workflows and recovery coordination over existing capabilities, not a new database backup engine. Related: #142, authority and recovery across Postgres and ClickHouse, #157, do 8000 cument/artifact ingestion and storage, and #25, full temporal semantics. Coordinate with the in-progress Postgres authority implementation.
Describe the solution you'd like
Reuse existing capabilities and close their scope gaps
Current main already provides:
erasure.export: an audited admin operation returning scoped memory, knowledge chunks, conversations and other sections, with per-section limits and truncation markers. It is not a complete memory/document archive with original files.namespace.snapshotandnamespace.restore: curated memory, types, grants and audit-tail snapshots, with optional embeddings. Their privileged snapshot semantics are not the contract for arbitrary uploaded memory files.dev/backup_loom.pyanddev/restore_loom.py: portable table dumps with manifests and integrity checks. The documented dump is per-table rather than a cross-table application snapshot.- Native/managed ClickHouse backups for deployment recovery, individual knowledge downloads, and source-specific memory import commands.
Reuse the authorization, serialization, source adapters, artifact handling and backend backup mechanisms that fit each workflow. Extend the product surface rather than adding a second set of uncoordinated import/export rules. Keep the administrative portability endpoint's existing contract explicit; wrapping a capped response cannot establish a complete export.
Product flows and permissions
Add dashboard entry points for Download, Import and Backups, with matching CLI and SDK workflows. Each shows the selected org/namespace/collection, content classes, current-versus-history coverage, document-original coverage, estimated size and destination before work begins. A human-readable Markdown export can accompany the machine-readable bundle, but it is a presentation format rather than the lossless recovery artifact.
Derive scope from the authenticated caller. A personal download covers the caller's authorized personal content; namespace/collection exports require the corresponding bulk-export authority. Being an author, viewer or org administrator does not automatically make every private collection exportable. Preserve restricted-namespace and high-sensitivity checks, audit content exports, and define any new self-service permission separately from administrative exports. Do not weaken the existing erasure.export gate to expose a download button.
Import requires write authority on the chosen destination. Trusted backup recovery, access-policy installation and production cutover remain administrative operations. Never accept an uploaded org, user, agent or database endpoint as authority to select a destination. Tenant placement follows #142; a user-scoped export does not imply a database per user.
Use asynchronous jobs for large exports, imports and restores, with bounded progress, cancellation and stable receipts. Download completed bundles through an authorized endpoint that checks current access; revocation or erasure invalidates hosted artifacts and pending downloads. Customer-downloaded copies are outside Loom's ability to retract, which the workflow states explicitly.
Portable bundle contract
Define a versioned archive containing a manifest, UTF-8 JSONL content records and raw document attachments/artifacts. The manifest declares format/schema version, source deployment and scope identifiers, snapshot/cut identity, included classes and revisions, dependencies, counts, byte sizes, checksums, omissions and completion state.
First-release coverage is current durable memory heads, their supported type/semantic metadata, knowledge documents, canonical document content and available original files. Enumerate supported memory classes so profile/procedure content is not silently omitted. History, tombstones, evidence attachments and derived relationships require explicit coverage flags and an authorized selection. Working memory, conversations, tasks, code indexes and operational telemetry are outside that bundle profile unless separately supported and selected. Full temporal replay is not implied by retaining a source timestamp.
Memory records preserve source keys/IDs, type, content, confidence annotations, provenance, expiry, sensitivity and source links, with pin/lock/attestation metadata represented as source annotations. Document records preserve logical identity, source/extraction versions, locators, original content hashes and dependencies. Do not export merely top-k search results, summaries or chunk snippets and call them the user's complete memories or documents.
Exclude service credentials, token-signing/root secrets, runtime leases, delivery claims, idempotency caches and access grants from ordinary portable bundles. Backend recovery packages have a different privileged inventory, including a separately secured key-recovery procedure where required. Embeddings and other derived indexes are excluded by default; re-embedding/reindexing uses the destination's qualified model and readiness protocol. Optional derived artifacts carry their exact model, dimensions and configuration and are not assumed portable.
Checksums establish integrity, not trust. A user can alter both an archive and its checksums. Any trusted-bundle signature must validate the issuer and scope under an explicit trust policy; it cannot bypass target authorization, current erasure state or admission rules.
Complete and consistent export
Capture a finite logical cut and immutable source revisions before enumeration. Use backend snapshot/journal facilities, or scoped fenced quiescence when the backend cannot provide an immutable cut. A wall-clock timestamp, largest observed version, or sequential reads of changing tables do not prove a consistent bundle. Report the actual consistency capability for each supported storage profile.
Stream or page every selected record with a stable cursor bound to the cut. Pin referenced document/artifact versions during assembly and verify bytes and hashes. If a dependency is outside the caller's scope, report an authorized omission without disclosing its protected identity or content. Do not fetch new document revisions midway through assembly or copy chunks against a different original. Distinguish a text-only source whose original was never supplied from a stored original that is missing; the latter fails required-original coverage.
Publish the manifest as complete only after all required records and selected dependencies are present and verified. Caps, missing originals, interrupted reads and unavailable stores produce explicit partial/failed states. A downloadable partial diagnostic bundle cannot be used as a complete backup. Keep export pins bounded, release them after expiry/cancellation, and do not let pins override explicit erasure.
Previewed import and memory quality
- Upload into private staging using Search documents and query tables across office, code and data files #157's bounded artifact workflow. Validate archive structure, declared versions, hashes, sizes, record types and destination capabilities before admitting content. Reject path traversal, executable archive entries and unsupported formats. Never execute SQL/DDL, serialized jobs or instructions embedded in the file.
- Produce a preview bound to the bundle digest and an explicit source-to-destination mapping. Show creates, exact duplicates, key collisions, potential semantic conflicts, expired records, missing dependencies and rejected rows. Distinguish checks already performed from semantic verdicts that still need admission. Preview is dry-run by default.
- Default to adding eligible new records and skipping existing keys and exact duplicates. Stage conflicts for review. Overwrites or semantic mutations of existing content require selected actions and expected destination revisions; a replacement import never clears unmentioned destination content. Deletion/pruning is a separate, explicitly authorized operation.
- On apply, reauthorize and revalidate current target state. Route memory writes through the normal provenance, blacklist, conflict, pin, lock, compare-and-set, audit and idempotency paths. If admission would change records beyond the approved plan, stop that item and refresh its plan instead of silently widening the mutation. A preview is not permission to bypass a concurrent pin or grant change.
- Write source attribution as origin metadata while the destination records the actual importing actor, imported source class, new local revision and publication time. Preserve stated validity/expiry without inventing freshness; expired records do not become active by receiving a new timestamp. Uploaded confidence, pins, locks and verification annotations cannot install local trust or protections automatically. Authority-bearing attestations require independent verification of issuer, content and permitted scope.
- Import documents through knowledge ingestion and source-version publication, rebind authorized references through a persisted ID map, and report unresolved or omitted dependencies. Long Markdown notes remain documents unless an explicit reviewed extraction produces memory claims. Use existing adapters for supported external formats rather than claiming arbitrary memory dumps are interchangeable.
Track idempotency by import identity, source record/revision, destination mapping and selected action. Repeating a job resumes its results rather than creating duplicate facts, audit effects or documents. Authorization and erasure checks run before replaying a result. Partial success is itemized and resumable; cancellation leaves an explicit committed subset and never implies that committed writes were undone.
Report imported, skipped, conflicted, refused, failed and pending-index counts separately. A mutation committed but waiting for search is not a generic upload failure; follow #142's committed/pending receipt and visibility behavior. Refresh the preview if its bundle, mapping or required destination state changes.
Backups and recovery
Support Amazon S3 as a first-class backup destination configured and operated from Loom's dashboard, CLI and SDKs. Add other S3-compatible stores only after qualifying their required APIs and restore behavior. Both ClickHouse-only and ClickHouse+Postgres deployments are release targets; a ClickHouse-only upload test does not complete CH+PG backup support.
Loom coordinates the backup, tracks each required component and publishes one application recovery manifest. It delegates physical backup/WAL mechanics to qualified native tools or provider APIs rather than implementing a new database backup engine. Keep backup coverage profile-specific:
| Deployment | Recovery inventory |
|---|---|
| ClickHouse authority | Native or managed ClickHouse recovery points covering authoritative memory, history and documents/originals, plus separately secured required configuration/key material |
| Postgres authority (#142) | Postgres base backup/WAL or managed PITR; required ClickHouse history/audit archives; compatible projection checkpoints or rebuild sources; authority epochs and deletion evidence |
| Document/object storage (#157) | Retained immutable original, normalized artifact and dataset-snapshot versions referenced by the selected recovery manifest |
Reuse backend-native and managed backups. Replication, a copy-on-write branch, an analytics export and the per-table portable dump do not by themselves establish an independent application recovery point. An operator backup inventory must include every authoritative substrate; derived search indexes may be rebuilt only if all required content, history and configuration survive and recovery time meets the SLO.
A recovery manifest names compatible database checkpoints/WAL positions, required journal/archive coverage, source/artifact versions, schema/model versions and projection watermarks. Independent snapshots of Postgres, ClickHouse and objects taken at approximately the same time are not proof of consistency. Verify that history moved out of the source journal is present in the backed-up archive, and that every live reference can resolve to retained bytes. Keep cross-store reconciliation and authority cutover under #142's protocol.
The S3 destination contract includes bucket, allowed prefix, region, qualified endpoint where applicable, workload/IAM role or managed secret reference, encryption/KMS key reference, retention, schedule and restore tier. Validate the required write, multipart, read, listing and lifecycle permissions before enabling a policy. Keep credentials out of portable manifests, logs and user downloads. Use scoped worker identities; arbitrary query plans or uploaded manifests cannot supply an external backup destination.
Manual and scheduled runs expose per-component progress, byte counts, checksums, failures, the last complete application recovery point and any WAL/archive gap. In a CH+PG deployment, capture Postgres base-backup/PITR evidence and required WAL coverage, compatible ClickHouse history/audit backup coverage, and every referenced document/artifact version. Existing source objects count only when their retained immutable versions satisfy the backup's independent recovery and deletion policy; a pointer to a mutable production object is not a backup. The final manifest records compatible checkpoints and object versions rather than assuming simultaneous snapshots.
For self-managed storage, qualify native ClickHouse S3 backup/restore and a Postgres base-backup/WAL adapter such as pgBackRest. For managed services, use their supported backup/restore APIs and discover customer-bucket support explicitly. Provider-owned PITR is not automatically a customer-owned S3 backup. If a managed provider cannot expose required backup artifacts or recovery operations, show that capability as unsupported; a logical export cannot be relabeled full native PITR. Qualify at least the advertised CH+PG combinations before declaring that profile supported.
Keep physical backup scope separate from tenant export scope. A shared Postgres cluster or ClickHouse table can contain multiple tenants: never send its full physical backup to one tenant's bucket. Shared-placement tenant destinations receive only authorized logical recovery bundles with their declared consistency and coverage. Full physical repositories remain operator-controlled unless the backed-up placement is exclusive to that tenant. A bucket prefix alone cannot filter another tenant's rows out of a database backup.
Publish a recovery point as complete only when all required components and dependency chains are present and integrity-checked. A failed Postgres component, missing ClickHouse archive or missing document version leaves the application backup incomplete even if some uploads succeeded. Make retries resumable and idempotent, clean only the run's own unreferenced staging objects, and keep backup-complete distinct from restore-verified status and the last successful restore drill.
Bound backup/restore resource use so jobs do not consume interactive capacity. Configure and test a recovery destination outside the production failure/deletion boundary; state the regional-disaster coverage rather than assuming all backups cover it. Retain decryption-key access for the declared recovery window, and coordinate S3 lifecycle rules with base/incremental backup and WAL dependencies. Optional Object Lock or archival tiers require explicit retention/erasure and restore-time policies; neither may silently defeat the advertised recovery or deletion contract.
Restore into an isolated, non-serving target first. Trusted recovery preserves validated source semantics and attestation material, rather than re-admitting each original fact as a new imported claim; ordinary upload cannot invoke this path. Validate schema compatibility, keys, hashes, references, protected states and required history. Disable restored credentials/jobs from invoking external effects until identity, delivery and environment state are reconciled. Do not restore obsolete grants, revocations or authority routing into an active deployment merely because they occur in an old snapshot.
Before serving, apply current deletion/erasure evidence through the recovery point and subsequent retained period, invalidate obsolete generations, rebuild or reconcile projections, and prove required reader visibility. Recovery data missing the erasure reconciliation evidence remains quarantined. Restore only at the registered tenant/placement boundary and use a fenced cutover; never overwrite a live tenant or another tenant's rows as an incidental restore action.
Verify recovery through actual reads, source downloads, citations, history and representative retrieval, not only row counts or an accepted backup command. Record each drill's coverage and measured recovery time. Rebuilt indexes cannot promise the original ranking if the old model/configuration is unavailable; report and qualify the resulting capability before advertising equivalent recovery.
Erasure, retention and trust boundaries
Inventory hosted exports, upload staging, preview reports, bundle attachments, job errors, temporary extraction files, restore targets, branches, backup copies and content-bearing receipts alongside the memory/document stores. Access changes and erasure fence publishers and revoke available hosted downloads. Apply writer-authorized retention and disclose backup expiry or cryptographic-erasure obligations; distinguish live-data erasure completion from retained recovery-media obligations.
Keep deletion evidence independently recoverable so restoring an older data backup cannot also roll back the only record of subsequent erasures. An old bundle or restored job must not silently resurrect erased identities. A deliberately authorized new assertion is a new write under current policy, not an implicit restoration of old permissions or provenance. Customer-controlled offline files are not part of a service-side deletion guarantee.
Delivery and acceptance
Implement as related workstreams in this issue:
- Define coverage profiles, bundle schema, permissions, import actions and backend consistency/recovery capabilities; document existing endpoints/tools and their limits.
- Complete scoped export jobs, integrity-checked bundles and authorized downloads, including document originals and a dashboard/CLI workflow.
- Staged upload, preview, reviewed mappings/actions, resumable import and destination admission/readiness integration.
- S3 destination management, scoped manual/scheduled backup orchestration, CH+PG recovery manifests, quarantine restore and erasure reconciliation for each advertised storage profile.
- Automated round-trip, fault, isolation and restore drills; publish supported coverage and measured recovery objectives.
Define numeric size, concurrency, latency and recovery budgets before qualification. No provider benchmark or destructive restore is required to file this proposal.
- Exported canonical content, semantic metadata, document bytes, locators and dependency hashes match the selected source cut; allowed portability transformations are documented and tested.
- Concurrent edits, source replacements, deletion and permission changes cannot mix revisions or leak content. Exports above existing section limits finish by streaming/paging or report incomplete.
- Repeated, interrupted and partially applied imports resume without duplicate logical effects. Preview/apply races, wrong scopes, forged annotations, unsupported formats, corrupt/missing attachments, expired data and malicious archives are covered.
- Normal imports honor blacklist, conflict, pinned/locked-peer and compare-and-set behavior; untrusted files cannot import grants, credentials, local verification or unrestricted actor identities.
- Document originals, full content and citations survive round-trip; unresolved dependencies are visible. Reindexing and model changes have explicit readiness and quality results.
- S3 backup and restore drills cover both ClickHouse-only and advertised ClickHouse+Postgres/object-storage profiles. Verify base/incremental and WAL chains, archive/journal overlap, document bytes, missing components, credential/key rotation, lifecycle expiry, interrupted uploads and cross-store gaps. A failed required component cannot leave an application backup marked complete.
- Tenant-owned destinations from shared placements contain no other tenant rows, database credentials or operator-only recovery state. Native physical backup artifacts are restricted to the authorized placement boundary; scoped logical bundles are labeled as such.
- Erasure/revocation applied after a backup remains effective after restore, branch activation and import retries. An unauthorized or incomplete recovery target cannot serve content or run copied jobs.
- A tenant/user cannot export, preview, import into, restore over or infer another tenant's/private collection data through artifacts, errors, status or reused job IDs.
- Report completion/error counts, streamed throughput, queue age, interactive p95/p99 impact, storage overhead and measured recovery time; the UI distinguishes a portable bundle from a qualified recovery point.
Update the operation registry, wire models and generated SDKs, MCP exposure, CLI, dashboard, settings references, erasure census and operational/architecture docs for the surfaces each workstream changes. No public interface should imply that the existence of an export button proves disaster-recovery coverage.
Describe alternatives you've considered
- A Download button wrapping
erasure.export. Reuses an existing scoped export but retains section caps and lacks a complete original-document/dependency bundle. Keep its administrative semantics and implement complete enumeration and coverage explicitly. - Upload JSON directly into tables or pass it to namespace restore. Bypasses import admission and can install source identities, grants or trust metadata under the wrong authority. Ordinary imports create reviewed destination writes; trusted recovery is separate.
- Reconstruct memory by asking an LLM for a summary. Useful as a human-readable view, but loses original claims, metadata and evidence and can invent facts. It is not the lossless export.
- Export every embedding. Adds size and model coupling. Preserve source content and rebuild by default; optional exact derived artifacts require a declared compatible profile.
- Use analytics exports, replicas or branches as the backup feature. Their coverage, failure boundary and consistency differ. They count only where a recovery protocol and restore drill establish the promised outcome.
- Build a custom physical backup engine. Backend-native recovery already exists. Loom needs coverage manifests, coordination, authorization, product workflows and tested recovery across its content substrates.
Where the need comes from
The need is mine, an AI tool helped shape the proposal
Additional context
This is a feature proposal based on inspection, not a report of a reproduced runtime failure. No live export, import, backup or restore was run and no new quality/load benchmarks were performed. The relevant paths were inspected in the checkout and confirmed unchanged on main at fda33a7c98345fae55db670d9bb004d4485aa707.
Source anchors:
- Existing backup, dump, snapshot and recovery procedures, analytics export scope.
- Export operation and limits, export authorization, exported content.
- Namespace snapshot/restore, existing Markdown-memory importer, knowledge content and original downloads.
Backend references: Postgres base backup/WAL and PITR, ClickHouse backup/restore, managed ClickHouse backup review and restore. S3 implementation references: native ClickHouse S3 backup/restore and pgBackRest S3 repositories and WAL archiving. These do not establish a cross-store Loom recovery guarantee without the manifest and drill requirements above.
Activity