Repository navigation
Inspect memories, tune extraction and ask questions through a Loom assistant #160
Description
Component
Dashboard
Use case
Users need to ask Loom whether their agents use its features appropriately, understand why recall or ingestion behaves poorly, and ask ordinary questions over the content they can access.
Examples:
- "Did this agent read memory before answering during the last week? What can we actually prove?"
- "Are we storing documents, coordination state or structured records as memory claims? Show examples and the appropriate destination."
- "Why does this namespace have repeated writes, poor recall or growing storage cost?"
- "Show what this user, agent or session remembers, then propose better categories and extraction instructions from those examples."
- "What did we decide about this feature? Cite the memory revision and design document."
- "Which ingestion jobs are failing, and which document versions are searchable?"
This makes sense as a conversational layer over Loom's existing diagnostics, retrieval and observability. It should explain evidence and propose actions. Format compliance alone does not prove factual correctness, retrieval quality or agent efficiency; an LLM cannot establish these by inspecting database rows alone.
Describe the solution you'd like
Build a Loom assistant with read-only investigation first, supporting both ClickHouse-only and ClickHouse+Postgres deployments. Quality is the first priority; scalability and latency are measured release requirements. Keep the assistant independent of the chat client. Evaluate LibreChat as an optional client using Loom's MCP tools before committing to an embedded interface or a new required service.
Inspect, tune, apply
Expose three clear stages in the product:
| Stage | Behavior | Permission to change data |
|---|---|---|
| Inspect | Read authorized memories and their sources, history, categories and available creation settings for a selected user, agent or session | Read-only |
| Tune | Draft category, extraction-instruction and supported setting changes from the selected evidence, with before/after examples | Read-only proposal; no configuration changes |
| Apply | Show the exact diff, scope and effects, then commit only the actions an authorized person approves | Explicit approval and normal write authorization |
User, agent and session are filters within the caller's authorized scope, not new tenant boundaries. Define each filter precisely: an agent-authored row or a session-created row is different from all shared knowledge that agent or session could read. Show those views separately, including empty and incomplete results, rather than claiming that an authorship filter reconstructs everything an agent knows.
Tune should change the process that creates memories as well as recommend record repairs. Read the applicable namespace guide, registered categories, extraction instructions and effective settings when authorized. Propose evidence-based changes with supporting and counterexamples, expected effects and limits. A preview may compare extraction outputs over a bounded, reviewed sample without publishing them; show model cost and coverage. Do not optimize only for fewer memories or shorter text at the expense of important facts and source fidelity.
Existing operator extraction policy is readable and writable only by an organization administrator; keep that boundary and its precedence. Do not invent a persisted per-user or per-session setting where only an organization, namespace or request setting exists. A proposal must name the actual supported target, all affected consumers and whether it changes future extraction only. Re-extraction, recategorization or rewriting historical memories is a separate selected action with its own impact preview and authorization. Tuning cannot silently weaken an administrator's mandatory policy, change grants or modify storage schema.
Scoped chat and project configuration
The entry screen explains that users can inspect what a user, agent or session remembers, then propose changes to the categories, instructions and settings that create those memories. Include an explicit scope selector and a Review changes panel with pending proposals and their approval state. Scope comes from the authenticated user's allowed placements and collections; a scope selected in chat cannot widen permissions.
Offer these conversation starters using Loom's own terminology:
| Starter | Expected result |
|---|---|
| Show relationships between facts | An authorized, bounded graph view with cited source facts, edge provenance and validity; distinguish asserted, inferred and disputed relationships |
| What is in this project? | A scoped inventory of available memories, documents, records, runs, integrations and capabilities, with counts, freshness and incomplete sections labeled |
| Analyze one user's memories | A selectable authorized user view, evidence-backed findings, and optional tuning proposals; distinguish authorship from shared visibility |
| Who are the users in this project? | A permitted, paginated directory or explicitly labeled observed identities for the selected scope; memory subjects, authors and registered members are different populations |
| Configure this project | Inspect available data and effective configuration, identify a supported goal, and draft a reviewed configuration plan with examples |
Resolve project to an existing registered organization/workspace/namespace/collection boundary and show it plainly. Do not invent a new tenant identity or silently treat every project as an organization-wide view. A user directory requires the applicable membership/directory permission; reading memories alone is not permission to enumerate all users. Omit hidden identities and aggregates that reveal their existence, and do not render the unset identity sentinel as a real user.
Reuse authorized entity-graph operations for relationships. A graph connection does not establish causality, truth or permission to read its endpoints. Check both endpoints and supporting facts; a hidden node or supporting source must not leak through edge labels or path explanations. Starting to persist or monitor a new relationship is a proposed write/subscription with its own approval, not an incidental effect of inspecting the graph.
The guided configuration flow analyses authorized data, usage and effective settings to propose custom extraction instructions, categories, namespace guidance and supported retrieval or other project settings. State which inputs were inspected and which were unavailable; do not claim full project analysis from a few retrieved examples. Ask for the user's goal when it cannot be inferred from the selected workflow, rather than assuming that every project should use every feature.
Each plan names the supported configuration target, current value/version, proposed value, evidence, expected behavior, affected agents/users, cost or latency implications, validation results and rollback limits. The UI shows the exact diff and lets an authorized approver accept individual actions, reject them or revise the plan. No configuration or domain content changes while drafting. Approval is explicit and scoped to that plan; a conversational suggestion or a starter click does not approve it. The assistant cannot configure additional rights or tools for itself.
Configuration applies prospectively unless the approved plan separately selects reprocessing of existing data. Evaluate proposed instructions/categories against representative examples and counterexamples before recommending them, and preserve source fidelity, required policy and available capabilities. Do not claim that custom settings are optimal without task-specific quality measurements. Use the same approval, authorization, concurrency and audit contract as record repair; this is one workflow in this issue, not an independent chatbot or configuration subsystem.
Reuse the existing diagnostic layer
Start with namespace.audit, namespace.validate, namespace.summary, namespace.guide_get, mentor.brief, mentor.guide, authorized agent activity, cost and retrieval-trace operations. Existing dashboard audit/mentor panels and the namespace optimization procedure already cover parts of this workflow. Extend gaps in these shared operations rather than implementing a second rule engine inside a system prompt.
Separate three kinds of result:
| Result | Examples | Required treatment |
|---|---|---|
| Contract violation | A value violates a declared schema or verified provenance requirement | Server checks the applicable contract and returns evidence |
| Usage heuristic | Double-encoded values, documents as claims, excessive version churn, low search activity | Explain the rule, counts, exceptions and uncertainty; a warning is not proof of misuse |
| Semantic recommendation | Split a claim, choose another plane, change metadata or revise a contradiction | Cite the source and intended use; present a proposal requiring review |
Preserve existing finding codes and severity. Version new rules, scope them to the actual record type and namespace policy, and show skipped checks. Valid typed records, supported JSON values, exact-key workflows and intentionally unused planes must not be labeled incorrect merely because another representation is common. Distinguish a disabled feature, ingestion/index backlog, insufficient access and missing telemetry from agent misuse.
Return structured findings that the CLI, dashboard, MCP clients and chat can share. Each finding includes its rule/version, source operation, scope, observation window, computation time, count/denominator, sample limit, evidence references and revisions, coverage/skipped checks, proposed remedy and applicable uncertainty. Human-readable explanations must preserve those fields. A sampled report cannot claim complete coverage.
What agent usage evidence can prove
Distinguish a tool attempt, successful read, committed write, indexed write, returned retrieval hit, explicit client acknowledgment and task outcome. Operation counts alone do not show that a result reached the model, influenced its answer or improved the task. Report "retrieved" or "not observed in this window" when that is all the evidence supports.
Add missing telemetry only where a concrete question needs it: correlate existing request, session, agent and revision identifiers; mark gaps, retention limits, delayed events and sampling. Optional client/outcome evidence must declare its source and trust. Do not infer causal quality gains from a higher call count or present self-reported adoption as independent verification. Keep the assistant's own reads identifiable so its audits do not inflate the agent usage it reports.
Access to ClickHouse and Postgres
Use this access path:
Chat client -> Loom assistant/tool dispatcher -> authorized Loom operations -> backend adapters -> ClickHouse and/or Postgres
The backend has the database access needed to answer the question; the model receives bounded authorized tool results. Do not give the model, browser or shared chat agent database credentials or an unrestricted SQL tool.
- In ClickHouse-only deployments, reuse the existing authority and authorized retrieval paths.
- Under Move live memory to Postgres and keep vector search in ClickHouse #142, read current state, grants, policy and committed job status from the Postgres authority; use ClickHouse for retrieval, historical analysis and telemetry where its documented contract permits. Revalidate search candidates and cite the actual visible revision. An old ClickHouse row cannot override a newer Postgres revision, erasure or grant.
- Declare cross-store report checkpoints, projection/event lag and partial results. Do not invent a transactionally consistent snapshot from independently read stores. Reauthorize referenced content before returning it; missing authority or revocation evidence fails closed. A backend outage is an unavailable check, not a healthy finding.
- Follow registered tenant/placement routing. The tenant is the organization, with namespace, collection and user restrictions preserved. The feature does not require a database per user or a database per tenant.
- Add bounded typed diagnostics for missing questions. Let backend code compile allowlisted aggregates and query templates with server-supplied authorization predicates. Keep arbitrary model-generated SQL out of the initial feature. Any later SQL capability needs separate qualification for parsing, functions, external access, execution budgets and authorization before aggregation.
A platform operator's infrastructure diagnostics and a customer's content assistant have different permissions. Namespace access does not imply access to all organization telemetry, another user's private collection or platform logs. Apply authorization before aggregation, sampling and model context construction; a filtered final answer cannot repair an earlier data leak.
Ordinary user questions
Use the existing authorized search/ask and exact-record operations for memories, knowledge, code, tracker, conversations and runs where available. Route document/data questions through #157 when that capability is released. Exact counts and numerical questions use authorized structured queries rather than estimates from retrieved text chunks.
Answers cite stable record/document identifiers and revisions, pages or source locators, and observation time. Distinguish source statements, calculated results and model interpretation. Resolve conflicting evidence explicitly and abstain when evidence is insufficient. Recommendations do not mark underlying claims verified or upgrade imported/derived provenance.
Provide general conversation when it is useful, but clearly separate it from an answer grounded in Loom. External web search or other connectors are optional and separately authorized. General conversation must not trigger a database-wide scan. The assistant's chat history is not automatically durable memory: saving claims or documents uses the ordinary admission and provenance workflow.
Chat interface and LibreChat decision
LibreChat supports MCP tools, reusable agents and per-user authentication mechanisms, making it a candidate for the first integration. Ship a documented integration using a small read-only tool set and evaluate it against a native dashboard chat using the same Loom operations.
Qualify a pinned release for identity delegation, transport compatibility, tool errors, citations, streaming/cancellation, tenant boundaries, deployment dependencies, license and maintenance. UI resource sharing is not Loom data authorization. Every tool call must use the initiating user's validated identity and no broader access than that user currently holds; neither a shared service token nor a user ID supplied in a prompt is a substitute.
Keep Loom as the authority for memory, documents, retrieval and diagnostic rules. LibreChat's own persistence may hold UI/conversation state under a declared policy; it must not silently create a second canonical memory store or independently index copies of tenant content. Inventory its transcripts, tool outputs, search caches and attachments for retention, erasure and #159 recovery coverage. An optional chat deployment must not add its databases to the mandatory Loom storage architecture.
Reviewed repairs
The first release investigates and recommends. A later repair workflow presents exact affected records, before/after values, destination plane, expected revisions and effects for approval by an authorized user. Use the existing namespace optimization procedure, dry runs and normal write operations; preserve redirects and provenance where applicable.
Approval binds to the displayed plan and content/revisions. Apply rechecks permissions, current state, pins, locks, blacklist and erasure fences. Changed records require a refreshed plan. Report partial results and make retries idempotent. No direct table updates, automatic deletion, unreviewed conflict resolution or background rewriting of a namespace. A request to audit is not permission to repair.
Category, instruction and setting changes need the same protection: retain a versioned diff and audit record, validate effective scope and policy, and reject stale approvals atomically. Add optimistic concurrency to an underlying configuration operation if it lacks it before exposing conversational apply; a read followed by an unconditional write is insufficient. Provide reviewed rollback for configuration, with separate status for any historical data reprocessing that rollback cannot undo.
Quality, scale and latency
Run deterministic checks without an LLM and use bounded, resumable background scans or maintained aggregates for expensive reports. Cache reusable facts and aggregates by effective authorization scope, rule version, window and backend checkpoint. Recheck authorization on read and invalidate or fence revoked/erased source content, including cached narratives and stored reports. Use incremental scans only with declared coverage and periodic reconciliation.
Keep analysis and model orchestration outside memory write/search critical paths. Bound query time, scanned rows/bytes, results, model context, tool calls, retries and concurrent jobs per tenant. Use workload isolation and fair scheduling; stream progress, support cancellation, and send expensive investigations to asynchronous jobs. A small question should fetch its evidence without running a whole-tenant audit.
Honor the organization's model policy, approved providers, spend caps and disabled-model mode. Deterministic reports remain available without synthesis. Retrieved documents, guides and tool text are untrusted data: they cannot grant tools, change policy, redirect credentials or approve repairs. The initial tool set excludes shell/code execution and arbitrary outbound destinations.
Before release, set numeric targets for finding precision/recall, citation correctness, answer correctness/abstention, p95/p99 latency, time to first output, scan/model cost and maximum impact on concurrent ingest/search. Compare candidate clients/models on the same labeled cases and evidence. Publish the measured tradeoffs; do not trade answer correctness for lower latency without stating the capability limit.
Delivery and acceptance
- Inventory existing diagnostics, telemetry coverage and access contracts. Define missing structured findings and labeled evaluation cases.
- Deliver read-only inspection, tuning proposals and cited questions through shared Loom tools, with a qualified optional LibreChat integration or a native interface chosen from the evaluation.
- Add scheduled reports, scoped incremental analysis and alerts using the same rules and coverage contracts.
- Add approved configuration changes and reviewed record repairs only after investigation, isolation and concurrency gates pass.
- Double encoding, oversized claim blobs, stale/conflicting records, valid typed records, intentional exact-key reads, delayed indexing and unavailable telemetry produce the expected distinct findings. Report precision/recall against reviewed labels.
- Answers cite the actual visible source revision and distinguish facts, heuristics, missing evidence and recommendations. A successful read is never reported as proven task adoption without supporting evidence.
- ClickHouse-only and ClickHouse+Postgres qualification includes stale projections, conflicting revisions, cross-store gaps, authority failover and unavailable backends. Unsupported checks are explicit.
- Cross-tenant, private collection, restricted telemetry and revoked-access tests cover tools, aggregates, samples, citations, chat sharing, reports, caches and exports. Revoked or erased content cannot reappear through saved chat context or cached answers; it is revalidated before reuse.
- Adversarial memories/documents/tool outputs cannot widen access, invoke unapproved tools, expose credentials or authorize repairs. Client-supplied scope is checked against validated identity on every tool call.
- Labeled ordinary questions cover memory, documents, code and supported structured queries, with contradiction and abstention cases. Mathematical answers match the authorized query result.
- Load qualification measures multi-tenant fairness, model/query cost, p95/p99 impact on normal ingest/search, cancellation, queue delay and recovery from interrupted scans. Fix numeric budgets before qualification; no paid benchmark is authorized by filing this issue.
- Read-only mode remains useful with models disabled. Any repair requires an approved plan and passes stale-revision, protected-record, idempotency and partial-failure tests through ordinary operations.
- Inspect and Tune perform no domain or configuration mutations; previews have only declared temporary/job/audit effects. User/agent/session views distinguish authorship from visibility. Category, instruction and setting proposals name their real scope, affected consumers, evidence and future-versus-historical effects; Apply rejects an unauthorized, stale or widened plan. Historical reprocessing requires separate selection.
- Scoped conversation starters cover project inventory, user analysis, permitted user enumeration and fact relationships. Tests distinguish members from observed identities and shared visibility from authorship; hidden users, nodes, edges and supporting sources cannot leak through counts or explanations. Configure-project tests cover per-action review, changed settings after preview, partial application, rollback and separately approved historical reprocessing.
- Chat persistence and derived reports are included in erasure, retention and backup inventories. Integration dependencies, supported versions and backend capabilities are documented.
Update applicable operation definitions, generated SDK/MCP surfaces, CLI/dashboard, policy settings, architecture and operational documentation as each workstream ships. Do not advertise a proposed operation or client capability as already implemented.
Describe alternatives you've considered
- Existing audit/mentor panels and CLI: already answer deterministic questions; keep them. Chat adds investigation across findings and cited explanations rather than replacing their rule engine.
- A bot with direct CH/PG administrator access: bypasses application policy, exposes unrelated data and couples prompts to storage schemas. Server-owned typed tools preserve authority and budgets.
- An LLM that reads the database and assigns one efficiency score: lacks ground truth and hides missing telemetry. Return inspectable findings and task-specific measurements.
- Require LibreChat for every deployment: adds operational and persistence commitments before validating the workflow. Keep it optional until the client comparison establishes its fit.
- Automatically repair every warning: heuristics can flag legitimate designs. Use a reviewed plan through normal admission and concurrency checks.
Where the need comes from
The need is mine, an AI tool helped shape the proposal
Additional context
Feature proposal based on documented capabilities, not a reproduced runtime failure. No live tenant audit, database export, repair, client deployment or paid benchmark was run. The linked Loom documents were checked against main at the pinned revision.
Related: #142 (Postgres authority and ClickHouse retrieval/history), #157 (document and structured-data questions), #159 (portable bundles and recovery), and the in-progress storage implementation #158. This assistant depends on their released contracts without changing their authority boundaries.
Existing Loom capabilities:
- Namespace health and mentor
- Namespace audit and reviewed optimization
- Mentor evidence and partial-result fields
- Agent activity, usage and cost
- Existing dashboard audit and mentor surfaces
- Operator extraction policy scope, permissions and precedence
Client references: LibreChat MCP tools, agents, MCP authentication configuration, access control, and default deployment dependencies. These establish available integration mechanisms, not a verified Loom integration or permission-equivalence guarantee.
Activity
Diagnostic reports read memory at the Postgres authority's barrier an…
Diagnostic reports read memory at the Postgres authority's barrier an…
Diagnostic reports read memory at the Postgres authority's barrier an…
Diagnostic reports read memory at the Postgres authority's barrier an…
Diagnostic reports read memory at the Postgres authority's barrier an…
Diagnostic reports read memory at the Postgres authority's barrier an…
Diagnostic reports read memory at the Postgres authority's barrier an…
Diagnostic reports read memory at the Postgres authority's barrier an…
Under Postgres, knowledge.inspect says which document commits ClickHo…
Under Postgres, knowledge.inspect says which document commits ClickHo…
Under Postgres, knowledge.inspect says which document commits ClickHo…
Under Postgres, knowledge.inspect says which document commits ClickHo…
Under Postgres, knowledge.inspect counts the ingestion jobs ClickHous…
Under Postgres, knowledge.inspect counts the ingestion jobs ClickHous…
Under Postgres, knowledge.inspect counts the ingestion jobs ClickHous…