8000 feat(search): find what a pipeline's model step wrote about an asset by adeelahmad · Pull Request #683 · adeelahmad/lense · GitHub
[go: up one dir, main page]
Skip to content

feat(search): find what a pipeline's model step wrote about an asset - #683

Merged
adeelahmad merged 1 commit into
mainfrom
claude/project-thread-xfmkzn
Oct 19, 2026
Merged

feat(search): find what a pipeline's model step wrote about an asset#683
adeelahmad merged 1 commit into
mainfrom
claude/project-thread-xfmkzn

Conversation

@adeelahmad
@adeelahmad adeelahmad commented Oct 19, 2026
Copy link
Copy Markdown
Owner

Requested by Adeel · project thread

Description

Before: what a pipeline's llm step wrote about an asset (a summary, notes, a classification) was saved on the asset, but search couldn't find it.

After: search finds the words of the output an llm step saves on an asset, and those of a workflow's Save output node after an LLM prompt node. GET /search and MCP's search return them as source: generated, with the step's name in output_label, and the web app shows "Written by ". Running the step again replaces them, and deleting the asset removes them. A step keeps its output out of search when Find what it writes in search is unticked (searchable: false); it is on otherwise. The upgrade step generated-search makes outputs a model wrote before searchable too.

Motivation and Context

Item from the personal feature plan sheet (search what the model wrote).

Screenshots (if appropriate):

n/a

Steps to reproduce (if appropriate):

Run a pipeline with an llm step that writes a summary, then search for a word only in that summary: the asset comes up, marked "Written by" the step.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change; ! in the title)

Checklist:

  • The title is a Conventional Commit.
  • My code follows the code style of this project (make hooks checks it on commit).
  • I added or updated tests (tests/api/test_generated_search.py, tests/domain/test_migrations.py, __tests__/pipelines-templates.test.ts, __tests__/search.test.ts).
  • My change requires documentation updates, and I have updated docs/ accordingly (docs/processing.md, Pipelines and templates; docs/api.md; docs/database.md, Upgrades).
  • Users need more than the title to understand it, so I wrote it under ## Unreleased in CHANGELOG.md. Written as changelog.d/generated-search.md, per build(release): changelog fragments in changelog.d/ instead of one Unreleased section #581.

Done questions (AGENTS.md)

  • Goal: delivers what the user wanted, merged on green CI.
  • Agents: usable through the API, MCP or an assistant tool. GET /search and MCP search return generated hits.
  • Zero setup: no new prerequisite, .env edit or manual step.
  • Fewest clicks and keystrokes. Nothing to do; one checkbox on a step to keep it out.
  • Configurable: new settings in the web app with defaults; wizard asks only what is required. searchable per step in the pipeline editor, on by default.
  • Light: default install still runs on a Raspberry Pi with cloud models.
  • Private and secure: data local and encrypted, passkeys, threat model updated. n/a: no new surface; the output is found only where the asset is readable, like its other text.
  • Cost: cheapest path, model calls in the activity ledger, budgets respected. n/a: no new model calls; it indexes text already written.
  • User decides: consequential or new actions wait for the user; defaults unchanged. Adds hits only; a step can opt out.
  • Upgrades: existing installs upgrade safely (upgrade step for stored-data changes). Upgrade step generated-search indexes existing outputs, safe to run twice (test_migrations.py).
  • Honest: sources cited; labelled built, in progress or planned. README built list and docs/requirements.md name test_generated_search.py.
  • Tests, docs and generated files updated (make openapi).
  • Open standards; dependencies keep Lens MIT. n/a: no dependencies.
  • Result reported with evidence; "refine later" items written down.
  • Reuse: existing Lens code and permissive libraries reused, nothing reinvented. Uses the existing word index.

Local CI (ci/local-ci.sh): FastAPI embedded passed. FastAPI server had 1 failure in 1,980 (test_mesh_service.py::test_the_keeper_and_the_sidecar_bring_up_the_primary, a 502 from the mesh test server, which this change doesn't touch) while the container ran several times slower than usual. Next.js and pre-commit were stopped for time, so GitHub CI decides those.

Generated by Claude Code

@adeelahmad adeelahmad self-assigned this Oct 19, 2026
@adeelahmad
adeelahmad merged commit 5c33d81 into main Oct 19, 2026
12 checks passed
@adeelahmad
adeelahmad deleted the claude/project-thread-xfmkzn branch October 19, 2026 13:34< 756B /relative-time>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@adeelahmad adeelahmad

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@adeelahmad @claude
0