8000 feat(search): find what a pipeline's model step wrote about an asset by adeelahmad · Pull Request #683 · adeelahmad/lense · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -207,7 +207,8 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p
[evidence PDFs](docs/api.md#evidence-pdfs), [published feeds](docs/feeds.md);
[checks](docs/requirements.md#learn-from-it))
- **Automate it.** Pipelines and workflows on a canvas (with Convert steps that make a video's soundtrack an audio
asset and a document's pages image assets, and Read steps that pick their own converter, such as anytopdf), routines on a schedule, batch runs over a collection, and
asset and a document's pages image assets, and Read steps that pick their own converter, such as anytopdf; what their model steps write about
an asset is found by search, under the step's name), routines on a schedule, batch runs over a collection, and
notifications to webhooks, Slack, Discord and Matterbridge, including when a routine's run fails or leaves graph
changes to review. Everything that happens in Lens (new assets, emails,
calendar events, files, feed posts, what people say to Lens, sensor readings, sensors switched on or off, notes,
Expand Down
7 changes: 7 additions & 0 deletions changelog.d/generated-search.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
- **Search finds what a pipeline's model step wrote.** The words of the output an `llm` step saves on an asset (and of
a workflow's Save output node after an LLM prompt node) are found by search, the API's `GET /search` and MCP's
`search`, as `source: generated` with the step's name in `output_label`; the web app shows "Written by <step>".
Running the step again replaces them and deleting the asset removes them. A step keeps its output out of search with
*Find what it writes in search* unticked (`searchable: false`); it is on otherwise. The upgrade step
`generated-search` makes outputs a model wrote before searchable too
([Pipelines and templates](docs/processing.md#pipelines-and-templates)).
10 changes: 7 additions & 3 deletions docs/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -1300,7 +1300,8 @@ GET /api/v1/mentions
```

`GET /search?q=` finds the moments where the words appear (in an asset's text or transcript, on screen in a video, in a resource's
supplementary transcripts, captions, translations and indexes, or in what a model says its shots and pages show) in the
supplementary transcripts, captions, translations and indexes, in what a model says its shots and pages show, or in
what a pipeline's model step wrote about it) in the
namespaces you can read, best first: every word
(English stemming), "quoted phrases" as written, `OR` between alternatives, `NOT` or `-` before a word or phrase to leave out the lines
with it; `ns`, `speaker`, `emotion`, `event` (a sound heard, such as laughter, applause or music, typed any way; music
Expand All @@ -1319,8 +1320,11 @@ or on the longest of the words when the text doesn't have them together.
Each hit's `source` says where it was found: `said` (a line of the asset's text or transcript), `screen` (text on screen), `page` (a document's or an
image's text: its `page`, from 0, and its `box` on it), `object` (a kind of object, where it's first seen: its time, or
on a document's pages its `page`, with the `frame` and the `box` it's best seen in), `described` (what a shot or a page
shows, [Descriptions](#descriptions): the shot's time, or the `page`, and its `frame`) or `file` (a line of a
supplementary file: its `file`, `file_role` and `file_label`, and which `line`). A file's lines have no `t0` when the
shows, [Descriptions](#descriptions): the shot's time, or the `page`, and its `frame`), `generated` (what a pipeline's
`llm` step, or a workflow's Save output node after an LLM prompt node, wrote about the asset: the output's
`output_key` and what wrote it, `output_label`, with no time or page; [Pipelines and
templates](processing.md#pipelines-and-templates)) or `file` (a line of a supplementary file: its `file`, `file_role`
and `file_label`, and which `line`). A file's lines have no `t0` when the
file doesn't say when they are; the web app opens those in the resource's Files tab. A `speaker`, `emotion` or `event`
filter keeps to `said` hits, and each hit says its `emotion` and `event`.

Expand Down
4 changes: 4 additions & 0 deletions docs/database.md
Original file line number Diff line number Diff line change
Expand Up @@ -123,6 +123,10 @@ The steps so far, in the order they run:
without a `head` is read as one line, and saved as one before anything is added to it.
* `mcp-sealed-secrets`: the tokens and keys MCP servers kept in plain text in their headers and env are sealed,
in every version ([Adding an MCP server](assistant.md#adding-an-mcp-server)); sealed ones are left as they are.
* `generated-search`: outputs a model wrote about an asset before search found them (an `output` whose `origin` names
a `model`, except a spec's note) get their words kept for search in `generated`, under their template's name
([Pipelines and templates](processing.md#pipelines-and-templates)). Outputs that already have them are left as they
are, so it's safe to run twice.

To add a step:

Expand Down
9 changes: 7 additions & 2 deletions docs/processing.md
Original file line number Diff line number Diff line change
Expand Up @@ -814,7 +814,11 @@ audio skip themselves for imported text. Steps:
- `clean_audio` ([Cleaning up audio](#cleaning-up-audio)), `transcribe`, `diarize`, `clean`
([Cleaning up transcripts](#cleaning-up-transcripts)), `analyze`, `summarize`, and `report` (the built-in report).
- `llm`: renders a prompt template, sends it to the configured model, checks the reply against the template's JSON
Schema and saves it as a named output (e.g. `meeting_notes`).
Schema and saves it as a named output (e.g. `meeting_notes`). Search finds the words of that output (its text and
numbers, not its field names), as `source: generated` with the step's name (else the template's) in `output_label`
([search](api.md#search)); running the step again replaces them, and deleting the asset removes them. Untick *Find
what it writes in search* (`searchable: false`) to keep an output out of search; it is on unless a step says so,
which changes nothing anyone relied on, since search didn't look at outputs before.
- `spec`: writes a technical spec from the transcript into a note linked to the asset, with the Spec from transcript
template unless the step names another prompt template ([Specs from transcripts](notes.md#specs-from-transcripts)).
- `report` with a template: renders an HTML page into the namespace's reports. These pages may not run scripts.
Expand Down Expand Up @@ -899,7 +903,8 @@ node passes what it makes along its connections.
other methods. Asset and graph workflows have it; automations call web addresses with their own action, which asks
first ([actions](automations.md#actions-and-intents)). Refine later: credentials passed in through a custom node's parameter
aren't sealed, and a sealed header is bound to its host rather than to one workflow.
- **Keeping results:** Save output (`outputs.<name>`), Set field (a custom field, kept in the metadata history), Save
- **Keeping results:** Save output (`outputs.<name>`; when an LLM prompt node is before it, search finds its words
under the node's label, unless its *Find it in search* box is unticked, `searchable: false`), Set field (a custom field, kept in the metadata history), Save
entities, and a Classify node set to act on a sure answer (it sets the content type, suggests the topic or adds the
tag; a try on the canvas only says what it would do).

Expand Down
1 change: 1 addition & 0 deletions docs/requirements.md
Original file line number Diff line number Diff line change
Expand Up @@ -255,6 +255,7 @@ request, to another check that proves the same thing; loosening a requirement or
| Requirement | Checks |
|---|---|
| Pipelines and workflows on a canvas | [test_workflows.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_workflows.py) `test_workflows_run_in_pipelines_drawn_as_graphs`; [workflow-canvas.test.ts](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/workflow-canvas.test.ts); [journeys.mjs](https://github.com/adeelahmad/lense/blob/main/ci/clicks/journeys.mjs) `pipeline-create`, `workflow-create` |
| What a pipeline's model step or a workflow's model node writes about an asset is found by search, by the API and MCP, under the step's name; running it again replaces those words, a step can opt out, deleting the asset removes them, and outputs written before are made searchable on upgrade | [test_generated_search.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_generated_search.py) `test_a_pipelines_model_step_is_found_by_search`, `test_searchable_is_an_llm_steps_setting`, `test_a_workflows_model_output_is_found_by_search`, `test_outputs_a_model_wrote_before_become_searchable`; [search.test.ts](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/search.test.ts) `says where what wasn't said was found: on screen, on a page, seen, what a shot or page shows, or a model wrote`; [pipelines-templates.test.ts](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/pipelines-templates.test.ts) `keeps an llm step's words in search unless it opts out` |
| Convert steps make new assets: a video's soundtrack as audio, a document's pages as images, each run through its own pipeline | [test_conversions.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_conversions.py) `test_a_videos_soundtrack_becomes_an_audio_asset`, `test_a_documents_pages_become_image_assets`, `test_settings_are_checked`; [pipeline-convert.test.ts](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/pipeline-convert.test.ts) `is added from the library ready to run: to audio unless images are picked` |
| A pipeline's Read step picks its own converter (auto, anytopdf or Lens's own) in place of the server's, checked as it's saved | [test_anytopdf.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_anytopdf.py) `test_a_read_step_picks_its_own_converter`; [pipeline-convert.test.ts](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/pipeline-convert.test.ts) `is kept only on a read step, and said in its summary` |
| A Classify node in workflows chooses from labels, content types or topics, and acting on a sure answer keeps something | [test_classify.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_classify.py) `test_a_workflow_classifies`; [workflow-flow.test.ts](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/workflow-flow.test.ts) `chooses from labels, content types or topics, and acting on a sure answer keeps something` |
Expand Down
4 changes: 3 additions & 1 deletion fastapi_backend/app/api/mcp_tools.py
Original file line number Diff line number Diff line change
Expand Up @@ -456,7 +456,8 @@ def _recording(ctx: Context, rid: int) -> dict[str, Any]:
"search",
"Search the archive",
"Full-text search for moments in the archive: the text of every asset — speech in audio and video (transcripts), "
"text shown on screen in videos or written in documents and images, supplementary files, and kinds of object seen. Words are all required "
"text shown on screen in videos or written in documents and images, supplementary files, kinds of object seen, and what "
"pipelines' model steps wrote about assets (found: generated, with the step in output). Words are all required "
'(matched after stemming), "quoted phrases" must appear as written, OR separates alternatives, NOT or a leading - leaves out lines with a word or phrase. Results are the best '
"moments first, each with the asset, the person (who said or wrote it), where in it, the matching text (matches in **bold**) "
"and a url that opens the asset at that moment. Use get_transcript for the lines around a moment.",
Expand Down Expand Up @@ -515,6 +516,7 @@ def search(
"line": h.idx,
"page": h.page + 1 if h.page is not None else None,
"file": h.file_label,
"output": h.output_label,
"text": _plain(h.snippet),
"url": ctx.link(h.recording_id, h.t0),
}
Expand Down
4 changes: 2 additions & 2 deletions fastapi_backend/app/api/v1/routes/search.py
Original file line number Diff line number Diff line change
Expand Up @@ -45,8 +45,8 @@ def search_transcripts(
),
) -> SearchResults:
"""Moments where the words appear (said in audio or video, shown on screen in a video, written in a document or in
a resource's supplementary transcripts, captions, translations and indexes, or the kinds of object seen in videos,
documents and images),
a resource's supplementary transcripts, captions, translations and indexes, the kinds of object seen in videos,
documents and images, what a model that can see said they show, or what a pipeline's model step wrote about them),
best first, in the namespaces you can read and the collections you were given a role on. A person, emotion or
sound filter keeps to their lines. By meaning, a moment is a passage about what was asked, though its words may differ."""
if ns:
Expand Down
3 changes: 2 additions & 1 deletion fastapi_backend/app/domain/deletion.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@

What goes: its transcript and everything derived from it (segments, speakers' appearances, chapters, entities'
mentions, terms, edits), video shots, text on screen and face tracks with their frames, the pages drawn of documents
and images, outputs and reports, shares,
and images, outputs (and the words of them search finds) and reports, shares,
notes, comments, highlights, supplementary files, permissions, requests for access, and its place in IP groups, fixed collections, chat
scopes and batch runs that haven't started it. Speakers and faces that only it had, and that nobody named, go too; named ones stay. Its jobs
are cancelled (a job that is running has to stop first), and harvesters hear a Delete when it was public.
Expand Down Expand Up @@ -41,6 +41,7 @@
"face_track",
"object_track",
"description",
"generated",
"share_link",
"share_embed",
"note",
Expand Down
4 changes: 3 additions & 1 deletion fastapi_backend/app/domain/migrations.py
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@

def steps():
"""[(name, fn(db))] in the order they run. Append only."""
from . import access, auth, chat, entities, hierarchy, jobs, mcp_servers
from . import access, auth, chat, entities, hierarchy, jobs, mcp_servers, pipelines

return [
# IIIF-only access levels (public, transcript, signed-in, private) become the access setting
Expand All @@ -66,6 +66,8 @@ def steps():
("mcp-sealed-secrets", mcp_servers.migrate_seal),
# entities the old rules called terms get the type the rules give them now (people, places, organisations...)
("entity-types", entities.migrate_types),
# what a model step wrote about a recording before then is found by search too
("generated-search", pipelines.migrate_generated),
]


Expand Down
1 change: 1 addition & 0 deletions fastapi_backend/app/domain/moving.py
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@
"face_track",
"object_track",
"description",
"generated",
"job",
"output",
"share_link",
Expand Down
Loading
Loading
0