FFFF feat(search): leave words out with NOT or -word by adeelahmad · Pull Request #603 · adeelahmad/lense · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,10 +77,10 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p
server](docs/mcp.md#one-namespaces-mcp-server), [chat completions](docs/api.md#chat-completions-for-any-app),
[API](docs/api.md), [the graph as files](docs/file-tree.md), [authentication](docs/authentication.md);
[checks](docs/requirements.md#connect-your-agents))
- **Ask, and get cited answers.** A chat assistant on every page, and by voice, searches your archive by words (and by
meaning, once turned on), reads it, cites it and acts on it with tools: changes wait for your approval. Ask it about
Lens itself ("what's the MCP address?", "is Telegram on?") and it answers from Lens's docs and this server, citing
both. Once an admin
- **Ask, and get cited answers.** A chat assistant on every page, and by voice, searches your archive by words (with
"phrases", OR, and -word to leave lines out) and by meaning, once turned on, reads it, cites it and acts on it with
tools: changes wait for your approval. Ask it about Lens itself ("what's the MCP address?", "is Telegram on?") and it
answers from Lens's docs and this server, citing both. Once an admin
turns them on, it also searches the web and reads pages, cited like the archive, and runs Python and bash in a sandbox
of each conversation's own. An admin's assistant searches and reads the mail, calendars and files connected to Lens
(Gmail, Google Calendar, Google Drive, S3 and the rest), cites messages and events, and brings them in once you
Expand Down
3 changes: 3 additions & 0 deletions changelog.d/search-not.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
- **Leave words out of a search.** `budget -draft` or `budget NOT draft` finds the lines about the budget that don't
mention a draft (or drafts), and `-"first draft"` leaves out a phrase; a typed AND changes nothing, as every word is
needed anyway. It works in the web app, the API, MCP and the assistant's search, and the "?" help shows it.
3 changes: 2 additions & 1 deletion docs/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -1168,7 +1168,8 @@ GET /api/v1/mentions
`GET /search?q=` finds the moments where the words appear (in an asset's text or transcript, on screen in a video, in a resource's
supplementary transcripts, captions, translations and indexes, or in what a model says its shots and pages show) in the
namespaces you can read, best first: every word
(English stemming), "quoted phrases" as written, `OR` between alternatives; `ns`, `speaker`, `emotion` and `recording`
(English stemming), "quoted phrases" as written, `OR` between alternatives, `NOT` or `-` before a word or phrase to leave out the lines
with it; `ns`, `speaker`, `emotion` and `recording`
narrow it, and `object` keeps to the assets a kind of object is seen in ([Objects](#objects));
`limit`/`offset` page through it. `total` counts the moments ranked so far (`capped` when there may be more).
With `facets=true` it also counts all the matching moments, whatever the page, by namespace, person, emotion and
Expand Down
6 changes: 4 additions & 2 deletions docs/processing.md
Original file line number Diff line number Diff line change
Expand Up @@ -324,7 +324,9 @@ about them come back as not found.
## Search

All words must appear, matched after English stemming ("exploit" also finds exploits and exploiting); "quoted phrases"
must appear as written; OR separates alternatives. Filter by namespace, person, emotion or asset. The words are
must appear as written; OR separates alternatives; NOT, or a `-` before a word or "phrase", leaves out the lines that
have it (`budget -draft` finds budget without draft or drafts, `budget NOT "first draft"`), and a typed AND changes
nothing. Filter by namespace, person, emotion or asset. The words are
looked up in the index of `search.engine`: a SQLite FTS5 file by default, SurrealDB's own index, or OpenSearch
([configuration](configuration.md#word-search)). For archives that aren't in English set `search.stemming: none` and
run `lens reindex`. There is no prefix search: a trailing `*` (`expl*`) is ignored and the word is searched as it is;
Expand All @@ -344,7 +346,7 @@ entity across the text run the step again.
A search is matched three ways (`mode` on `GET /search`, the Match switch in the web app):

* **auto** (the default): by its words and by meaning, fused by reciprocal rank, when search by meaning is set up and
the query has no "quoted phrases" or OR (those ask for exactly those words); else by its words.
the query has no "quoted phrases", OR or NOT (those ask for exactly those words); else by its words.
* **keyword**: only the words, as above.
* **semantic**: only by meaning.

Expand Down
1 change: 1 addition & 0 deletions docs/requirements.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,7 @@ request, to another check that proves the same thing; loosening a requirement or
| A chat assistant on every page | [test_chats.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_chats.py) `test_asking_from_a_page`; [page-chat.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/page-chat.test.tsx); [ask-place.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/ask-place.test.tsx) |
| By voice | [assistant-mode.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/assistant-mode.test.tsx) `starts a voice chat from the mic`; [test_voice.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_voice.py) `test_what_is_said_comes_back_as_text` |
| It searches by words, and by meaning once search by meaning is set up | [test_chats.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_chats.py) `test_tools_approvals_and_checking`; [test_semantic.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_semantic.py) `test_chat_retrieval_by_meaning`, `test_off_or_unconfigured` |
| Word search takes "phrases", OR, and NOT or -word to leave lines out | [test_search_index.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_search_index.py) `test_not_and_minus_leave_lines_out`, `test_the_queries_parts` |
| Answers cite their sources, and only what the person may read | [test_chats.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_chats.py) `test_chat_cites_only_what_you_can_read`; [chat.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/chat.test.tsx) `splits [n], [n, m] and [n][m]` |
| It acts with tools, and changes wait for approval | [test_chats.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_chats.py) `test_tools_approvals_and_checking`; [test_approvals.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_approvals.py) `test_the_assistant_lists_and_declines_but_never_approves` |
| Web search and reading pages, cited like the archive, off until an admin turns it on | [test_web_search.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_web_search.py) `test_an_answer_cites_the_web_and_the_conversation_keeps_it`, `test_reading_pages_and_pdfs`; [chat-web-citations.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/chat-web-citations.test.tsx) |
Expand Down
2 changes: 1 addition & 1 deletion fastapi_backend/app/api/mcp_tools.py
Original file line number Diff line number Diff line change
Expand Up @@ -439,7 +439,7 @@ def _recording(ctx: Context, rid: int) -> dict[str, Any]:
"Search the archive",
"Full-text search for moments in the archive: the text of every asset — speech in audio and video (transcripts), "
"text shown on screen in videos or written in documents and images, supplementary files, and kinds of object seen. Words are all required "
'(matched after stemming), "quoted phrases" must appear as written, OR separates alternatives. Results are the best '
'(matched after stemming), "quoted phrases" must appear as written, OR separates alternatives, NOT or a leading - leaves out lines with a word or phrase. Results are the best '
"moments first, each with the asset, the person (who said or wrote it), where in it, the matching text (matches in **bold**) "
"and a url that opens the asset at that moment. Use get_transcript for the lines around a moment.",
Arg("query", "string", 'what to look for: words, "a phrase", or alternatives separated by OR', required=True, max_length=500),
Expand Down
2 changes: 1 addition & 1 deletion fastapi_backend/app/api/v1/routes/public.py
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ def get_public_collection(
def search_public(
acl: Acl,
db: Db,
q: str = Query("", description='words, "phrases", OR between alternatives'),
q: str = Query("", description='words, "phrases", OR between alternatives, NOT or -word to leave out'),
limit: int = Query(20, ge=1, le=100),
offset: int = Query(0, ge=0),
) -> PublicSearch:
Expand Down
2 changes: 1 addition & 1 deletion fastapi_backend/app/api/v1/routes/search.py
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ def search_transcripts(
user: CurrentUser,
db: Db,
cfg: Cfg,
q: str = Query(description='words, "phrases", OR between alternatives'),
q: str = Query(description='words, "phrases", OR between alternatives, NOT or -word to leave out'),
ns: str | None = None,
speaker: int | None = None,
emotion: str | None = None,
Expand Down
7 changes: 5 additions & 2 deletions fastapi_backend/app/domain/public.py
Original file line number Diff line number Diff line change
Expand Up @@ -313,13 +313,16 @@ def readable(db, who):


def _title_match(groups):
"""Every word and phrase of one of the query's alternatives in the title (ignoring case)."""
"""Every word and phrase of one of the query's alternatives in the title (ignoring case), and none it leaves out."""
alts, p = [], {}
for k, g in enumerate(groups):
words = [w.lower() for w in g["words"] + g["phrases"]]
alts.append("(" + " AND ".join(f"string::contains(string::lowercase(title ?? ''), $t{k}_{j})" for j in range(len(words))) + ")")
p.update({f"t{k}_{j}": w for j, w in enumerate(words)})
return "(" + " OR ".join(alts) + ")", p
nots = [w.lower() for w in (groups[0].get("not") or [])] if groups else [] # and none it leaves out (NOT, -)
p.update({f"n{j}": w for j, w in enumerate(nots)})
left_out = "".join(f" AND !string::contains(string::lowercase(title ?? ''), $n{j})" for j in range(len(nots)))
return "(" + " OR ".join(alts) + ")" + left_out, p


def search(db, q, who, limit=20, offset=0):
Expand Down
58 changes: 41 additions & 17 deletions fastapi_backend/app/domain/search.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,9 @@
"""Full-text search over every segment through SurrealDB's BM25 index, with highlighted snippets.

Words are ANDed and matched after English stemming (exploit finds exploits and exploiting),
"quoted phrases" must appear as written, and OR separates alternatives.
"quoted phrases" must appear as written, OR separates alternatives, and NOT or a leading - leaves out the lines that
have a word or "phrase" (`budget -draft`, `budget NOT "first draft"`). AND may be typed between words; it changes
nothing, as every word is needed anyway.
"""

from __future__ import annotations
Expand All @@ -24,29 +26,47 @@
FACET_VALUES = 50 # values listed per facet


def _term(p):
"""A token as a word or a "phrase" ('' for punctuation alone)."""
if len(p) > 2 and p[0] == p[-1] == '"':
return " ".join(re.findall(r"[\w'’.-]+", p[1:-1]))
return re.sub(r"[^\w'’.-]", "", p.rstrip("*")).strip(".-")


def parse_query(q):
groups, cur = [], {"words": [], "phrases": []}
for p in re.findall(r'"[^"]+"|\S+', q or ""):
if p.upper() == "OR":
if cur["words"] or cur["phrases"]:
"""The query as up to four groups, any of which will do: {words, phrases, not}. `not` (the words and phrases after
NOT or a -) is the same for every group: a line with any of them is left out."""
groups, cur, out, negate = [], {"words": [], "phrases": []}, [], False
for p in re.findall(r'-?"[^"]+"|\S+', q or ""):
if p.upper() in ("OR", "AND", "NOT"):
if p.upper() == "OR" and (cur["words"] or cur["phrases"]):
groups.append(cur)
cur = {"words": [], "phrases": []}
negate = p.upper() == "NOT"
continue
if len(p) > 2 and p[0] == p[-1] == '"':
inner = " ".join(re.findall(r"[\w'’.-]+", p[1:-1]))
if inner:
cur["phrases"].append(inner)
continue
w = re.sub(r"[^\w'’.-]", "", p.rstrip("*")).strip(".-")
if w:
cur["words"].append(w)
if p.startswith("-") and len(p) > 1:
negate, p = True, p[1:]
t = _term(p)
if t and negate:
out.append(t)
elif t:
cur["phrases" if p[0] == p[-1] == '"' and len(p) > 2 else "words"].append(t)
negate = False
if cur["words"] or cur["phrases"]:
groups.append(cur)
return groups[:4]
return [{**g, "not": out[:8]} for g in groups[:4]]


def describe(groups):
return " OR ".join(" ".join(g["words"] + [f'"{x}"' for x in g["phrases"]]) for g in groups)
left_out = [f'-"{x}"' if " " in x else f"-{x}" for x in (groups[0].get("not") or [])] if groups else []
return " OR ".join(" ".join(g["words"] + [f'"{x}"' for x in g["phrases"]]) for g in groups) + "".join(f" {x}" for x in left_out)


def excluded(text, groups):
"""Whether a line has a word or phrase the query leaves out (NOT, -): the word's other forms too (-draft leaves
out drafts), as stemming would."""
low, nots = (text or "").lower(), (groups[0].get("not") or []) if groups else []
return any(re.search(r"\b" + re.escape(t.lower()) + (r"\b" if " " in t else ""), low) for t in nots)


def snippet(marked, width=110):
Expand Down Expand Up @@ -160,6 +180,8 @@ def search(
low = r["text"].lower()
if not any(all(re.search(r"\b" + re.escape(p.lower()) + r"\b", low) for p in g["phrases"]) for g in groups):
continue
if excluded(r["text"], groups):
continue
marks = [r.get(f"h{k}") for k in range(1, len(groups) + 1) if isinstance(r.get(f"h{k}"), str)]
best = max(marks, key=lambda h: h.count(M0)) if marks else r["text"]
r["_score"] = sum(abs(r.get(f"s{k}") or 0) for k in range(1, len(groups) + 1)) # BM25; some engines return it negated
Expand Down Expand Up @@ -290,8 +312,8 @@ def _mode(cfg, mode, groups, can):
raise ValueError("mode is keyword, semantic, hybrid or auto")
if mode == "keyword":
return "keyword", None
if mode == "auto" and (len(groups) > 1 or any(g["phrases"] for g in groups)):
return "keyword", None # "phrases" and OR ask for exactly those words
if mode == "auto" and (len(groups) > 1 or any(g["phrases"] or g.get("not") for g in groups)):
return "keyword", None # "phrases", OR and NOT ask for exactly those words
if cfg is None or not semantic.configured(cfg):
return "keyword", (None if mode == "auto" else "search by meaning is off, or has no embeddings server")
if not can:
Expand Down Expand Up @@ -594,6 +616,8 @@ def _layer(db, groups, where_f, cap, base, table, extra, source):
low = r["text"].lower()
if not any(all(re.search(r"\b" + re.escape(p.lower()) + r"\b", low) for p in g["phrases"]) for g in groups):
continue
if excluded(r["text"], groups):
continue
marks = [r.get(f"h{k}") for k in range(1, len(groups) + 1) if isinstance(r.get(f"h{k}"), str)]
r["_snip"] = snippet(max(marks, key=lambda h: h.count(M0)) if marks else r["text"])
r["_score"] = sum(abs(r.get(f"s{k}") or 0) for k in range(1, len(groups) + 1))
Expand Down
8 changes: 5 additions & 3 deletions fastapi_backend/app/domain/textindex.py
Original file line number Diff line number Diff line change
Expand Up @@ -175,12 +175,14 @@ def _quote(term):


def match_expr(groups):
"""An FTS5 query for search groups: each group's words and phrases all present, any group will do."""
"""An FTS5 query for search groups: each group's words and phrases all present, any group will do, and none of
the words or phrases it leaves out (`not`)."""
parts = []
for g in groups:
terms = [_quote(t) for t in g["words"] + g["phrases"] if t.strip()]
nots = [_quote(t) for t in g.get("not") or [] if t.strip()]
if terms:
parts.append("(" + " ".join(terms) + ")")
parts.append("(" + " ".join(terms) + ")" + (" NOT (" + " OR ".join(nots) + ")" if nots else ""))
return " OR ".join(parts)


Expand Down Expand Up @@ -347,7 +349,7 @@ def _find(self, table, groups, limit, marks):
must = [{"match": {"text": {"query": " ".join(g["words"]), "operator": "and"}}}] if g["words"] else []
must += [{"match_phrase": {"text": p}} for p in g["phrases"]]
if must:
should.append({"bool": {"must": must}})
should.append({"bool": {"must": must, "must_not": [{"match_phrase": {"text": t}} for t in g.get("not") or []]}})
if not should:
return []
body = {"size": limit, "_source": False, "query": {"bool": {"should": should, "minimum_should_match": 1}}}
Expand Down
2 changes: 1 addition & 1 deletion fastapi_backend/app/schemas/views.py
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ class ViewUpdate(RequestModel):
class SearchState(RequestModel):
"""A search as /search takes it (its namespace is the saved search's)."""

q: str = Field(min_length=1, max_length=500, description='words, "phrases", OR between alternatives')
q: str = Field(min_length=1, max_length=500, description='words, "phrases", OR between alternatives, NOT or -word to leave out')
speaker: int | None = None
emotion: str | None = Field(None, max_length=40)
recording: int | None = None
Expand Down
28 changes: 28 additions & 0 deletions fastapi_backend/tests/domain/test_search_index.py
Original file line number Diff line number Diff line change
Expand Up @@ -207,3 +207,31 @@ def test_opensearch_needs_an_address(tmp_path):
assert saved["opensearch_password"] == "pw" and saved["engine"] == "opensearch"
finally:
db.close()


@pytest.mark.parametrize("engine", [*OUTSIDE, "surrealdb"])
def test_not_and_minus_leave_lines_out(tmp_path, engine):
"""NOT, or a - before a word or "phrase", leaves out the lines that have it; a typed AND changes nothing."""
cfg = make_cfg(tmp_path, url="surrealkv://" + str(tmp_path / "kv"), search=_engine(engine))
db = store.connect(cfg)
try:
for name, text in (("ep1.txt", PODS1), ("ep2.txt", PODS2)):
(tmp_path / name).write_text(text)
ingest.import_transcript(db, cfg, "pods", tmp_path / name, log=quiet)
total = {q: search.search(db, q)["total"] for q in ("capsid", "capsid -published", "capsid NOT model", "capsid AND published")}
assert total == {"capsid": 4, "capsid -published": 3, "capsid NOT model": 2, "capsid AND published": 1}
assert search.search(db, 'capsid -"human designs"')["total"] == 3
assert search.search(db, "Dyno -capsid")["total"] == 2
assert search.search(db, "exploit OR capsid NOT benchmark")["total"] == 4 # 7, less the 3 with "benchmark"
assert search.search(db, "-capsid")["total"] == 0 # nothing to look for
assert search.search(db, "capsid -published -model")["query"] == "capsid -published -model"
finally:
db.close()


def test_the_queries_parts():
assert search.parse_query('a AND b NOT c -"d e" OR f') == [
{"words": ["a", "b"], "phrases": [], "not": ["c", "d e"]},
{"words": ["f"], "phrases": [], "not": ["c", "d e"]},
]
assert textindex.match_expr(search.parse_query("a -c")) == '("a") NOT ("c")'
6 changes: 3 additions & 3 deletions nextjs-frontend/app/openapi-client/types.gen.ts
Original file line number Diff line number Diff line change
Expand Up @@ -22151,7 +22151,7 @@ export type SearchCreate = {
/**
* Q
*
* words, "phrases", OR between alternatives
* words, "phrases", OR between alternatives, NOT or -word to leave out
*/
q: string;
/**
Expand Down Expand Up @@ -38167,7 +38167,7 @@ export type SearchTranscriptsData = {
/**
* Q
*
* words, "phrases", OR between alternatives
* words, "phrases", OR between alternatives, NOT or -word to leave out
*/
q: string;
/**
Expand Down Expand Up @@ -46305,7 +46305,7 @@ export type SearchPublicData = {
/**
* Q
*
* words, "phrases", OR between alternatives
* words, "phrases", OR between alternatives, NOT or -word to leave out
*/
q?: string;
/**
Expand Down
Loading
Loading
0