8000 feat(chat): thumbs up or down on answers by adeelahmad · Pull Request #702 · adeelahmad/lense · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion README.md
10BC8
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,8 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p
say yes yourself (an API key, an app or an agent can only decline). Give it a goal that needs many look-ups and deep
research comes back with a plan to approve, edit or revise, then runs in the background and returns a cited report,
checked against its sources, to keep as a note, hear aloud or turn into a podcast. Regenerating an answer or editing a
question makes a new version beside the old one, and nothing is lost. Each answer keeps a record of what produced it
question makes a new version beside the old one, and nothing is lost. A thumbs up or down marks an answer good or bad
(pressed again, it's taken back), and agents rate answers the same way over MCP. Each answer keeps a record of what produced it
(the model, its tools, how long it took and what it cost), opened from its header. Copy an answer as Markdown or plain
text, a question or a code block, and export a conversation as Markdown, PDF or JSON, its citations as footnotes that
link back to the moment in Lens. It reads answers aloud with a speech provider, macOS `say`, or Kokoro (54
Expand Down
6 changes: 6 additions & 0 deletions changelog.d/rate-answers.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
- **Thumbs up or down on answers.** Under a finished answer in chat, a thumbs up marks it good and a thumbs down bad;
pressing the one you gave takes it back. The rating is saved on the answer, shown when the conversation opens and
carried in its JSON export (`lens.rating`), the label training data is built from; it makes no model call. Agents
rate answers with `PUT /api/v1/chats/{cid}/messages/{mid}/rating` or the MCP tool `rate_answer`, and read it back on
each message ([Rating answers](docs/assistant.md#rating-answers)). Nothing needs upgrading: older answers have no
rating.
1 change: 1 addition & 0 deletions docs/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -1824,6 +1824,7 @@ POST /api/v1/chats/{cid}/messages
GET /api/v1/chats/{cid}/messages/{mid}
POST /api/v1/chats/{cid}/stop
POST /api/v1/chats/{cid}/messages/{mid}/check
PUT /api/v1/chats/{cid}/messages/{mid}/rating
GET /api/v1/approvals
GET /api/v1/approvals/count
GET /api/v1/approvals/{aid}
Expand Down
11 changes: 11 additions & 0 deletions docs/assistant.md
Original file line number Diff line number Diff line change
Expand Up @@ -126,6 +126,17 @@ An app's chat completions keep no messages, so their record comes back in the re
cost ([Chat completions](api.md#chat-completions-for-any-app)). Answers from before Lens kept records have none; Details
shows what they have and says the rest wasn't recorded.

### Rating answers

Under a finished answer, a thumbs up says it was a good answer and a thumbs down a bad one; pressing the one you gave
takes it back, so there's nothing to confirm. The rating is saved on the answer (`rating`: `up`, `down` or none, with
`rated_at`), shown when the conversation opens on any device, and carried in the conversation's JSON export (`lens.rating`)
([Copy and export](#copy-and-export)). It changes nothing about the answer and makes no model call: it labels which
answers were good, the signal training data is built from. Agents give it with `PUT
/api/v1/chats/<id>/messages/<mid>/rating` (`{"rating": "up"}`, `"down"` or `null`) or the MCP tool `rate_answer`, and
read it back on each message from `GET /api/v1/chats/<id>` and `read_chat`. Only the conversation's owner rates its
answers; a question can't be rated.

### Copy and export

A conversation, or one message of it, comes out as a file to keep or hand on: `GET
Expand Down
3 changes: 2 additions & 1 deletion docs/mcp.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ has the same tools, but sees only that namespace, whatever else the person who s
that take a namespace use this one when none is named, `list_namespaces` and `list_files` show only it, and the tools
about the server itself (`file_storage`, `monitoring`, `services`, `mcp_servers`, `sandbox_gateway`, `install_settings`,
`mesh` and the ones that change them) aren't there, even for admins. Nor are the person's own conversations (`list_chats`,
`read_chat`, `rename_chat`, `export_chat`), projects (`list_projects`, `read_project`, `create_project`,
`read_chat`, `rename_chat`, `rate_answer`, `export_chat`), projects (`list_projects`, `read_project`, `create_project`,
`update_project`, `schedule_project`) and playlists (`list_playlists`, `add_to_playlist`), which belong to no namespace.
An admin counts as an owner of the namespace there.

Expand Down Expand Up @@ -106,6 +106,7 @@ For an app that speaks OpenAI's API rather than MCP, the namespace can also answ
| `list_chats` | the person's conversations with Lens's [assistant](assistant.md#chat), newest first, by words in their title, each with its link |
| `read_chat` | one of them: what it draws on, and the version showing (or the one ending at a `message`), each message with its versions, the tools an answer used, its citations with links and its record; `tree` adds every message's parent and role |
| `rename_chat` | name one of them, or have Lens [title](assistant.md#titles) it from how it starts (no `title`). Needs a write-scope token |
| `rate_answer` | a thumbs up or down on one of their answers, or `clear` to take it back ([Rating answers](assistant.md#rating-answers)); saved on the answer, it changes nothing about it. Needs a write-scope token |
| `export_chat` | one of them as Markdown or JSON ([Copy and export](assistant.md#copy-and-export)): the version showing, one `message`, every version (`all_versions`) or OpenAI's `{"messages": [...]}` shape (`as_messages`) |
| `list_projects` | the person's [projects](projects.md), the most recently changed first: each one's goal, namespaces, coordinator conversation and how many threads are open |
| `read_project` | one of them: its goal and instructions, namespaces, coordinator and threads (each one's title, whether it's done and its last message); read a thread with `read_chat` |
Expand Down
1 change: 1 addition & 0 deletions docs/requirements.md
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,7 @@ request, to another check that proves the same thing; loosening a requirement or
| On a Docker install, Settings starts and stops the sandbox's gateway next to Lens, and says why a machine can't run it | [test_sandbox.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_sandbox.py) `test_the_gateway_is_started_and_stopped_from_settings`, `test_a_machine_that_cant_run_the_gateway_says_why`; [sandbox-gateway.test.ts](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/sandbox-gateway.test.ts) `says what the machine is doing, and whether Lens reaches the gateway` |
| Deep research: a goal gets a plan to approve, runs in the background and comes back as a cited report, saved as a note or a podcast episode | [test_research.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_research.py) `test_a_goal_gets_a_plan_that_waits_for_approval_and_can_be_revised`, `test_an_approved_plan_runs_and_its_report_comes_back_checked`, `test_a_report_becomes_a_note_and_a_podcast`; [chat-research.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/chat-research.test.tsx) `approves the plan with Enter, sending the edits`, `saves it as a note, asking where when it could go in several namespaces`, `turns the next message into a research goal, and says when it can't`; [journeys.mjs](https://github.com/adeelahmad/lense/blob/main/ci/clicks/journeys.mjs) `chat-research`; [baseline.json](https://github.com/adeelahmad/lense/blob/main/ci/clicks/baseline.json) `chat-research` ≤ 4 clicks |
| Regenerating an answer or editing a question makes a new version beside the old one | [test_chat_versions.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_chat_versions.py) `test_regenerate_edit_switch_and_go_on`; [chat-versions.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/chat-versions.test.tsx) `regenerating an answer`, `the version switch`; [journeys.mjs](https://github.com/adeelahmad/lense/blob/main/ci/clicks/journeys.mjs) `chat-regenerate`; [baseline.json](https://github.com/adeelahmad/lense/blob/main/ci/clicks/baseline.json) `chat-regenerate` ≤ 4 clicks |
| A thumbs up or down on an answer is saved, shown when the conversation opens, carried in its JSON export and given by agents over MCP | [test_chat_ratings.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_chat_ratings.py) `test_rating_an_answer_is_saved_shown_and_exported`, `test_someone_elses_answer_cant_be_rated`, `test_agents_rate_answers_over_mcp`; [chat-rating.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/chat-rating.test.tsx) `thumbs up saves the rating, and pressing it again takes it back` |
| Each answer keeps a record of what produced it: the model, its tools, how long it took and what it cost | [test_chat_records.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_chat_records.py) `test_an_answer_records_what_produced_it`, `test_a_regenerated_answer_has_its_own_record`; [chat-details.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/chat-details.test.tsx) `says what produced it: model and host, scope, tools, use, tokens, cost, time and how it ended` |
| A conversation exported as Markdown, PDF or JSON, its citations as footnotes linking back to Lens, by people and by agents over MCP | [test_chat_export.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_chat_export.py) `test_markdown`, `test_json`, `test_pdf_is_printed_by_what_the_server_has`, `test_agents_read_and_export_over_mcp`; [chat-copy-export.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/chat-copy-export.test.tsx) `downloads what the server makes, named as the server names it`, `prints in the browser when the server can't make the PDF` |
| Answers, questions and code blocks copied from the web app, an answer as Markdown or plain text | [chat-copy-export.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/chat-copy-export.test.tsx) `gives Markdown with citations as footnotes linking back to Lens, or plain text`, `copies a question`, `copies a code block and says Copied` |
Expand Down
26 changes: 26 additions & 0 deletions fastapi_backend/app/api/mcp_tools.py
Original file line number Diff line number Diff line change
Expand Up @@ -2253,6 +2253,7 @@ def _chat_message(ctx: Context, m: dict[str, Any]) -> dict[str, Any]:
"citations": [chat_export.citation(ctx.web, p) for p in m.get("passages") or []] or None,
"record": m.get("meta"),
"stopped": m.get("stopped") or None,
"rating": m.get("rating"),
"notice": m.get("notice"),
"error": m.get("error"),
"files": [a.get("filename") for a in m.get("attachments") or []] or None,
Expand Down Expand Up @@ -2319,6 +2320,31 @@ def rename_chat(ctx: Context, chat_id: int, title: str | None) -> dict[str, Any]
return {"chat_id": chat_id, **named, "url": f"{ctx.web}/chat/{chat_id}"}


@tool(
"rate_answer",
"Rate an answer",
"Thumbs up or down on an answer in one of your conversations, as the person would give it in the web app, or "
"`clear` to take it back. The rating is saved on the answer and shown with it: a label for training data that says "
"which answers were good, not a change to the answer. Needs a token with the write scope.",
Arg("chat_id", "integer", "the conversation (from list_chats)", required=True),
Arg("message", "integer", "the answer (an assistant message's id from read_chat)", required=True),
Arg("rating", "string", "up for a good answer, down for a bad one, clear to remove it", required=True, enum=("up", "down", "clear")),
writes=True,
personal=True,
)
def rate_answer(ctx: Context, chat_id: int, message: int, rating: str) -> dict[str, Any]:
if not ctx.user.can_write:
raise ToolError("this token is read-only: ask for a token with the write scope to rate answers")
chat_routes._own_chat(ctx.db, chat_id, ctx.user)
try:
got = chat.rate(ctx.db, chat_id, message, None if rating == "clear" else rating)
except KeyError:
raise ToolError("that isn't an answer in this conversation") from None
except ValueError:
raise ToolError("rating is up, down or clear") from None
return {"chat_id": chat_id, "message": message, "rating": got, "url": f"{ctx.web}/chat/{chat_id}"}


@tool(
"export_chat",
"Export a conversation",
Expand Down
13 changes: 13 additions & 0 deletions fastapi_backend/app/api/v1/routes/chats.py
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,8 @@
HeadUpdate,
MessageCreate,
MessageDetail,
MessageRating,
RatingUpdate,
StopResult,
)
from app.schemas.common import Created, Ok
Expand Down Expand Up @@ -873,6 +875,17 @@ def check_message(cid: int, mid: int, user: Writer, db: Db, cfg: Cfg) -> AnswerC
return out


@router.put("/chats/{cid}/messages/{mid}/rating")
def rate_message(cid: int, mid: int, body: RatingUpdate, user: Writer, db: Db) -> MessageRating:
"""Thumbs up or down on an answer (`rating` null clears it). It's saved on the message, shown when the
conversation opens and carried in its JSON export: a label for training data, not a change to the answer."""
_own_chat(db, cid, user)
try:
return MessageRating(id=mid, rating=chat.rate(db, cid, mid, body.rating))
except KeyError:
raise HTTPException(404, "not found") from None


def _via(user: Principal) -> str:
"""Where a pick is made: web for a signed-in session, else token (an API key) or oauth (an app)."""
return "web" if user.via == "access" else user.via
Expand Down
17 changes: 16 additions & 1 deletion fastapi_backend/app/domain/chat.py
Original file line number Diff line number Diff line change
Expand Up @@ -376,14 +376,29 @@ def history(db, cid, ids=None):
"""A conversation's messages, oldest first: every version's, or only those with `ids` (a path)."""
return db.rows(
"SELECT record::id(id) AS id, role, content, passages, created_at, stopped ?? false AS stopped, steps ?? [] AS steps, "
"attachments ?? [] AS attachments, notice, error, check, model, context, parent, meta, kind FROM chat_message WHERE chat = $c"
"attachments ?? [] AS attachments, notice, error, check, rating, model, context, parent, meta, kind FROM chat_message WHERE chat = $c"
+ (" AND id IN $ids" if ids is not None else "")
+ " ORDER BY id",
c=cid,
ids=[R("chat_message", i) for i in ids or []],
)


RATINGS = ("up", "down")


def rate(db, cid, mid, rating):
"""Rate an answer of conversation `cid`: "up", "down" or None to clear it, saved on the message with when. KeyError
when `mid` isn't one of its answers. Ratings label answers for the training data (good or bad), nothing else."""
m = db.one("SELECT chat, role FROM $r", r=R("chat_message", mid))
if not m or m["chat"] != cid or m["role"] != "assistant":
raise KeyError(mid)
if rating not in (*RATINGS, None):
raise ValueError(rating)
db.q("UPDATE $r SET rating = $v, rated_at = IF $v = NONE THEN NONE ELSE time::now() END", r=R("chat_message", mid), v=rating)
return rating


# ---------- versions: a conversation is a tree, each message under the one it follows; the head is the version showing
HEAD = "head" # where add, path and run_command go on from by default: the conversation's head

Expand Down
2 changes: 1 addition & 1 deletion fastapi_backend/app/domain/chat_export.py
Original file line number Diff line number Diff line change
Expand Up @@ -204,7 +204,7 @@ def _message(m):

def _lens(web, m):
"""What only Lens knows of a message."""
keep = ("model", "stopped", "notice", "error", "check", "context", "versions")
keep = ("model", "stopped", "notice", "error", "check", "rating", "context", "versions")
out = {k: m.get(k) for k in keep if m.get(k)}
if m.get("attachments"):
out["attachments"] = [store.clean({k: a.get(k) for k in ("filename", "size", "recording")}) for a in m["attachments"]]
Expand Down
10 changes: 10 additions & 0 deletions fastapi_backend/app/schemas/chats.py
Original file line number Diff line number Diff line change
Expand Up @@ -270,6 +270,7 @@ class ChatMessage(ResponseModel):
notice: str | None = Field(None, description="e.g. the model can't use tools, so the answer came from a search")
error: str | None = Field(None, description="why there's no answer")
check: AnswerCheck | None = Field(None, description="its latest source check (POST .../messages/{mid}/check)")
rating: Literal["up", "down"] | None = Field(None, description="your thumbs up or down on an answer (PUT .../messages/{mid}/rating)")
model: str | None = Field(None, description="the model that wrote the answer")
parent: int | None = Field(None, description="the message it follows; null for a first question")
versions: Versions | None = Field(None, description="it and the others said in its place, from regenerating or editing")
Expand Down Expand Up @@ -332,6 +333,15 @@ class Chat(ChatSummary):
)


class RatingUpdate(RequestModel):
rating: Literal["up", "down"] | None = Field(description="up (a good answer), down (a bad one), or null to clear it")


class MessageRating(ResponseModel):
id: int = Field(description="the answer")
rating: Literal["up", "down"] | None


class HeadUpdate(RequestModel):
message: int = Field(description="a message of the version to show: the head moves to the newest message below it")

Expand Down
Loading
Loading
0