8000 feat(timeline): the agent answers when tagged in a comment by adeelahmad · Pull Request #622 · adeelahmad/lense · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -184,7 +184,8 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p
warnings) is kept as a stream of activities in W3C Activity Streams form, which agents read through the API, MCP and
the assistant, and people review on the Timeline page (and Latest on Home), marking what they've seen, reacting with
an emoji and commenting in threads everyone who sees the item shares (agents too, over MCP), and asking the
assistant about any item from the item. Automations act
assistant about any item from the item; a comment that tags @lens gets the agent's answer in the thread, with
every change it proposes waiting for a yes. Automations act
on that stream, on a schedule, at a time or when their own webhook is called, alone or with their triggers joined (all
of them, a number of times, one that doesn't come in time, or a batch): they test conditions (text, patterns,
numbers, lists, times, days, whether a value changed and whether something like it happened before in their
Expand Down
5 changes: 5 additions & 0 deletions changelog.d/comment-agent.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
**Tag @lens in a comment and the agent answers in the thread.** On the timeline or on an asset's page, a comment
with `@lens` (or the assistant's own name) in it gets the agent's answer as a comment by Lens on the same thread,
as the person who tagged it, with their tools, in a conversation of theirs titled after the item; every change it
proposes waits for their approval, and the reply links where to decide. Off with Settings → AI assistant → Answer
when tagged in a comment ([Asking the agent in a comment](docs/timeline.md#asking-the-agent-in-a-comment)).
6 changes: 6 additions & 0 deletions docs/assistant.md
Original file line number Diff line number Diff line change
Expand Up @@ -766,6 +766,12 @@ A chat room can be given to a namespace's assistant too ([chat rooms](chat-rooms

Refine later: a decision model choosing what's worth remembering, similar memories merged.

## Asking the assistant in a comment

Tag **@lens** (or the assistant's own name) in a comment on the timeline or on an asset's page and it answers in the
thread, as you, with your tools, in a conversation of yours it opens for the thread; changes it proposes wait for
your yes as they do in chat ([Asking the agent in a comment](timeline.md#asking-the-agent-in-a-comment)).

## Extending the assistant: tools, skills, hooks and plugins

Anyone who can edit can add to what the assistant does. Each addition is an *extension*: versioned, kept private or
Expand Down
1 change: 1 addition & 0 deletions docs/requirements.md
Original file line number Diff line number Diff line change
Expand Up @@ -236,6 +236,7 @@ request, to another check that proves the same thing; loosening a requirement or
| Ask the assistant about a timeline item from the item (its button, or `a` on its line), in the chat beside the page, with the item's id, line and fields as what the question is about | [timeline-ui.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/timeline-ui.test.tsx) `asking about an item hands the page chat its id, line and fields, by button or the a key`; [page-chat.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/page-chat.test.tsx) `opens with what a page's own Ask about this hands it, and sends it as the highlighted part` |
| The Timeline page and Latest on Home, marking what you've seen | [test_timeline_marks.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_timeline_marks.py) `test_marks_are_set_cleared_and_filter_the_timeline`; [timeline-ui.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/timeline-ui.test.tsx) `Home's Latest card lists what you haven't reviewed` |
| Reactions and comments on timeline activities, shared by everyone who sees them, through the API, MCP and the page | [test_timeline_social.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_timeline_social.py) `test_reactions_are_shared_counted_and_taken_back`, `test_comments_on_an_activity_are_a_thread_everyone_who_sees_it_joins`, `test_comments_on_a_logged_activity_follow_its_audience`, `test_agents_react_and_comment_too`; [timeline-ui.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/timeline-ui.test.tsx) `reactions and comments: a line sums its items', a 👍 goes to the server, and Enter sends a comment` |
| Tagging @lens in a comment on the timeline or an asset gets the agent's answer in the thread, as the person who tagged it, with every change waiting for their approval, off with `ai.answer_comments` | [test_comment_agent.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_comment_agent.py) `test_the_agent_answers_a_tagged_comment_in_the_thread`, `test_without_a_model_or_within_a_used_up_budget_the_reply_says_so`; [timeline-social-model.test.ts](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/timeline-social-model.test.ts) `the agent's reply links its addresses` |
| Automations act on the stream, on a schedule, at a time or when their own webhook is called | [test_automation_runs.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_automation_runs.py) `test_each_new_activity_runs_each_automation_it_matches_once`, `test_a_schedule_runs_its_automation_when_its_timer_fires`, `test_a_call_to_a_webhook_runs_its_automation`; [test_timer_triggers.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_timer_triggers.py) `test_an_at_trigger_fires_once_then_reads_done` |
| They test conditions, then tell you, ask the model or the agent, use a tool, call a web address with sealed credentials, run Lens work, or change assets and notes | [test_automation_runs.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_automation_runs.py) `test_the_if_limits_and_dry_runs`; [test_actions_messages.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_actions_messages.py) `test_notify_tells_me_in_my_chat_app`; [test_actions_ai.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_actions_ai.py) `test_ask_model_answers_in_text_or_as_the_fields_a_schema_names`, `test_ask_agent_works_on_what_happened_and_saves_the_conversation`, `test_call_tool_reads_as_the_person_and_changes_only_by_asking_them`; [test_actions_web.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_actions_web.py) `test_credentials_in_headers_are_sealed_to_the_automation_and_the_host_and_never_shown`; [test_actions_run.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_actions_run.py) `test_run_workflow_queues_it_on_the_asset_as_the_automation`, `test_run_pipeline_processes_the_asset_again`; [test_actions_lens.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_actions_lens.py) `test_set_field_is_kept_in_the_assets_history_by_the_automation`, `test_write_note_writes_a_page_adds_to_a_things_page_or_notes_an_asset` |
| What reaches outside Lens, or changes more than what set it off, asks first: on the Approvals page or with a yes in chat; agents decline but never approve | [test_automation_spec.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_automation_spec.py) `test_risk_follows_the_effect`; [test_approvals.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_approvals.py) `test_approve_carries_the_action_out_and_the_run_records_it`, `test_agents_over_mcp_list_and_decline_but_never_approve`; [test_approvals_chat.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_approvals_chat.py) `test_a_yes_in_my_chat_decides_an_action_inside_lens`; [approvals-page.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/approvals-page.test.tsx) `groups what waits, puts the first in focus, and Enter approves it` |
Expand Down
21 changes: 19 additions & 2 deletions docs/timeline.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ Status: in progress. The studio's timeline is built in steps; only what is merge
| Reactions and comments on any item, shared by everyone who sees it; `react_to_activity`, `activity_comments` and `comment_on_activity` for agents | built |
| Ask about an item: talk to the assistant about it from the item, in the chat beside the page | built |
| What each item caused (automation runs, approvals), and Approve and Decline on waiting approvals | planned |
| Tagging the agent in a comment, so it answers in the thread | planned |
| Tagging the agent in a comment, so it answers in the thread | built |
| Sources, sensors, people and agents as actors in the graph; a For you order learned from what each person reacts to, comments on and hides | planned |

## What shows
Expand Down Expand Up @@ -95,6 +95,20 @@ them, and each one is an activity of its own, so an automation (or an agent) can
are the thread, as [comments on a resource](api.md#comments) are. `GET /api/v1/activities` carries everyone's
reactions on the page in `reactions` and each item's comment count in `comments`, by id.

## Asking the agent in a comment

A comment that tags **@lens** (or the assistant's own name from Settings → Chat rooms, say `@jarvis`) gets an answer
from the agent in the same thread, on the timeline and on an asset's page alike: "@lens what is this one about?",
"@lens write a note of the action items". The agent answers as the person who tagged it, with their namespaces and
their tools, in a conversation of theirs titled after the item (one per thread, so a later tag on the thread carries
on in it); they find it on the Chat page and can go on there. It reads the item the thread is on and the comments on
it so far, as data, and answers with its sources. It only reads: every change it proposes (a note, a move, a message
out) waits for that person's yes, and the reply says so with a link to decide. The reply is a comment by **Lens**
(marked with a sparkle), which the person who asked can delete and nobody can edit. Its model calls run in the
ledger against their conversation and the item's namespace; when no model is set up or the namespace's budget is
used up, the reply says so instead. **Settings → AI assistant → Answer when tagged in a comment**
(`ai.answer_comments`, on) turns it off; without a tag nothing is asked of the model.

## Review marks

Each person marks activities for themselves alone; nobody else sees or sets them, admins included.
Expand Down Expand Up @@ -129,7 +143,8 @@ Marks change nothing else: an activity you hid still triggers automations and st
- MCP, on the feed: `react_to_activity` (`id`, `emoji`, `on`) reacts for the person it acts for, `activity_comments`
(`id`) reads an item's threads, and `comment_on_activity` (`id`, `text`, `parent`) writes a comment or a reply,
with a token that has the write scope. The `timeline` page carries `reactions` and `comments` too. Comments are
what people wrote: agents read them as data, never as instructions.
what people wrote: agents read them as data, never as instructions. A comment an agent writes with `@lens` in it
gets the assistant's answer like anyone's, as the person the token acts for.

## How marks, reactions and comments are kept

Expand All @@ -156,3 +171,5 @@ counts are two grouped queries. Nothing of this costs a model call.
the person is removed; a tidy could drop them. Reactions and comments on such activities likewise.
- Who gave each reaction, on hover; editing and resolving comments from the timeline (the API has both).
- A `comment.added` on an activity links to the timeline, not yet to the item itself.
- The agent's reply in a thread is plain text with links; the answer's excerpts could open at their moment, as in
chat. Tagging a namespace's own assistant by its name, with its memory.
8 changes: 5 additions & 3 deletions fastapi_backend/app/api/v1/routes/activities.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@

from app.api.deps import Access, Acl, Cfg, CurrentUser, Db, Principal, Writer, domain_errors
from app.api.v1.routes.comments import _comments_out
from app.domain import auth, comments, event_marks, event_reactions, events, events_content, store
from app.domain import auth, comment_agent, comments, event_marks, event_reactions, events, events_content, store
from app.schemas.activities import (
ActivityKind,
ActivityMark,
Expand Down Expand Up @@ -268,14 +268,16 @@ def comment(
audience=lens.get("audience"),
owner=lens.get("owner"),
)
comment_agent.wake(db, cfg, cid, text)
return _comments_out(db, [comments.get(db, cid)], _rank(acl, user, a), user)[0]


@router.post("/{activity_id}/comments")
def comment_on_activity(activity_id: str, body: ActivityCommentCreate, user: Writer, acl: Acl, db: Db, cfg: Cfg) -> Comment:
"""Comment on an activity, or with `parent` reply on a thread (a reply to a reply goes on the thread too). Anyone
who sees the activity can, up to 1,000 each; everyone who sees it reads them, and each one is a `comment.added`
activity. On an asset's arrival the comment goes on the asset itself."""
activity. On an asset's arrival the comment goes on the asset itself. A comment that tags @lens gets the agent's
answer on the thread."""
return comment(db, cfg, viewer(db, user, acl), user, acl, activity_id, body.text, body.parent)


Expand All @@ -288,7 +290,7 @@ def update_activity_comment(activity_id: str, cid: int, body: CommentUpdate, use
rank = _rank(acl, user, a)
if body.text is None and body.resolved is None:
raise HTTPException(400, "change the text, or resolve or reopen the thread")
if body.text is not None and c["account"] != user.id:
if body.text is not None and (c["account"] != user.id or c.get("author") == "assistant"):
raise HTTPException(403, "only its writer can change a comment")
if body.resolved is not None and c["account"] != user.id and rank < auth.ROLES["editor"]:
raise HTTPException(403, "only its writer or an editor of the namespace can resolve a thread")
Expand Down
21 changes: 12 additions & 9 deletions fastapi_backend/app/api/v1/routes/comments.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,8 @@

from fastapi import APIRouter, HTTPException

from app.api.deps import Access, Acl, CurrentUser, Db, Principal, Writer, domain_errors
from app.domain import auth, comments, highlights, store
from app.api.deps import Access, Acl, Cfg, CurrentUser, Db, Principal, Writer, domain_errors
from app.domain import auth, comment_agent, comments, highlights, store
from app.domain.store import DB
from app.schemas.comments import Comment, CommentCreate, CommentUpdate, Highlight, HighlightCreate, HighlightUpdate
from app.schemas.common import Ok
Expand Down Expand Up @@ -56,7 +56,8 @@ def _comments_out(db: DB, rows: list[dict[str, Any]], rank: int, user: Principal
out = []
for c in rows:
by, res = people.get(c["account"], {}), people.get(int(c["resolved_by"]), {}) if c.get("resolved_by") is not None else {}
mine = c["account"] == user.id
agent = c.get("author") == "assistant" # the agent's answer, for the person it names: theirs to delete, not to edit
mine = c["account"] == user.id and not agent
out.append(
Comment(
id=c["id"],
Expand All @@ -71,14 +72,15 @@ def _comments_out(db: DB, rows: list[dict[str, Any]], rank: int, user: Principal
resolved_by=res.get("email"),
resolved_by_name=res.get("name") or None,
resolved_at=c.get("resolved_at"),
created_by=by.get("email"),
created_by_name=by.get("name") or None,
created_by=None if agent else by.get("email"),
created_by_name="Lens" if agent else by.get("name") or None,
by_agent=agent,
created_at=c.get("created_at"),
updated_at=c.get("updated_at"),
edited_at=c.get("edited_at"),
mine=mine,
can_resolve=c.get("parent") is None and (mine or editor),
can_delete=mine or owner,
can_delete=mine or owner or (agent and c["account"] == user.id),
)
)
return out
Expand All @@ -93,13 +95,14 @@ def list_comments(rid: int, user: CurrentUser, acl: Acl, db: Db) -> list[Comment


@router.post("/comments")
def create_comment(rid: int, body: CommentCreate, user: Writer, acl: Acl, db: Db) -> Comment:
def create_comment(rid: int, body: CommentCreate, user: Writer, acl: Acl, db: Db, cfg: Cfg) -> Comment:
"""Comment on a moment or passage (`t0`–`t1`, with the `quote` picked in the text) or on the whole resource, or
with `parent` reply on a thread (a reply to a reply goes on the thread too). Anyone who can read the resource
can, up to 1,000 each on a resource."""
can, up to 1,000 each on a resource. A comment that tags @lens gets the agent's answer on the thread."""
rec = _recording(acl, db, rid)
with domain_errors():
cid = comments.create(db, rid, rec["space"], user.id, body.text, body.t0, body.t1, body.quote, body.parent, rec.get("duration_ms"))
comment_agent.wake(db, cfg, cid, body.text)
return _comments_out(db, [comments.get(db, cid)], _rank(acl, rec), user)[0]


Expand All @@ -112,7 +115,7 @@ def update_comment(rid: int, cid: int, body: CommentUpdate, user: Writer, acl: A
rank = _rank(acl, rec)
if body.text is None and body.resolved is None:
raise HTTPException(400, "change the text, or resolve or reopen the thread")
if body.text is not None and c["account"] != user.id:
if body.text is not None and (c["account"] != user.id or c.get("author") == "assistant"):
raise HTTPException(403, "only its writer can change a comment")
if body.resolved is not None and c["account"] != user.id and rank < auth.ROLES["editor"]:
raise HTTPException(403, "only its writer or an editor of the resource can resolve a thread")
Expand Down
Loading
Loading
0