8000 feat(budgets): the assistant and agents answer where budgets stand by adeelahmad · Pull Request #645 · adeelahmad/lense · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -238,8 +238,9 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p
- **See what it costs.** An activity ledger of every model call, with reports, charts and budgets that cap spending.
Setup asks for one overall budget after a cloud model, with a soft limit that warns and a hard limit that stops paid
calls. A meter in the top bar shows spend against the budget, and the Costs page breaks it down by day, model and
what cost most. Models are priced by tokens or by the hour, and speech providers by the minute of audio or the
character read aloud. Routine choices go to small decision models, and the LLM only when it adds value.
what cost most; ask the assistant, or an agent over MCP, where budgets stand. Models are priced by tokens or by the
hour, and speech providers by the minute of audio or the character read aloud. Routine choices go to small decision
models, and the LLM only when it adds value.
([Activity and costs](docs/activity.md), [budgets](docs/budgets.md#one-budget-for-everything),
[the meter](docs/budgets.md#the-meter-and-the-costs-page); [checks](docs/requirements.md#see-what-it-costs))
- **Scale out.** Scaling groups add workers when work waits and let them go when it doesn't, by target, step and
Expand Down
4 changes: 4 additions & 0 deletions changelog.d/budgets-tool.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
- **Ask where budgets stand.** The assistant answers "where do my budgets stand?" with a new `budgets` tool, and agents
call it over MCP: what each namespace you can read spent this period against its budget, what's left and whether
it's ok, near or over, and for admins all of Lens against the overall budget and every other budget
([Budgets](docs/budgets.md#asking-the-assistant-or-an-agent)).
2 changes: 2 additions & 0 deletions docs/assistant.md
Original file line number Diff line number Diff line change
Expand Up @@ -310,6 +310,8 @@ When the configured model supports function calling, chat becomes an agent.
Lens in Settings → Sources, Gmail, Google Calendar, Google Drive and S3 among them, and brings what it finds into a
namespace, sends mail, adds events and writes files, each once you approve ([below](#mail-calendars-and-files)).
- **Lens itself:** questions about Lens rather than the archive go to `lens_help` ([below](#questions-about-lens-itself)).
- **Costs and budgets:** "where do my budgets stand?" goes to `budgets`: what each namespace spent this period against
its budget, and for admins all of Lens and every other budget ([Budgets](budgets.md#asking-the-assistant-or-an-agent)).

### Questions about Lens itself

Expand Down
12 changes: 11 additions & 1 deletion docs/budgets.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,8 +86,18 @@ change (admins), and every other budget with how much of it is used.
`GET /api/v1/budgets/meter?ns=&period=&detail=` serves both. Anyone who can read a namespace sees its meter; all of
Lens is admins'. A namespace's spend is the calls that count for it: its jobs, and requests to its own pages.

## Asking the assistant or an agent

Ask the assistant "where do my budgets stand?" or "what have I spent this month?" and it answers with the `budgets`
tool; agents call the same tool over MCP. It gives what each namespace you can read spent this period (its budget's
own day, week or month, else the month) against its budget, with what is left and whether it's ok, near or over. For
admins it adds all of Lens against the overall budget, and every routine's, automation's, pipeline's, workflow's and
scaling group's budget with where it stands. Figures are USD from the activity ledger, and `estimate: true` marks one
that's a floor because a priced call didn't say what it used. It reads only; setting a budget stays with admins in the
web app and `PUT /api/v1/budgets`.

## Refine later

Forecasting from a routine's schedule (its runs left this period), per-person budgets, budget warnings sent through
notifications and the weekly digest, and the assistant answering "where do my budgets stand?" in chat. The meter could
notifications and the weekly digest. The meter could
count chats in a namespace (they aren't attributed to one yet) and show a forecast to the end of the period.
1 change: 1 addition & 0 deletions docs/mcp.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,7 @@ For an app that speaks OpenAI's API rather than MCP, the namespace can also answ
| `find` | assets by any attribute: faces seen, people heard, entities, topics, kind, content type, dates (a day, month or year), tags, language, objects, emotions, length, collection and words in them; ids or names. Each says why it matched, with moments and links to cite ([Finding by any attribute](processing.md#finding-by-any-attribute)) |
| `classify` | which of some labels fits an asset or a text, how sure, and every label ranked: labels you give, or Lens's content types or the namespace's topics; the decision model answers, else the language model. Read only: it changes nothing ([Classifying](processing.md#classifying)) |
| `list_namespaces` | the namespaces you can read, with their assets, hours, people and your role |
| `budgets` | where spending stands: each namespace you can read this period against its budget, and for admins all of Lens and every other budget, each ok, near or over ([Asking the assistant or an agent](budgets.md#asking-the-assistant-or-an-agent)) |
| `list_recordings` | assets, newest first, by namespace, title words, dates, person, tag or kind, and whether they're flagged NSFW (`sensitive`) |
| `list_speakers` | the people of your namespaces, the ones with the most talk time first |
| `get_recording` | one asset: when, how long, people, summary and key points, chapters, the entities and keywords most mentioned, its link, and whether it's flagged NSFW |
Expand Down
1 change: 1 addition & 0 deletions docs/requirements.md
Original file line number Diff line number Diff line change
Expand Up @@ -283,6 +283,7 @@ request, to another check that proves the same thing; loosening a requirement or
| Models priced by tokens or by the hour, and speech providers by the minute heard or the character read | [test_activity.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_activity.py) `test_models_are_costed_by_tokens_by_time_or_not_at_all`; [test_speech.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_speech.py) `test_speech_calls_are_costed_per_minute_heard_and_per_character_read`; [settings-prices.test.ts](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/settings-prices.test.ts) `round-trips every unit` |
| One overall budget, asked after a cloud model, with a soft limit that warns and a hard limit that stops paid calls | [test_budgets.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_budgets.py) `test_one_budget_for_everything_with_a_soft_and_a_hard_limit`; [test_setup.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_setup.py) `test_the_wizard_asks_only_what_has_no_answer` |
| A meter in the top bar; the Costs page by day, model and what cost most | [budget-meter.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/budget-meter.test.tsx) `the top bar's budget meter`; [test_budgets.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_budgets.py) `test_the_meter_follows_the_namespace_and_its_budget` |
| The assistant and agents over MCP answer where budgets stand (`budgets`), each person seeing only their namespaces | [test_budgets.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_budgets.py) `test_agents_ask_where_budgets_stand` |
| The Costs page's charts | No check yet ([#486](https://github.com/adeelahmad/lense/issues/486)) |
| Routine choices go to small decision models, the LLM only when it adds value | [test_decisions.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_decisions.py) `test_the_decision_model_answers_when_it_has_a_key`, `test_the_language_model_decides_without_one_and_when_it_fails` |

Expand Down
17 changes: 17 additions & 0 deletions fastapi_backend/app/api/mcp_tools.py
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@
approvals,
auth,
blobs,
budgets,
chat,
chat_export,
decide,
Expand Down Expand Up @@ -617,6 +618,22 @@ def list_namespaces(ctx: Context) -> dict[str, Any]:
}


@tool(
"budgets",
"Where budgets stand",
"Where spending stands: what each namespace you can read spent this day, week or month (its budget's period, else "
"the month) against its budget, and for admins all of Lens against the overall budget and every other budget "
"(routines, automations, pipelines, workflows, scaling groups) with its state: ok, near or over. Money is USD from "
"the activity ledger; estimate: true means a figure is a floor.",
Arg("namespace", "string", "only this namespace"),
)
def budgets_tool(ctx: Context, namespace: str | None = None) -> dict[str, Any]:
spaces = {ctx.acl.namespace(namespace)} if namespace else ctx.acl.readable()
if spaces is None: # an admin: every namespace
spaces = set(store.space_names(ctx.db))
return budgets.overview(ctx.db, spaces, bool(ctx.user.admin) and not namespace)


SORTS = ("-date", "date", "title", "-title", "duration", "-duration")
MEDIA = ("audio", "video", "transcript", "document", "image")

Expand Down
24 changes: 24 additions & 0 deletions fastapi_backend/app/domain/ai_tools.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@
from . import (
auth,
batches,
budgets,
chat as chatmod,
classify as classifymod,
connection_tools,
Expand Down Expand Up @@ -63,6 +64,17 @@
_S = {"type": "string"}
_LIST = {"type": "array", "items": {"type": ["string", "integer"]}}
TOOLS = [
(
"budgets",
"Where spending stands: what each namespace in scope spent this day, week or month (its budget's period, else "
"the month) against its budget, and for admins all of Lens against the overall budget and every other budget "
"(routines, automations, pipelines, workflows, scaling groups) with its state: ok, near (past its warning share) "
"or over. Money is USD from the activity ledger; estimate: true means a figure is a floor. Use it for questions "
"like where do my budgets stand or what have I spent. namespace: just that one.",
{"namespace": _S},
[],
False,
),
(
"lens_help",
"Lens itself rather than the archive: what it does and how (setting it up, its settings, the assistant, chat "
Expand Down Expand Up @@ -663,6 +675,7 @@ def builtin_names():
"mesh",
"connections",
"lens_help",
"budgets",
} | {
t[0]
for t in TOOLS
Expand Down Expand Up @@ -736,6 +749,7 @@ def builtin_names():
"extensions": {"label": "Extensions and skills", "tools": [t[0] for t in AUTHOR_TOOLS] + ["use_skill"]},
"server": {"label": "The server", "tools": [t[0] for t in ops_tools.ADMIN_TOOLS]},
"help": {"label": "Lens's docs", "tools": ["lens_help"]},
"costs": {"label": "Costs and budgets", "tools": ["budgets"]},
"browser": {"label": "Your browser", "tools": [t[0] for t in browsermod.TOOLS]},
}

Expand Down Expand Up @@ -1673,6 +1687,16 @@ def _about_allowed(self, a):
"""Whether an activity is about the assets of a conversation about some assets (any, in one that isn't)."""
return self.allowed is None or events.ref_id((a.get("object") or {}).get("id"), "recording") in self.allowed

def t_budgets(self, namespace=None):
spaces = self.readable
if namespace:
names = store.space_names(self.db)
spaces = {s for s in self.readable if names.get(s) == namespace}
if not spaces:
raise ValueError(f"namespace {namespace} isn't in this conversation")
view = budgets.overview(self.db, spaces, self.admin and not namespace)
return view, budgets.said(view)

def t_timeline(
self, kinds=None, namespace=None, tag=None, since=None, until=None, q=None, unreviewed=None, order=None, before=None, limit=20
):
Expand Down
73 changes: 73 additions & 0 deletions fastapi_backend/app/domain/budgets.py
Original file line number Diff line number Diff line change
Expand Up @@ -392,3 +392,76 @@ def meter(db, resource, budget_ref=None, period=None, detail=False, now=None):
if resource is None:
out["by_namespace"] = activity.top(db, "space", since, 50)
return out


# ---------- for agents: where it all stands ----------
def _stands(m):
"""A meter as an agent reads it: what was spent this period, and the budget it's measured against."""
st = m.get("budget")
out = {"period": m["period"], "since": m["since"], "spent": m["spent"], "calls": m["calls"], "budget": None}
if st and st.get("budget"):
b = st["budget"]
out["budget"] = {
"usd": b.get("usd"),
"tokens": b.get("tokens"),
"period": b.get("period"),
"state": st["state"],
"share": st["share"],
"left": st.get("left"),
"warn_at": b.get("warn_at"),
}
return out


def overview(db, spaces, admin, now=None, limit=20):
"""Where spending stands, for the assistant's and MCP's `budgets` tools: each namespace in `spaces` (ids the person
can read) against its budget, and for admins all of Lens against the overall budget and every other budget with
where it stands. Money is USD from the activity ledger; `estimate` says a figure is a floor."""
names = store.space_names(db)
out = {
"namespaces": [
{"namespace": names.get(sid), **_stands(meter(db, f"space:{sid}", now=now))}
for sid in sorted(spaces, key=lambda s: names.get(s) or "")[:limit]
]
}
if admin:
out["all"] = _stands(meter(db, None, ALL, now=now))
rows = []
for b in all_budgets(db):
if b["resource"] == ALL or b["resource"].startswith("space:"):
continue # said above
st = status(db, b["resource"], b, now)
table, _, key = b["resource"].partition(":")
row = db.one("SELECT name, title FROM $r", r=R(table, int(key))) if key.isdigit() else None
rows.append(
{
"resource": b["resource"],
"name": (row or {}).get("name") or (row or {}).get("title"),
"usd": b.get("usd"),
"tokens": b.get("tokens"),
"period": b.get("period"),
"spent": st.get("spent"),
"state": st["state"],
"share": st.get("share"),
"left": st.get("left"),
}
)
out["budgets"] = rows
return out


def said(view):
"""The overview in a line: all of Lens (for admins), else the namespaces, and what is near or over."""

def one(m, who):
b, usd = m.get("budget"), m["spent"]["usd"]
spent = f"{'about ' if m['spent'].get('estimate') else ''}${usd:.2f}"
if b and b.get("usd") is not None:
return f"{who}: {spent} of ${b['usd']:g} this {m['period']} ({b['state']})"
return f"{who}: {spent} this {m['period']}" + (f" ({b['state']})" if b else ", no budget")

parts = [one(view["all"], "all of Lens")] if "all" in view else [one(m, m["namespace"]) for m in view["namespaces"][:3]]
tight = [r["name"] or r["resource"] for r in view.get("budgets", []) if r["state"] in ("near", "over")]
if tight:
parts.append(f"near or over: {', '.join(tight[:5])}")
return "Read budgets: " + "; ".join(parts) if parts else "Read budgets: nothing to show"
52 changes: 51 additions & 1 deletion fastapi_backend/tests/api/test_budgets.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,8 @@

import pytest

from app.domain import activity, budgets, decide, jobs, routines, store
from app.domain import activity, ai_tools, budgets, decide, jobs, routines, store
from tests.api.test_mcp import tool
from tests.helpers import drain, login, make_user, seed

R = store.R
Expand Down Expand Up @@ -231,3 +232,52 @@ def test_the_meter_follows_the_namespace_and_its_budget(client, db, h):
assert client.get("/api/v1/budgets/meter", headers=e, params={"ns": "pods"}).json()["spent"]["usd"] == 0.3
assert client.get("/api/v1/budgets/meter", headers=e).status_code == 403
assert client.get("/api/v1/budgets/meter", headers=e, params={"ns": "news"}).status_code == 404


def test_agents_ask_where_budgets_stand(client, db, cfg, h):
sid, other = store.ns_id(db, "pods"), store.ns_id(db, "news")
rid = routine(client, h, "Digest")
activity.record("out", "model.chat", [f"space:{sid}", f"routine:{rid}"], db=db, model="gpt", cost_usd=0.3, tokens_in=100)
activity.record("out", "model.chat", [f"space:{other}"], db=db, model="gpt", cost_usd=2.0, tokens_in=10)
budgets.put(db, "all", usd=5)
budgets.put(db, f"space:{sid}", usd=0.5)
budgets.put(db, f"routine:{rid}", usd=0.35)

# an admin over MCP: all of Lens, each namespace, and every other budget with its state
out = tool(client, h, "budgets")
assert out["all"]["spent"]["usd"] == 2.3 and out["all"]["budget"]["usd"] == 5 and out["all"]["budget"]["state"] == "ok"
pods = next(n for n in out["namespaces"] if n["namespace"] == "pods")
assert pods["spent"]["usd"] == 0.3 and pods["budget"]["left"]["usd"] == 0.2 and pods["budget"]["state"] == "ok"
assert next(n for n in out["namespaces"] if n["namespace"] == "news")["budget"] is None
assert out["budgets"] == [
{
"resource": f"routine:{rid}",
"name": "Digest",
"usd": 0.35,
"tokens": None,
"period": "month",
"spent": {"usd": 0.3, "tokens": 100, "estimate": False},
"state": "near",
"share": 0.8571,
"left": {"usd": 0.05},
}
]
h, "budgets", namespace="news")
assert [n["namespace"] for n in one["namespaces"]] == ["news"] and "all" not in one and "budgets" not in one

# someone who isn't an admin sees only the namespaces they can read
make_user(db, "ed@x.io", "editor password 1", roles={"pods": "editor"})
e = login(client, "ed@x.io", "editor password 1")
mine = tool(client, e, "budgets")
assert [n["namespace"] for n in mine["namespaces"]] == ["pods"] and "all" not in mine and "budgets" not in mine

# the assistant: the same, said in a line
box = ai_tools.Toolbox(db, cfg, {"id": 1, "email": "root@x.io"}, {sid, other}, {sid, other}, {}, None, admin=True)
view, said = box.t_budgets()
assert view["all"]["spent"]["usd"] == 2.3
assert said == "Read budgets: all of Lens: $2.30 of $5 this month (ok); near or over: Digest"
assert "budgets" in {s["function"]["name"] for s in box.specs()}
viewer = ai_tools.Toolbox(db, cfg, {"id": 2, "email": "v@x.io"}, {sid}, set(), {}, None)
view, said = viewer.t_budgets()
assert [n["namespace"] for n in view["namespaces"]] == ["pods"] and "all" not in view
assert said == "Read budgets: pods: $0.30 of $0.5 this month (ok)"
Loading
0