8000 feat(chat): the assistant's own email address, over IMAP and SMTP by adeelahmad · Pull Request #314 · adeelahmad/lense · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
6db7154
feat(chat): built-in Telegram bot, on a shared chat-channel layer
claude Oct 10, 2026
77145b3
Merge main into the Telegram branch: /clear and /compact for every ch…
claude Oct 10, 2026
fec8c3d
Merge main into the Telegram branch
claude Oct 10, 2026
e36eec0
wip: mailbox settings and channel hooks
claude Oct 10, 2026
ed3dee5
Merge main into the agent mailbox branch
claude Oct 10, 2026
c99a052
Merge main (v0.7.0) into the Telegram branch
claude Oct 10, 2026
72f3d8e
feat(chat): the assistant's own email address, over IMAP and SMTP
claude Oct 10, 2026
f0e0cad
Merge remote-tracking branch 'origin/claude/project-thread-cnpj08' in…
claude Oct 10, 2026
1b430fa
Merge remote-tracking branch 'origin/main' into claude/project-thread…
claude Oct 10, 2026
214a720
Merge main into the Telegram branch
claude Oct 10, 2026
59cf675
Merge main into the Telegram branch
claude Oct 10, 2026
91b83f4
Merge remote-tracking branch 'origin/claude/project-thread-cnpj08' in…
claude Oct 10, 2026
87f8648
Merge main into the Telegram branch
claude Oct 10, 2026
affe178
Merge main into the agent mailbox branch: forwarded mail counts as ou…
claude Oct 10, 2026
6841e34
Merge main into the Telegram branch
claude Oct 10, 2026
43ad6fc
Merge remote-tracking branch 'origin/claude/project-thread-cnpj08' in…
claude Oct 11, 2026
99d2a77
Merge main into the agent mailbox branch: safe links and injection flags
claude Oct 11, 2026
14d9948
Merge main into the agent mailbox branch
claude Oct 11, 2026
84c414c
Merge main into the Telegram branch
claude Oct 11, 2026
1dbe789
ci(clicks): switch to a cloud model through the presets
claude Oct 11, 2026
2842ac4
Merge main into the Telegram branch (clicks fix #350)
claude Oct 11, 2026
2efedec
Merge remote-tracking branch 'origin/pr293' into claude/project-threa…
claude Oct 11, 2026
8000
231309c
Merge main (Telegram #293) into the agent mailbox branch
claude Oct 11, 2026
0442f6e
Merge main (Slack #287) into the agent mailbox branch
claude Oct 11, 2026
568ae57
Merge main (README #351) into the agent mailbox branch
claude Oct 11, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,13 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version
([Chat rooms](docs/chat-rooms.md#telegram-built-in)). Chat apps now share one channel layer, so the next one is an
adapter.

- **The assistant by email.** Give it an address of its own in Settings › Assistant email (the address and an app
password; the servers are guessed) and the people you let in can email or forward things to it: it answers by
email in the same thread, lists the excerpts it cites with links, and imports what's attached, a forwarded email
whole. It answers only mail their provider vouched for (DMARC, DKIM or SPF), never automatic mail, and holds a
stranger's mail until you let them in. `lens+pods@…` or a routing rule sends mail to a namespace's assistant
([The assistant by email](docs/assistant-email.md)).

## 0.7.0 <small>October 10, 2026</small> {id="0.7.0"}

- **Prometheus metrics.** Lens can serve `/metrics` for Prometheus and Grafana: jobs by status and pipeline, imports,
Expand Down
14 changes: 7 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,9 +67,10 @@ Everything below is built and merged. Work that is still in a pull request is li
meaning, once turned on), reads it, cites it and acts on it with tools: changes wait for your approval. It reads
answers aloud with a speech provider, macOS `say` or StyleTTS2 on this machine (opt-in), and `/clear` and
`/compact` keep long conversations short. Each namespace can have its own assistant with its own name, instructions
and memory, answering in Telegram and Slack (built in, with voice notes and files), and in Discord, Matrix,
WhatsApp and other chat rooms through Matterbridge.
([Chat](docs/assistant.md), [chat rooms](docs/chat-rooms.md), [speech providers](docs/speech-providers.md))
and memory, answering in Telegram and Slack (built in, with voice notes and files), in Discord, Matrix, WhatsApp
and other chat rooms through Matterbridge, and by email at an address of its own that you email or forward things
to. ([Chat](docs/assistant.md), [chat rooms](docs/chat-rooms.md), [email](docs/assistant-email.md),
[speech providers](docs/speech-providers.md))
- **Extend the assistant.** Add tools (a prompt, a web request, a graph drawn on the canvas, or sandboxed Python),
skills, hooks and remote MCP servers, by code, on the canvas or just by asking. Install skills and plugins from
marketplaces in the open plugin format, or from a repository or an archive.
Expand Down Expand Up @@ -140,8 +141,6 @@ lives in the web app. ([Components](docs/components.md), [speech providers](docs

These are open pull requests, not yet part of Lens:

- **Email:** the assistant's own email address, over IMAP and SMTP
([#314](https://github.com/adeelahmad/lense/pull/314)).
- **Assistant safety:** running MCP servers that are programs (stdio) in a sandbox
([#304](https://github.com/adeelahmad/lense/pull/304)).
- **Search:** a Similar tab, and similar content for text selected anywhere
Expand Down Expand Up @@ -180,8 +179,9 @@ The guides are on the [docs site](https://adeelahmad.github.io/lense/) and in th
[updates and backups](docs/updates.md), [remote access](docs/remote-access.md), [HTTPS](docs/https.md)
- **Agents and apps:** [MCP server](docs/mcp.md), [API](docs/api.md), [authentication](docs/authentication.md),
[access](docs/access.md)
- **Use:** [chat and batch runs](docs/assistant.md), [chat rooms](docs/chat-rooms.md), [podcasts](docs/podcasts.md),
[notes](docs/notes.md), [web clipper](docs/web-clipper.md),
- **Use:** [chat and batch runs](docs/assistant.md), [chat rooms](docs/chat-rooms.md),
[the assistant by email](docs/assistant-email.md), [podcasts](docs/podcasts.md), [notes](docs/notes.md),
[web clipper](docs/web-clipper.md),
[the assistant in your browser](docs/browser-assistant.md), [sensors](docs/sensors.md),
[processing](docs/processing.md), [video](docs/video.md), [notifications](docs/notifications.md)
- **Knowledge:** [graph and Cypher](docs/graph.md), [graph history](docs/graph-history.md), [topics](docs/topics.md),
Expand Down
88 changes: 88 additions & 0 deletions docs/assistant-email.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
# The assist 8BD0 ant by email

Give the assistant an email address of its own, and the people you let in can email or forward things to it: it
answers by email, in the same thread, and imports what's attached. Lens reads the mailbox over IMAP and sends over
SMTP, so it needs no public address and no mail server of its own.

## Set it up

1. Make an address for the assistant at any provider (for example `lens@yourdomain` or a new Gmail or Fastmail
account). Most providers give an **app password** for programs like Lens: use that, not your sign-in password.
2. Open **Settings → Assistant email**, enter the address and the password, and turn on **Answer email**.
3. Email the assistant. You show on the settings page under **Wrote to it, not let in**: click **Let in**, and that
email is answered.

**Check the mailbox** signs in to read and to send, without reading or sending anything, and says what's wrong if it
can't. The mail servers are guessed from the address (Gmail, Outlook.com, iCloud, Fastmail, Yahoo, GMX, Zoho,
mailbox.org and Yandex are known; anything else is tried as `imap.<domain>` and `smtp.<domain>`); set them under
**Mail servers** when the guess is wrong. Mail that arrived before it was turned on isn't answered.

An agent sets it up the same way: `PUT /api/v1/settings/mailbox` with `address`, `password` and `enabled`, then
`POST /api/v1/settings/mailbox/test`; `GET /api/v1/settings/mailbox` says how it's doing and who has written. The
assistant itself can change these settings when you approve it (`change_settings mailbox`).

## Who it answers

Anyone can send email to an address, and anyone can write any address in From, so it answers only mail that is both:

- **from someone let in**: an address in **People it answers** (one click on the settings page), and
- **vouched for by their provider**: the server that received it for the mailbox found that DMARC passed, or DKIM or
SPF passed for the sender's domain (its `Authentication-Results` header). Gmail, Outlook, iCloud, Fastmail and the
other big providers add that header to every message.

Nothing else is answered, and nothing tells a stranger the address is read: someone not let in shows on the settings
page, and their last emails wait there until you let them in. Auto-replies, bounces, mailing lists and its own mail
are never answered. Mail that will be answered is marked read in the mailbox; anything else stays unread.

With a mail server of your own: if it doesn't remove `Authentication-Results` headers that arrive with a message, set
**Trust checks by** to the name it signs its own with, so a forged one isn't believed. If it adds none but refuses
forged mail itself, **Whose mail counts → Trust the From address** answers by From alone.

## What it does with an email

Each email thread is a conversation in Chat for the account it answers as (**Answers as**, the admin who turned it
on unless you choose another), so it reads what that account can read. A new email's question is its subject and
what it says; a reply's question is what was written above the quoted part. Its answer lists the excerpts it cites,
each with a link to that moment or page in Lens, quotes your question under it, and is plain text: nothing in it
loads from another server when you read it.

- **Attachments** are imported at once (**What's attached → Imported at once**) into the namespace the assistant
picks, if that namespace exists; one it would have to create waits for your approval, as does everything with
**Imported once approved**.
- **A forwarded email** is imported whole, as an email with its attachments. The assistant also reads its text, as
something someone else wrote: it is marked as data, so instructions written in it aren't followed
([third-party content](assistant.md#third-party-content-and-prompt-injection)), and once it has read it, its
changes wait for your approval, the import too.
- **Approving by email**: reply "yes" (or "no", "yes 2", "yes all") to what it proposes, for entity merges and edits
and new namespaces; settings, runs and extensions are approved in the web app, linked from the answer.
- `/clear` and `/compact` as an email's text work as in Chat.

## Which namespace

Email to the address with a namespace after a `+` (`lens+pods@yourdomain`, where the provider delivers plus
addresses) goes to that namespace's assistant. Otherwise the first rule in **Where mail goes** that matches decides,
one per line:

```text
from: *@acme.com = acme
subject: invoice; from: billing@x.com, accounts@x.com = finance
subject: newsletter = skip
```

A rule's conditions are on `from`, `to` (To, Cc and the address it arrived at), `subject`, `list` (List-Id) and
`mailbox`, joined by `;`, with patterns joined by `,`. Patterns work as in an email watch's routing rules
([Processing](processing.md#email-imap-and-calendar-feeds-ical)): one without `*` or `?` matches anywhere in the
text. `skip` leaves the email unanswered. When no rule matches, or a rule names a namespace that doesn't exist, Lens
picks the namespace for each question, as in an open chat room ([Chat rooms](chat-rooms.md)).

## How it works

The mailbox is an adapter of the chat-channel layer (`app/domain/channels.py`), next to Matterbridge and Telegram: one
server process at a time reads it, every **Looks for new mail every** seconds (30 by default), and what happens to a
message (the conversation, routing, approvals, files) is the same as in a chat room. The reading is the IMAP code of
email sources (`feeds.py`); answers go out over SMTP with `In-Reply-To`, `References` and
`Auto-Submitted: auto-replied`, so they stay in the thread and other robots don't answer them. Model calls are in the
activity ledger like any chat's.

Refine later: IMAP IDLE instead of looking every few seconds, a copy of answers in Sent, finding the servers of more
providers (autoconfig), letting in whole domains, and answers read aloud as audio attachments.
3 changes: 2 additions & 1 deletion docs/chat-rooms.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,8 @@ namespace's assistant with **Chats for a namespace's assistant** (`-100123456789
Give Lens a bot of its own: a bot that Matterbridge also reads answers neither reliably (**Check the bot** says so).

Telegram is one adapter of the chat-channel layer (`app/domain/channels.py`) that Matterbridge is another of; a new
chat app adds an adapter (reading, sending, files) and gets everything in a room for free.
chat app adds an adapter (reading, sending, files) and gets everything in a room for free. The assistant's own email
address is one too ([The assistant by email](assistant-email.md)).

## Slack, built in

Expand Down
19 changes: 18 additions & 1 deletion fastapi_backend/app/api/v1/routes/admin.py
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@
llm,
local_llm,
machine,
mailbox,
matterbridge,
monitoring,
nfs_share,
Expand Down Expand Up @@ -56,6 +57,7 @@
IndexQueued,
LlmTestResult,
LocalLlmStatus,
MailboxStatus,
MailTestResult,
NfsStatus,
Removed,
Expand Down Expand Up @@ -91,7 +93,7 @@ def update_settings(section: str, body: dict[str, Any], user: AdminWriter, reque
if "*" not in hosts and here not in hosts:
raise HTTPException(400, f"that list leaves out {here}, the address you're using, and would lock you out")
if (
section in ("bridge", "telegram", "slack")
section in ("bridge", "telegram", "slack", "mailbox")
and body.get("enabled")
and "account" not in body
and not (request.app.state.archive.current().get(section) or {}).get("account")
Expand Down Expand Up @@ -252,6 +254,21 @@ def test_slack(user: AdminWriter, cfg: Cfg, db: Db) -> BridgeTestResult:
return BridgeTestResult(ok=error is None, error=error)


@router.get("/settings/mailbox")
def mailbox_status(user: AdminReader, cfg: Cfg, db: Db) -> MailboxStatus:
"""How the assistant's mailbox is doing, and who has written to it (whether each is let in, and their mail vouched
for by their provider)."""
return MailboxStatus(**mailbox.status(db, cfg))


@router.post("/settings/mailbox/test")
def test_mailbox(user: AdminWriter, cfg: Cfg, db: Db) -> BridgeTestResult:
"""Check that Lens can sign in to read the mailbox (IMAP) and to send from it (SMTP), and that the account to
answer as exists. Nothing is read or sent."""
error = mailbox.MAILBOX.check(db, cfg)
return BridgeTestResult(ok=error is None, error=error)


@router.post("/settings/documents/node/test")
def test_conversion_node(user: AdminWriter, cfg: Cfg) -> ConversionNodeTestResult:
"""Check that the conversion node (documents.anytopdf_url) answers at its address and takes Lens's token."""
Expand Down
23 changes: 19 additions & 4 deletions fastapi_backend/app/domain/ai_tools.py
Original file line number Diff line number Diff line change
Expand Up @@ -349,19 +349,34 @@ def read_before(history, summary=None):

class Toolbox(ops_tools.OpsTools):
"""`admin` adds the server tools (ops_tools.py), with `base` (archive.yaml's config) to save settings over; `act`
makes their changes at once instead of asking for approval. `browser`: the person's connected side panel
(browser.py), whose browser the assistant can then use."""
makes their changes at once instead of asking for approval, `act_on` only those of these tools (a chat channel's:
channels.Channel.acts). `browser`: the person's connected side panel (browser.py), whose browser the assistant can
then use."""

def __init__(
self, db, cfg, user, readable, editable, scope, chat_id, base=None, admin=False, act=False, said="", outside=(), browser=None
self,
db,
cfg,
user,
readable,
editable,
scope,
chat_id,
base=None,
admin=False,
act=False,
said="",
outside=(),
browser=None,
act_on=(),
):
self.db, self.cfg, self.user, self.chat, self.said = db, cfg, user, chat_id, said
# where this answer has read text from outside Lens so far (`outside`: before any tool ran, such as the page it
# was asked from or a chat room's message): once it has, tools that send data out of Lens or change the server
# ask first, even where they'd act at once (wary)
self.outside = list(dict.fromkeys(outside))
self.links = set() # web addresses the tools gave back, as written (untrusted.defang)
self.base, self.admin, self.act = base or cfg, admin, act
self.base, self.admin, self.act, self.act_on = base or cfg, admin, act, set(act_on or ())
self.scope = scope or {}
self.readable, self.editable = set(readable), set(editable)
names = store.space_names(db)
Expand Down
43 changes: 32 additions & 11 deletions fastapi_backend/app/domain/bridge.py
Original file line number Diff line number Diff line change
Expand Up @@ -275,11 +275,15 @@ def route(db, cfg, account, cid, q, ns=None):


# ---------- answering ----------
def answer(db, cfg, base, account, cid, q, files=()):
def answer(db, cfg, base, account, cid, q, files=(), act_on=(), outside=None):
"""Answer a question in a conversation as the account would be answered in the web app (without streaming), with
the files sent with it (held uploads: channels.hold).
Returns (text, approvals): the text as the room gets it, its links safe to preview (`safe_links`)."""
text, approvals, keep = _answer(db, cfg, base, account, cid, q, files)
the files sent with it (held uploads: channels.hold); `act_on` are the changes made at once (Channel.acts), and
`outside` where the question's text came from outside Lens (Channel.outside, a Matterbridge room's by default), so
that changes ask first. Returns (text, approvals): the text as the room gets it, its links safe to preview
(`safe_links`)."""
if outside is None:
outside = MATTERBRIDGE.outside(cfg, {})
text, approvals, keep = _answer(db, cfg, base, account, cid, q, files, act_on, outside)
safe = safe_links(cfg, db, text, keep)
if safe != text: # what an injection may have tried, for admins to see
broken = len(untrusted.links(text) - untrusted.links(safe))
Expand All @@ -298,7 +302,7 @@ def safe_links(cfg, db, text, keep=()):
return untrusted.defang(text, keep, [app_url(cfg, db)])


def _answer(db, cfg, base, account, cid, q, files=()):
def _answer(db, cfg, base, account, cid, q, files, act_on, outside):
roles = auth.roles(db, account)
readable = set(roles)
editable = {s for s in roles if auth.allows(roles, s, "editor")}
Expand All @@ -317,8 +321,6 @@ def _answer(db, cfg, base, account, cid, q, files=()):
steps, approvals, notice = [], [], None
wrote = cfg["llm"].get("model") if llm.configured(cfg) else None
if llm.configured(cfg) and cfg["ai"].get("tools"):
# anyone in a room with no list of who may talk to Lens is someone the account doesn't vouch for
anyone = not [u for u in cfg["bridge"].get("users") or [] if u.strip()]
box = ai_tools.Toolbox(
db,
cfg,
Expand All @@ -330,7 +332,8 @@ def _answer(db, cfg, base, account, cid, q, files=()):
base,
account["admin"],
said=q,
outside=(["chat room"] if anyone else []) + ai_tools.read_before(past, summary),
outside=list(outside) + ai_tools.read_before(past, summary),
act_on=act_on,
)
try:
text = None
Expand Down Expand Up @@ -446,6 +449,23 @@ def reply_text(cfg, text, approvals, cid, db=None):
return out


def sources(db, cfg, cid):
"""The excerpts the conversation's last answer cites, a line each with its link into the web app, for a channel
that shows them under the answer (Channel.sources); "" when it cites none."""
from app.email import app_url

row = db.one("SELECT id, passages FROM chat_message WHERE chat = $c AND role = 'assistant' ORDER BY id DESC LIMIT 1", c=cid) or {}
used = [p for p in row.get("passages") or [] if isinstance(p, dict) and p.get("used") and p.get("recording_id")]
if not used:
return ""
base, lines = app_url(cfg, db), []
for p in used:
at = f"?page={p['page'] + 1}" if p.get("page") is not None else f"?t={int((p.get('t0') or 0) // 1000)}"
CAA1 where = ", ".join(str(x) for x in (p.get("title") or "Untitled", p.get("time")) if x)
lines.append(f"[{p['n']}] {where}: {base}/resources/{p['recording_id']}{at}")
return "Sources:\n" + "\n".join(lines)


def _media(db, cfg, ch, m):
"""A message's voice note as its text (when it has none and is for Lens), and a note when there's nothing to say
but files: the message as handle() reads it, and what to say instead of answering (or None)."""
Expand Down Expand Up @@ -484,7 +504,7 @@ def handle(db, cfg, base, m, ch=None):
ch.send(cfg, m, cant)
return True
return False
ns = room_namespace(cfg, m)
ns = m.get("namespace") or room_namespace(cfg, m)
sid = {v: k for k, v in store.space_names(db).items()}.get(ns) if ns else None
home = ns_assistant.profile(db, sid) if sid is not None else None
by_name = named(db, cfg, m) if sid is None else None
Expand Down Expand Up @@ -518,15 +538,16 @@ def handle(db, cfg, base, m, ch=None):
chat.add(db, cid, "assistant", said)
ch.send(cfg, m, said)
return True
text, approvals = answer(db, cfg, base, account, cid, q[:MAX_CHARS], files)
text, approvals = answer(db, cfg, base, account, cid, q[:MAX_CHARS], files, ch.acts(cfg), ch.outside(cfg, m))
except Exception: # noqa: BLE001 - still say something in the room
log.exception("%s: answering failed", ch.key)
text, approvals, note = "Something went wrong while answering. Try again.", [], None
out = reply_text(cfg, text, approvals, cid, db)
if m.get("heard"): # what the voice note was heard as, so a mishearing shows
out = f"(Heard: “{q}”)\n\n{out}"
out = "\n".join([*failed, out]) if failed else out
ch.send(cfg, m, out + (f"\n\n{note}" if note else ""))
cites = sources(db, cfg, cid) if ch.sources else ""
ch.send(cfg, m, out + (f"\n\n{cites}" if cites else "") + (f"\n\n{note}" if note else ""))
try: # a room conversation goes on for good: older messages are folded into its summary
chat.compact(db, cfg, cid)
except Exception: # noqa: BLE001 - tried again after the next answer
Expand Down
Loading
Loading
0