8000 feat(assistant): run Python and bash in an OpenShell sandbox per conversation by adeelahmad · Pull Request #473 · adeelahmad/lense · GitHub
[go: up one dir, main page]

Skip to content 7FFF
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,12 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version
- **Describe an automation in words.** `POST /api/v1/automations/draft` asks the model once to turn a sentence into
a draft automation to try and save, and says what didn't fit and what it couldn't express
([Describe it in words](docs/automations.md#describe-it-in-words)).
- **The assistant runs Python and bash.** Turn on Settings → Code sandbox and point Lens at an NVIDIA OpenShell
gateway, and the assistant gets `run_python`, `run_bash`, `sandbox_files` and `import_sandbox_file`: code runs in a
sandbox of the conversation's own, with none of Lens's data or keys and no network unless you allow the package
registries or the public web. Files attached to the conversation go in; a file it makes reaches a namespace once the
person approves. Off by default; starting the gateway from Settings is in progress
([Code sandbox](docs/assistant.md#code-sandbox)).

## 0.13.0 <small>October 16, 2026</small> {id="0.13.0"}

Expand Down
53 changes: 52 additions & 1 deletion docs/assistant.md
Original file line number Diff line number Diff line change
Expand Up @@ -151,6 +151,8 @@ When the configured model supports function calling, chat becomes an agent.
- **Your browser** (Chrome and Edge): asked from the Lens extension's side panel, the assistant can open, read, click
and type in that browser and download files into the conversation, on sites you allow, asking before anything that
sends, deletes or buys. See [The assistant in your browser](browser-assistant.md).
- **Python and bash** (off until an admin turns it on): `run_python` and `run_bash` run code in a sandbox of the
conversation's own, to compute, analyse or convert files. See [Code sandbox](#code-sandbox).
- **The web:** when an admin turns on web search, `web_search` and `web_read` look up what the archive doesn't hold,
cited like excerpts ([below](#web-search-and-reading-pages)).

Expand Down Expand Up @@ -200,6 +202,53 @@ gets two more tools, for everyone who may use the assistant:
Refine later: an opt-in SearXNG service in the Docker Compose file (SearXNG is AGPL, so it would stay an extra),
prices per search for the paid providers, web access per namespace, and keeping recent results to search less.

## Code sandbox

Built: an admin turns it on in Settings → Code sandbox (`sandbox.enabled`, off by default) and points Lens at an
[NVIDIA OpenShell](https://github.com/NVIDIA/OpenShell) gateway (`sandbox.gateway`, `openshell-gateway:8080` by
default). Then the assistant can run Python and bash for the people it's turned on for: those who can edit a namespace,
and admins (`sandbox.who`: `editors`), or admins only (`admins`). Viewers never get it.

- **One sandbox per conversation.** The first run in a conversation creates its sandbox, `lens-chat-<id>`, from
`sandbox.image` (`python:3.12-slim` by default, about 45 MB to download) with `sandbox.cpu` (1 core) and
`sandbox.memory` (512Mi). It keeps its files between runs in that conversation, and no other conversation sees them.
A sandbox nobody ran code in for `sandbox.idle_minutes` (15) is stopped; the next run starts it again, files and
all. One unused for `sandbox.keep_hours` (24) is deleted with its files, and deleting the conversation deletes its
sandbox at once. Changing the image or the network makes the conversation's next run start a fresh sandbox; CPU and
memory apply to new sandboxes.
- **The tools.** `run_python` (code) and `run_bash` (a script) each run for at most `sandbox.seconds` (60) and give
back stdout, stderr and the exit code: the model reads the start of stdout and the end of stderr, up to 10,000
characters, and is told when more was cut off. `files` copies files attached to the conversation in first, by their
own names (25 MB each, 100 MB a run). `sandbox_files` lists what the sandbox holds. `import_sandbox_file` copies a
file out (25 MB at most) as if the person had attached it and puts it in a namespace with `import_files`, so it
waits for their approval; its type must be one uploads take.
- **Nothing of Lens's inside.** A sandbox holds no keys, no database address and no archive: only the code, and the
files a run copies in. What it prints reaches the model marked as third-party text, like any tool's result.
- **Its network** (`sandbox.network`): `off` (the default) allows no traffic in or out at all; `packages` reaches
only the Python and npm registries (pypi.org, files.pythonhosted.org, registry.npmjs.org), so `pip install` works;
`web` reaches public addresses on ports 80 and 443, never private ones (Lens's own services, your home network, the
cloud metadata address). With the network on, a run after the answer read text from outside Lens waits for the
person's approval (below), since code could carry what it read out.
- **How it's kept apart.** OpenShell runs each sandbox as a container under its supervisor: the code runs as an
unprivileged user with no capabilities, Landlock lets it write only its working folder and `/tmp` and read the
image's programs, seccomp filters its system calls, and its egress is denied unless the policy above allows it.
- **Cost.** It makes no model calls of its own, and the compute is your machine's. Each run is a row in the
[activity ledger](activity.md) (`sandbox.run`, with its time, language and exit code), counted for the conversation.
- **The gateway.** It needs Docker 28 or newer and a Linux kernel with Landlock (6.2 or newer, with Landlock turned
on); it runs on Linux on amd64 and arm64 and on macOS on Apple silicon. Lens's own client for it (the `sandbox`
Python extra, OpenShell's SDK, Apache-2.0) is fetched once the sandbox is on ([Components](components.md)).

In progress: starting the OpenShell gateway from Settings (a `sandbox` profile of the Docker Compose install, turned
on like monitoring) with a check that says why it can't run on a machine; until then, run a gateway yourself on a
network only Lens reaches and set `sandbox.gateway`. Also in progress: the step in the conversation showing the code
(folded) and its output, and handing a file back as a download on the answer.

Refine later: automations' agent runs without a conversation get no sandbox yet; running admins' Python extension
tools in the sandbox so people who aren't admins can write them; installs in a [microsandbox](microsandbox.md) (the
gateway on the host, or OpenShell's MicroVM driver); the Podman driver for hosts without Docker; GPUs; a direct sandbox
API for agents (today an agent asks the assistant); one real run against OpenShell in CI (the tests use a fake
gateway with OpenShell's own client).

## Third-party content and prompt injection

The assistant reads text nobody vouched for: what was said in recordings, documents, text read off screens, web pages
Expand Down Expand Up @@ -237,7 +286,9 @@ apart from what the person asks:
other than Lens's own, the page it was asked from, the namespace assistant's memories, a message in a chat room
open to anyone, or an earlier answer in the conversation that read any of these), anything that could carry what
it read out of Lens waits for your approval, even where it would otherwise run at once: your own `read` web tools,
a [web search or page read](#web-search-and-reading-pages) you or a tool didn't give it, other servers' MCP tools and Python tools with the network, the changes a setup conversation makes itself, and, in
a [web search or page read](#web-search-and-reading-pages) you or a tool didn't give it, other servers' MCP tools
and Python tools with the network, code in the [code sandbox](#code-sandbox) when its network is on, the changes a
setup conversation makes itself, and, in
[your browser](browser-assistant.md), opening or downloading an address or typing text you didn't write. The
approval card says why ("asked first: this answer read text from outside Lens: tool search_transcripts"). Notes and
memories are still written at once: they stay in Lens, and are marked as data when read back.
Expand Down
1 change: 1 addition & 0 deletions docs/components.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ never fetch a speech or voice model.
| DuckDuckGo search | `web.search` is `duckduckgo` | ddgs (MIT, about 30 MB with lxml and primp), the client the assistant's [web search](assistant.md#web-search-and-reading-pages) reads DuckDuckGo's results with |
| yt-dlp | `documents.video_urls` is on | the yt-dlp program, release 2026.08.19 (Unlicense), to download videos and podcasts imported by address |
| Chat model, embedding model | the LLM provider or embeddings server is Ollama | pulled on that server (`/api/pull`) |
| Code sandbox client | the [code sandbox](assistant.md#code-sandbox) is on (`sandbox.enabled`) | OpenShell's Python SDK (Apache-2.0) and gRPC, about 10 MB, to reach its gateway |
| Outlook .msg emails | listed in `components.also` | extract-msg (GPL-3.0, so only when asked for) |

- Python packages go into `data_dir/python`, at the versions in `uv.lock`. Packages the image already has win.
Expand Down
21 changes: 20 additions & 1 deletion docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -107,7 +107,8 @@ Start from `fastapi_backend/archive.example.yaml`, which documents every key. Th
## Settings in the app

Admins can change, in Settings: transcription, speaker separation, speech providers, voice IDs, analysis, the LLM
provider and key, a local model, the AI assistant, search (word search and search by meaning), reports and graph,
provider and key, a local model, the AI assistant, the [code sandbox](assistant.md#code-sandbox), search (word
search and search by meaning), reports and graph,
video, OCR, faces and objects, [NSFW classification](nsfw.md), workers, sign-in, components (and the edition, full or
lean), access and embedding (whether other devices on the network reach Lens, embed frame ancestors, text and transcript
upload limit, allowed hosts, trusted proxies, session length), remote access, email, chat rooms, notifications,
Expand Down Expand Up @@ -492,6 +493,24 @@ follow ([Automations](automations.md#contract)), their timers, and how long thei
| `events.poll_seconds` | 5 | how often followers look for new activities when nothing wakes them, 1–3600 |
| `events.keep_days` | 30 | how long activities with no row of their own (the event log) are kept, 1–3650 days |

## Code sandbox

Where the assistant runs Python and bash ([Code sandbox](assistant.md#code-sandbox)). Settings → Code sandbox; the
image, CPU, memory, run time and how long sandboxes are kept are under its fold:

| Setting | Default | |
|---|---|---|
| `sandbox.enabled` | false | on: the assistant gets `run_python`, `run_bash`, `sandbox_files` and `import_sandbox_file` |
| `sandbox.who` | `editors` | who may use it: `editors` (people who can edit a namespace, and admins) or `admins` |
| `sandbox.network` | `off` | `off` (no traffic), `packages` (the Python and npm registries) or `web` (public addresses on ports 80 and 443) |
| `sandbox.gateway` | `openshell-gateway:8080` | the OpenShell gateway's gRPC address, `host:port` |
| `sandbox.image` | `python:3.12-slim` | the container image sandboxes are made from; it needs `python3` and `bash` |
| `sandbox.cpu` | `1` | cores a sandbox may use, like `1`, `0.5` or `500m` (0.1–64) |
| `sandbox.memory` | `512Mi` | memory a sandbox may use, like `512Mi` or `2Gi` (64Mi–64Gi) |
| `sandbox.seconds` | 60 | the longest one run may take, 1–600 seconds |
| `sandbox.idle_minutes` | 15 | a sandbox nobody ran code in for this long is stopped (its files stay), 1–1440 |
| `sandbox.keep_hours` | 24 | one unused for this long is deleted with its files, 1–720 |

## Trusted proxies

IP groups ([Access](access.md#ip-groups)) match the visitor's address. The server takes it from the connection or,
Expand Down
6 changes: 4 additions & 2 deletions fastapi_backend/app/api/v1/routes/chats.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
from app.api.deps import Access, Acl, Cfg, CurrentUser, Db, Principal, Writer
from app.api.media import sign_urls
from app.api.v1.routes.collections import own_collection
from app.domain import ai_tools, approvals, auth, auto_scope, browser, chat, llm, store, uploads
from app.domain import ai_tools, approvals, auth, auto_scope, browser, chat, llm, sandbox, store, uploads
from app.domain.store import DB
from app.schemas.chats import (
AnswerCheck,
Expand Down Expand Up @@ -165,9 +165,11 @@ def update_chat(cid: int, body: ChatUpdate, user: Writer, acl: Acl, db: Db, cfg:


@router.delete("/chats/{cid}")
def delete_chat(cid: int, user: Writer, db: Db) -> Ok:
def delete_chat(cid: int, user: Writer, db: Db, cfg: Cfg) -> Ok:
"""Delete a conversation, with its code sandbox and the files in it."""
_own_chat(db, cid, user)
db.run(["DELETE chat_message WHERE chat = $c", "DELETE $r"], c=cid, r=R("chat", cid))
sandbox.forget(db, cfg, cid)
return Ok()


Expand Down
73 changes: 70 additions & 3 deletions fastapi_backend/app/domain/ai_tools.py
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@
ops_tools,
recsets,
render,
sandbox as sandboxmod,
search as searchmod,
browser as browsermod,
similar,
Expand Down Expand Up @@ -459,7 +460,16 @@

def builtin_names():
"""The assistant's own tool names, which extensions can't take."""
every = TOOLS + ops_tools.ADMIN_TOOLS + ops_tools.FILE_TOOLS + AUTHOR_TOOLS + MEMORY_TOOLS + AUTOMATION_TOOLS + browsermod.TOOLS
every = (
TOOLS
+ ops_tools.ADMIN_TOOLS
+ ops_tools.FILE_TOOLS
+ AUTHOR_TOOLS
+ MEMORY_TOOLS
+ AUTOMATION_TOOLS
+ browsermod.TOOLS
+ sandboxmod.TOOLS
)
return {t[0] for t in every + webmod.TOOLS} | {"use_skill"}


Expand All @@ -480,7 +490,11 @@ def builtin_names():
"use_skill",
"mark_activities",
"decline_approval",
} | {t[0] for t in TOOLS + ops_tools.ADMIN_TOOLS + ops_tools.FILE_TOOLS + AUTHOR_TOOLS + MEMORY_TOOLS + AUTOMATION_TOOLS if t[4]}
} | {
t[0]
for t in TOOLS + ops_tools.ADMIN_TOOLS + ops_tools.FILE_TOOLS + AUTHOR_TOOLS + MEMORY_TOOLS + AUTOMATION_TOOLS + sandboxmod.TOOLS
if t[4]
}


def read_before(history, summary=None):
Expand All @@ -501,7 +515,8 @@ class Toolbox(ops_tools.OpsTools):
"""`admin` adds the server tools (ops_tools.py), with `base` (archive.yaml's config) to save settings over; `act`
makes their changes at once instead of asking for approval, `act_on` only those of these tools (a chat channel's:
channels.Channel.acts). `browser`: the person's connected side panel (browser.py), whose browser the assistant can
then use. `account`: the person an automation acts for, when `user` is the automation itself (automation:<id>,
then use. The conversation's code sandbox (sandbox.py) is there when an admin turned it on and the person may use it.
`account`: the person an automation acts for, when `user` is the automation itself (automation:<id>,
the by of what it writes): their approvals, extensions, view of the timeline and marks on it; `origin` ({automation, run,
node}) is kept on the approvals it asks for."""

Expand Down Expand Up @@ -550,12 +565,16 @@ def __init__(
# the browser the person is talking from, while its side panel is connected (and an admin hasn't turned it off)
on = browser and browsermod.enabled(cfg) and browsermod.live(db, browser, user["id"])
self.browser = str(browser) if on else None
# Python and bash in the conversation's own sandbox, once an admin turned it on (sandbox.enabled, sandbox.who)
self.sandbox = chat_id is not None and sandboxmod.offered(cfg, admin, bool(self.editable))
self.web = webmod.enabled(cfg) # web search and page reading, once an admin picked a provider (web.py)

def system_note(self):
"""What the model is told besides its usual instructions: the skills it can follow, and context that hooks add
for this question."""
out = self._home_note() + (browsermod.NOTE if self.browser else "")
if self.sandbox:
out += sandboxmod.NOTE.format(network=sandboxmod.NETWORK_NOTE[sandboxmod.settings(self.cfg)["network"]])
if self.web and webmod.NAMES & {s["function"]["name"] for s in self.specs()}:
out += webmod.NOTE
if not self.ext:
Expand Down Expand Up @@ -629,6 +648,8 @@ def specs(self):
tools += [t for t in MEMORY_TOOLS if self.home[0] in self.editable or not t[4]]
if self.browser:
tools += browsermod.TOOLS
if self.sandbox:
tools += sandboxmod.TOOLS
if self.web:
tools += webmod.TOOLS
if self.ext:
Expand Down Expand Up @@ -828,6 +849,43 @@ def t_browser_import(self, ref=None, url=None, tab_id=None, page=None):
attached = {"upload_id": uid, "filename": up["filename"], "size": up["size"], "next": "import_files puts it in a namespace"}
return attached, f"Attached {up['filename']} from the browser"

# ---- the conversation's code sandbox (sandbox.py): Python and bash on the OpenShell gateway ----
def _sandbox(self, fn, *args):
try:
return fn(self.db, self.cfg, self.chat, *args)
except sandboxmod.SandboxError as e:
raise ValueError(str(e)) from None

def _sandbox_run(self, kind, source, files):
source = str(source or "")
files = [str(f) for f in ([files] if isinstance(files, str) else files or [])]
if sandboxmod.sends_out(self.cfg) and self.wary():
# code that reaches another host can carry out what the answer read: after text from outside Lens, which
# may have put it there, a run asks first
first = next((x.strip() for x in source.splitlines() if x.strip()), "")[:80]
args = store.clean({"code" if kind == "python" else "command": source, "files": files or None})
lang = "Python" if kind == "python" else "bash"
where = "the package registries" if sandboxmod.settings(self.cfg)["network"] == "packages" else "the web"
return self._approval(f"run_{kind}", args, f"Run {lang} in the code sandbox, which can reach {where}: {first}" + self.why())
out = self._sandbox(sandboxmod.run, self.account, kind, source, files)
return out, sandboxmod.summary(kind, source, out)

def t_run_python(self, code, files=None):
return self._sandbox_run("python", code, files)

def t_run_bash(self, command, files=None):
return self._sandbox_run("bash", command, files)

def t_sandbox_files(self):
out = self._sandbox(sandboxmod.listing, self.account)
n = len(out.get("files") or []) + int(out.get("more") or 0)
return out, f"Looked at the sandbox's files ({n})"

def t_import_sandbox_file(self, path, namespace=None):
# out of the sandbox as one of the person's held uploads, then imported as a file they attached would be
up = self._sandbox(sandboxmod.hold, self.user, path)
return self.t_import_files([up["id"]], namespace)

# ---- the web (web.py): searching it and reading pages, once an admin picked a search provider ----
def _web(self, tool, args, what, own):
"""A web search or page read, as (what it found, None), or (None, the approval it waits for): the words or the
Expand Down Expand Up @@ -1789,6 +1847,15 @@ def _carry_out(db, cfg, a, user, editable, decision, base, admin, readable):
result = author_extension(db, a["tool"], args, user, admin, readable if readable is not None else editable, editable)
elif a["tool"] in ("switch_automation", "run_automation"):
result = _automation_approved(db, cfg, a["tool"], args, user, admin)
elif a["tool"] in ("run_python", "run_bash"): # a run that can reach another host, asked first (Toolbox._sandbox_run)
if a.get("chat") is None or not sandboxmod.offered(cfg, admin, bool(editable)):
raise PermissionError("the code sandbox is off, or not yours to use")
kind = a["tool"].removeprefix("run_")
source = args.get("code") if kind == "python" else args.get("command")
try:
result = {"result": sandboxmod.run(db, cfg, a["chat"], a.get("account") or user["id"], kind, source, args.get("files") or [])}
except sandboxmod.SandboxError as e:
raise ValueError(str(e)) from None
elif a["tool"] in ("change_settings", "create_namespace", "import_files", "import_urls", "invite_person", "change_person"):
result = ops_tools.apply(db, cfg, base or cfg, a["tool"], args, user, editable, admin)
elif a["tool"] == "decline_approval": # the person's yes to declining another of their approvals
Expand Down
Loading
Loading
0