8000 feat(assistant): an optional local classifier for prompt injection by adeelahmad · Pull Request #347 · adeelahmad/lense · GitHub
[go: up one dir, main page]
Skip to content

feat(assistant): an optional local classifier for prompt injection - #347

Merged
adeelahmad merged 1 commit into
mainfrom
claude/project-thread-mj7ggd
Oct 11, 2026
Merged

feat(assistant): an optional local classifier for prompt injection#347
adeelahmad merged 1 commit into
mainfrom
claude/project-thread-mj7ggd

Conversation

@adeelahmad
@adeelahmad adeelahmad commented Oct 10, 2026
Copy link
Copy Markdown
Owner

Requested by Adeel · project thread

Description

Before: flagging text that reads like instructions to an AI (#329) had two choices: the free patterns, which are kept narrow and miss rephrased attacks, or asking the decision model about every tool result, which costs a call each and sends the text to that model.

After: Settings → AI assistant → Flag text that reads like instructions to an AI → classifier (ai.injection_check: classifier) runs each tool result the patterns let through past ProtectAI's deberta-v3-base-prompt-injection-v2 (Apache-2.0) on ONNX Runtime, on this machine. Nothing leaves it and no tokens are paid for. The model (about 740 MB, at a pinned revision, checked against its SHA-256) and the new injection extra are fetched when the setting asks for them (Settings → Components lists it); until then the patterns carry on alone. Long text is read 512 tokens at a time, up to eight windows, and the most suspicious window counts. Each check is a row in the activity ledger (injection_check), with the tokens read and the time taken.

It's off by default: the default stays the free patterns.

Seventh of the prompt-injection series (#302 marking, #309 asks first, #329 flags, #338 room links, #343 MCP, #344 audit). Last: the threat-model update.

Motivation and Context

The coordinator's brief for "multi layered security" named a detection layer from the decision model or a proven open-source detector. This adds the detector, kept light by default (Raspberry Pi) and local when on (private, no per-call cost).

How: injection_model.py loads the ONNX model and tokenizer once, scores windows of [CLS] + ids + [SEP], feeds only the inputs the session asks for, and logs activity.call("injection_check", ...). untrusted.flags(..., model=True) calls it when the setting is classifier; it reads the strings a JSON result carries (untrusted._plain), because raw JSON looks like code to the model. components.InjectionModel fetches it on first use. Measured on the real model: it flags injections well but also some polite requests written for people ("please add the tracking number to the link"), so it only warns; docs say so.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change; ! in the title)

Checklist:

  • The title is a Conventional Commit.
  • My code follows the code style of this project (make hooks checks it on commit).
  • I added or updated tests.
  • My change requires documentation updates, and I have updated docs/ accordingly.
  • Users need more than the title to understand it, so I wrote it under ## Unreleased in CHANGELOG.md. (n/a: the title and docs/assistant.md say it)

Done questions (AGENTS.md)

  • Goal: the local detector layer of the injection defenses; merged on green CI.
  • Agents: the setting is in the settings API, and the component in the components API.
  • Zero setup: picking classifier fetches everything itself; no .env edit.
  • Fewest clicks and keystrokes: one select.
  • Configurable: ai.injection_check in Settings → AI assistant, default patterns.
  • Light: off by default; the 740 MB model and ONNX Runtime are fetched only when picked.
  • Private and secure: runs on the user's hardware, sends nothing; the model is pinned and checksummed; the threat-model update (third-party row) is the last PR of this series.
  • Cost: no tokens; each check is an injection_check row in the activity ledger.
  • User decides: opt-in; a warning, never an action; existing defaults unchanged.
  • Upgrades: no stored data changes.
  • Honest: docs say it reads English best, misses some attacks and flags some honest requests.
  • Tests (tests/domain/test_injection_model.py on a tiny ONNX model and tokenizer: flags, windows, ledger rows, fallback), docs (assistant.md, components.md); uv.lock regenerated with uv lock.
  • Open standards: ONNX; model and libraries are Apache-2.0, so Lens stays MIT.
  • Result reported with evidence; refine later: a threshold setting (fixed at 0.8 now).

🤖 Generated with Claude Code

https://claude.ai/code/session_01BVez82cREgCJtFk6yHReed

Generated by Claude Code

@claude
feat(assistant): an optional local classifier for prompt injection
17ffe43 
ai.injection_check "classifier" runs each tool result the free patterns
let through past ProtectAI's deberta-v3-base-prompt-injection-v2
(Apache-2.0) on ONNX Runtime, on this machine: nothing leaves it and no
tokens are paid for. The model (about 740 MB, pinned revision, checked
against its SHA-256) and the `injection` extra are fetched when the
setting asks for them (Settings > Components); until then the patterns
carry on alone. Long text is read 512 tokens at a time, up to eight
windows. Each check is an activity-ledger row (injection_check). Off by
default; the default stays the free patterns.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BVez82cREgCJtFk6yHReed
@adeelahmad adeelahmad self-assigned this Oct 10, 2026
@adeelahmad
adeelahmad merged commit 0758c3c into main Oct 11, 2026
17 checks passed
@adeelahmad
adeelahmad deleted the claude/project-thread-mj7ggd branch October 11, 2026 00:40
This was referenced Oct 11, 2026
Merged
Merged
adeelahmad added a commit that referenced this pull request Oct 11, 2026
@adeelahmad @claude
docs: list the prompt-injection layers and the setup budget as built (#…
0587117 
…349)

"Stay private and safe" now covers flagging instruction-like text (patterns or the optional local classifier, #329,
#347), asking before the browser acts on what the assistant wrote (#333), breaking up links in chat-room answers
(#338), the archive as data over MCP (#343) and the audit log of what was caught (#344). "See what it costs" adds the
overall budget asked for in setup (#281). "In progress" drops what merged and adds #337, #341 and #346.


Claude-Session: https://claude.ai/code/session_01HDx7PPUauyjMn1LZs8JAsS

Co-authored-by: Adeel Ahmad <noreply@anthropic.com>
adeelahmad pushed a commit that referenced this pull request Oct 14, 2026
@claude
docs(threat-model): claim the prompt-injection egress controls
59ee829 
P-INJECTION-EGRESS: once an answer has read text from outside Lens,
outbound tools and setup changes ask first, and chat-room answers keep
only the links people, the archive, tools or Lens wrote. Correctness-only,
a bypass is VALID-HARDENING (Q26). The Assistant x-prompt-injection row
is now claimed; N-PROMPT says marking and flagging make injection less
likely and are not claimed; MCP results and the optional classifier are
described; Q26 lists #302, #309, #329, #338, #343, #344 and #347.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BVez82cREgCJtFk6yHReed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@adeelahmad adeelahmad

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@adeelahmad @claude
0