Repository navigation
feat(assistant): flag third-party text that reads like instructions to an AI - #329
Merged
Merged
Conversation
…o an AI A tool result that says "ignore previous instructions", claims new instructions or a new role, addresses an AI reading it, asks to hide something from the person, reveal the system prompt, call a tool or put data into a link, or carries control tokens or invisible Unicode tag characters, gets a warning on its marker for the model, and the answer's step says so. Invisible tag characters are taken out of marked text. ai.injection_check: patterns (free, the default), model (the decision model also checks what the patterns let through, one routine call per tool result, through the activity ledger), or off. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BVez82cREgCJtFk6yHReed
8000
17 of 19 tasks
19 of 22 tasks
adeelahmad
added a commit
that referenced
this pull request
Oct 10, 2026
…PRs (#335) The README and Get started say the web installer can install with the defaults in one click (#331). "In progress" adds flagging instruction-like third-party text (#329), asking before the browser acts on what the assistant wrote (#333) and cloud model presets (#334). Claude-Session: https://claude.ai/code/session_01HDx7PPUauyjMn1LZs8JAsS Co-authored-by: Adeel Ahmad <noreply@anthropic.com>
This was referenced Oct 10, 2026
adeelahmad
added a commit
that referenced
this pull request
Oct 11, 2026
…349) "Stay private and safe" now covers flagging instruction-like text (patterns or the optional local classifier, #329, #347), asking before the browser acts on what the assistant wrote (#333), breaking up links in chat-room answers (#338), the archive as data over MCP (#343) and the audit log of what was caught (#344). "See what it costs" adds the overall budget asked for in setup (#281). "In progress" drops what merged and adds #337, #341 and #346. Claude-Session: https://claude.ai/code/session_01HDx7PPUauyjMn1LZs8JAsS Co-authored-by: Adeel Ahmad <noreply@anthropic.com>
adeelahmad
pushed a commit
that referenced
this pull request
Oct 14, 2026
P-INJECTION-EGRESS: once an answer has read text from outside Lens, outbound tools and setup changes ask first, and chat-room answers keep only the links people, the archive, tools or Lens wrote. Correctness-only, a bypass is VALID-HARDENING (Q26). The Assistant x-prompt-injection row is now claimed; N-PROMPT says marking and flagging make injection less likely and are not claimed; MCP results and the optional classifier are described; Q26 lists #302, #309, #329, #338, #343, #344 and #347. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BVez82cREgCJtFk6yHReed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Requested by Adeel · project thread
Description
Before: a poisoned transcript, email, web page or tool result reached the model marked as data (#302), but nothing told the model or the person that a piece of it was trying to give orders, and Unicode's invisible tag characters could spell out instructions nobody sees on screen.
After: a tool result that says "ignore previous instructions", claims to be new instructions or a new role, addresses an AI reading it, asks to hide something from the person, to reveal the system prompt, to call a tool or to put data into a link, or carries chat control tokens or hidden characters, gets a warning on its marker for the model (
<untrusted-… warning="reads like instructions to an AI: ignore previous instructions">), and the answer's step says so: "Read 12 line(s) of Mallory's talk (reads like instructions to an AI: ignore previous instructions)". Invisible tag characters are taken out of marked text.Settings → AI assistant → Flag text that reads like instructions to an AI (
ai.injection_check):patterns(the default, free, kept narrow so honest text isn't flagged),model(the decision model also checks each tool result the patterns let through: one routine call each, through the activity ledger), oroff.Third of the prompt-injection series (#302 marked outside text as data, #309 asks before sending data out). Next: chat-room link defanging, MCP result labelling, audit log rows, an optional local classifier, and the threat-model update.
Motivation and Context
Adeel asked to handle third-party prompt injection with layered security. This is the detection layer: independent of marking and approvals, a warning and never the one that stops an action (the approvals in #309 do that).
How:
untrusted.flags(text, cfg, model=False)runs narrow regexes (each with a reason), a control-token check and a hidden-character check over the text and over the strings its JSON carries; withmodel=Trueandai.injection_check = "model"it asksdecide.choosewhen nothing matched.untrusted.wrap(..., why=)adds the warning attribute.Toolbox.callflags each untrusted tool result and adds the reason to the step summary.Types of changes
!in the title)Checklist:
make hookschecks it on commit).docs/accordingly.## UnreleasedinCHANGELOG.md. (n/a: the title and docs/assistant.md say it)Done questions (AGENTS.md)
ai.injection_checkin Settings → AI assistant, defaultpatterns.modelis one decision-model call per tool result, in the activity ledger and under budgets.🤖 Generated with Claude Code
https://claude.ai/code/session_01BVez82cREgCJtFk6yHReed
Generated by Claude Code