Repository navigation
Retained sampled suffix can carry logprobs under a different causal prefix #883
Description
Status reconciliation — September 11, 2026 (Schulman)
Default sampled-lineage attribution remains open; PR #885 is explicitly HELD by Brad. Do not merge it, add a blanket manual-history rejection rule or treat private experiment constraints as the shared API contract.
The independent prompt-aware capture-deduplication fix #886 is already merged at 66f644d. Earlier comments calling it unmerged are historical. #886 does not include #885 and does not resolve this separate default-attribution question.
Planned lane: Schulman and subagents, parked pending Brad's scope decision; no new implementation or launch is initiated by this status update. Manually assigned/edited histories remain supported.
Historical report (preserved):
Native multi-history tokenization can mark a retained response suffix
SAMPLED|EXACT and select it for training after earlier sampled tokens have been
removed from its conditioning context. Its stored logprob still belongs to the
original longer prefix. This is independent of the terminal renderer-tail work
in #882 and does not require the experimental raw-capture adapter.
Minimal synthetic capture:
| Exchange | Captured prompt | Captured output |
|---|---|---|
| First | [1] |
[2, 3] |
| Second | [1, 3, 4] |
[5] |
The second request retains the first assistant's message text, but its prompt
tokens omit token 2. With multi_history=True and
reconcile_text_equivalent_tokenizations=False, ART emits a second history
[1, 3, 4, 5], marks token 3 as SAMPLED|EXACT|ASSISTANT|OUTPUT (23), and gives
it the first response's logprob -0.3. first_occurrence_masks(where=SAMPLED)
selects it: prefix [1, 3] differs from the actual sampled prefix [1, 2, 3].
The probability was observed for 3 | [1, 2], not 3 | [1].
A CPU regression using the repository's existing synthetic exchange fixture:
import art
from art.trajectories import TokenFlag
from tests.unit.trajectories.test_tokenize import _chat_exchange
def test_retained_suffix_requires_its_original_causal_prefix():
trajectory = art.Trajectory(
exchanges=art.trajectories.TrajectoryExchanges(
chat_completions=[
_chat_exchange([1], [2, 3]),
_chat_exchange([1, 3, 4], [5], offset=1),
]
)
)
tokenized = trajectory.tokenize(
multi_history=True,
reconcile_text_equivalent_tokenizations=False,
)
masks = tokenized.tensorize().first_occurrence_masks(where=TokenFlag.SAMPLED)
captured_prefixes = {(1, 2), (1, 2, 3), (1, 3, 4, 5)}
for history, mask in zip(tokenized.histories, masks, strict=True):
for index, selected in enumerate(mask.tolist()):
if selected:
assert tuple(history.tokens[:index + 1]) in captured_prefixesThis reproduces on ART 1ce5d7213e2d9cea2dd8a04a608966c24f4629a0 and current
077abf6fa7a3a51f9ff255c37f8886c6c8256082 (both _tokenize.py SHA256
cf8745b44edf3fb22a377246ccb81fc476153f4d96d5680e9c953180c4cff134), and the
reviewed terminal-boundary overlay 7bcd560cc117e593bbc8fad17a856ffc8dcbf667
(9922a80bafeccc89d1439b4c04bc08f63daccf047b37084e01d819e6d2ec0dd2).
The existing first-occurrence trie behaves as designed: it keys model-visible
prefixes, so it cannot repair an earlier attribution error.
The source is _retained_output_suffix, which searches every output[start:]
against the later prompt. _tokenize_exact_projected_chat_history then attaches
the suffix's original logprobs and sampled/exact flags at len(prompt).
Its complete-output requirement only applies to earlier finish_reason=length
sources; a non-length source can therefore lose its original causal conditioning.
Independent offline analysis also found this shape in two real captured
rollouts: 321 and 896 positions respectively remained first-occurrence-selected
after two leading sampled tokens were omitted. Raw content is private; the
synthetic regression above contains no research payload and reproduces the
same native behavior. This establishes attribution risk, not a measured GPU
training-quality effect.
Expected behavior: every sampled position selected for importance weighting
must retain the complete conditioning under which its stored logprob was
captured. A changed-context suffix can remain ordinary conditioning, but must
not reuse the old probability as an exact sampled probability under the new
prefix. Preserve the full original sampled source in another history or refuse
when exact attribution cannot be established; avoid silently dropping observed
samples. The experiment-local adapter is being tightened separately while the
appropriate shared history/attribution behavior is reviewed.
Activity
bradhilton commentedon Sep 10, 2026
McCarthy: CLEAR for the identity-only correction at local
9fe17e39db8ad237ae304920b158b69f2f66ebf9, against reviewedda4f165a0a98c5d9539c74d854cce1d15e6810f7. No blocking finding in this scope.Only
_sampled_evidence_fingerprintchanges at runtime. Captured prompts now distinguish otherwise identical source evidence for Chat, Messages, Responses generations and Completions. Chat/Completions use the existing token extractor's choice-first, response-fallback precedence; null/missing choice prompts fall back, while a supplied choice prompt shadows response metadata. Selected-source identity stays stable through deep/JSON copies, moving the same effective prompt to the fallback, and unrelated choices/generations. The change reaches both trace keys and the source signatures used to deduplicate projected sources; it does not add a causal validator or alter rendering policy.Independent verification in a clean exact-head worktree:
da4f165give 10 failures / 6 passes, reproducing prompt aliasing and the dropped sampled-prefix witness._tokenize.pySHA256 is1e48cadbfca55d3440fcbd1484f63a31ffbf71ca3520df189b625c5f168cdc92. Ruff check/format and whitespace checks pass.The added prompt serialization is proportional to captured-prompt length, as disclosed. This clearance does not close #883, establish arbitrary manual-history correctness, or authorize adoption by the separately hash-pinned research drivers. Local candidate remains held for root's PR/integration decision; no shared edits, allocation, merge or deployment.
Evidence:
/home/brad/.local/share/mccarthy/art883-identity-review-20260910/REVIEW.mdand associated logs/oracle.bradhilton commentedon Sep 10, 2026
Schulman: Current scope after Brad's manual-history clarification: arbitrary manually assigned/edited histories remain supported. No blanket causal-provenance validator or new rejection rule is being implemented for those histories.
Standalone PR #886 is now independent of #885 at
3627311444748067482c486909d3642644509efd, based on accepted main9a16366d79eebc36e00f75670965e2629c012b75. It includes the captured prompt in source deduplication identity._history.pyequals main, #885 is not an ancestor, both exact-head reviews clear it, and correct-base CI passes. It is ready but unmerged.PR #885 remains held as a separate default-history attribution proposal: it stops automatically attaching an earlier generation's sampled lineage to a later history when that generation's complete captured conditioning/output is absent. It preserves the original source branch and does not prohibit explicit user edits. Whether to change that default attribution remains Brad's decision; the earlier proposal for a universal final provenance guard is not the current plan.
Experiment 049's strict raw G/V conditioning policy is opt-in experiment code, separately qualified. It must not be treated as a new shared restriction on manually edited histories.