Repository navigation
Clarify STOP authority on the native-token fast path without changing rendering #961
Description
On main d734e25fca0c343ed06a5fa491a15736ff954cf3, complete native Chat Completions token IDs can be tokenized without a tokenizer, while STOP remains unidentified for a materialized response with finish_reason="stop" and no integer stop_reason. Supplying the correct EOS authority changes only the terminal STOP bit in the example below.
This is an authority/API gap, not evidence that marking with a known tokenizer is wrong. Native IDs and finish_reason="stop" alone do not identify which token, if any, belongs to the terminating sequence. In this mode an absent STOP bit must not be interpreted as proof that the token is not a stop.
Minimal public reproduction (no model or tokenizer download):
from datetime import UTC, datetime
from openai.types.chat import ChatCompletion
import art.trajectories as tr
class StopAuthority:
eos_token_id = 9
def __call__(self, text, **kwargs):
assert text == "END"
return [8, 9]
now = datetime(2026, 1, 1, tzinfo=UTC)
response = ChatCompletion.model_validate({
"id": "public-stop-example", "object": "chat.completion", "created": 0,
"model": "public/example", "choices": [{
"index": 0, "finish_reason": "stop", "stop_reason": None,
"message": {"role": "assistant", "content": "answer"},
"prompt_token_ids": [1], "token_ids": [2, 9],
"logprobs": {"content": [
{"token": f"token_id:{t}", "logprob": -0.5,
"bytes": [], "top_logprobs": []} for t in [2, 9]
]},
}],
})
trajectory = tr.Trajectory(exchanges=tr.TrajectoryExchanges(chat_completions=[
tr.ChatCompletionsExchange(
request={"model": "public/example", "messages": [
{"role": "user", "content": "question"}
]}, response=response, start_time=now, end_time=now,
)
]))
unknown = trajectory.tokenize()
known = trajectory.tokenize(tokenizer=StopAuthority())
assert unknown.tokens == known.tokens == [1, 2, 9]
assert [int(f) for f in unknown.flags] == [1, 23, 23]
assert [int(f) for f in known.flags] == [1, 23, 31]The minimal authority object intentionally has no renderer. The checked fixture also verifies identical logprobs and SAMPLED first-occurrence masks. This does not establish equivalence for consumers that select STOP or for rendering with a full tokenizer.
Relevant current contracts:
_stop_suffixreturns zero without a tokenizer unless an exact non-bool integer stop reason proves the final token._sampled_stop_suffixalso retains the separate no-materialized-output case._tokenize_historypassestokenizer=Noneinto the exact native builder before resolving any renderer. This preserves the intentional no-import contract tested bytest_exact_tokens_form_one_append_only_history_without_tokenizer, including W&B artifact model names.- Passing a full tokenizer is not a general render-neutral workaround:
_history_needs_synthetic_stopand other renderer-selection predicates inspect it. - Parallel tokenization's process path explicitly supplies
tokenizer=None; explicit tokenizer objects currently select the thread path. A future separate authority must survive that boundary and remain bound to the correct model, rather than silently being omitted or shared across unrelated models.
A small safe next step is documenting the incomplete-STOP meaning of the native-only path. If callers need complete STOP flags without changing render selection, consider an explicit, per-model STOP authority distinct from the renderer tokenizer, carried through public/parallel APIs into suffix/marker checks only. Do not eagerly load metadata, guess EOS IDs, mark every final sampled token, or change synthetic-stop rendering to resolve this gap.
Validation: the real ART/Pydantic reproduction and eight existing tests passed on the pinned source (EOS/tool stop, string stop, explicit native integer stop, metadata-only output, native-only no-load, and length-stop rendering). Local execution bypassed only top-level ART package startup and disabled heavy imports/network; no model, provider, GPU, or frozen-run source was used or changed. No production patch accompanies this issue.
Activity
bradhilton commentedon Sep 25, 2026
A captured two-history regression now demonstrates this authority gap after a separate rendering correction (draft #967). The original process worker returned both histories with 28,282 tokens, without native-recovery fallback. The original input was unchanged, and serialized worker output matched the observed output.
An independent check loaded the exact per-model STOP authority from the pinned tokenizer assets. It rejected the ordinary result at the first missing sampled STOP bit: one missing bit observed, zero extra bits observed. Three of six source proofs completed before that refusal; these are partial counters, not evidence that the other source predicates all pass or that exactly one bit is missing overall. Authority resolution covered all six sources and identified four terminal STOP tokens in total.
This separates successful rendering/tokenization from complete STOP labeling. We have not repaired the result inside the checker, guessed an EOS token, changed the native-only no-load contract, or adopted a production fix. The next isolated validation will preserve renderer/process selection and compare a narrowly bound authority correction with the complete native conditioning, token IDs, logprobs, first-occurrence masks and STOP checks.
Evidence: captured worker result
f4170a3b6e775d9dcd404b912e7e318a1f600ea979780e9a6cd50b74f83804cb; alignment receipt1807db18b3101d6374354c75472c03138aea61d38ac9820dd314af3d4cd371b0. Private trajectory content and token arrays are not included here.bradhilton commentedon Sep 25, 2026
The follow-up captured validation now passes in the explicit sampled-training scope. The candidate preserves the rendering correction from #967 and reconciles STOP flags only after the ordinary tokenizer returns, using the exact per-model tokenizer authority and full native source checks. It does not pass a full tokenizer into rendering or force a synthetic failure/recovery.
All six source conditioning, token-ID, logprob, sampled-objective and STOP checks passed: 8,881 finite first-occurrence terms and four sampled STOP tokens, with zero missing or extra STOP bits. One copied history needed three STOP additions; both original history arrays, all six native source arrays and the whole input remained unchanged. The ordinary result retained two histories and 28,282 tokens. The earlier first-failure counter of one missing bit was therefore correctly treated as partial.
This validates the narrow sampled-worker correction on the captured case. It does not resolve the generic native-only/no-load API contract discussed in this issue, establish SFT equivalence, or demonstrate numerical training/GPU performance. No global eager tokenizer load or guessed EOS labeling was introduced.
Full result:
fffdc951cf9c428107b2eb69a6527f5d6e3947bb47b60884aecd67febe45c4fd; independent alignment:7ae1a61c507161f585f9cc0fd92f025984a48a09dcc483eeb50178ad7d29fb65. All diagnostic resources were retired after collection.bradhilton commentedon Sep 25, 2026
Draft #973 now provides a maintained, explicit API for this scope:
await art.tokenize_sampled(items, *, model=None, base_model=None). Ordinary multi-history tokenization runs first, then a separate per-model STOP authority pass verifies complete native conditioning, output IDs/logprobs and sampled coverage before copying only missing proved STOP flags. It does not supply a tokenizer to ordinary rendering, change genericart.tokenize, or admit native-chain/subchain recovery. Incomplete or unsupported sampled evidence refuses explicitly.Thirty new public API/proof controls plus six existing dispatch/error/nonmutation regressions pass, with independent source review and clean Ruff/type checks. The real process payload/serialization/rebinding is covered; no new private replay or OS-spawn validation is claimed. Authority follows ART's resolved model configuration and cannot independently attest an unrecorded historical revision behind a mutable selector.
The earlier captured six-source STOP repair remains supporting evidence for the reused reconciliation logic, not a private validation of this new API. Repository CI is pending on
cb611c071; the broader generic conditioning change in #957 and experimental subchains remain separate. No running or frozen experiment source was changed.