8000 feat(sdk): add System One evaluation capability by chen-ran · Pull Request #50 · felinics/twilight · GitHub
[go: up one dir, main page]
Skip to content
< 8000 /div>

feat(sdk): add System One evaluation capability - #50

Draft
chen-ran wants to merge 1 commit into
mainfrom
feat/system-one-evaluation
Draft

feat(sdk): add System One evaluation capability#50
chen-ran wants to merge 1 commit into
mainfrom
feat/system-one-evaluation

Conversation

@chen-ran
@chen-ran chen-ran commented Sep 19, 2026
Copy link
Copy Markdown
Member

What

Adds Evaluate, a new SDK capability for System One models — models that evaluate a state against typed questions and return structured answers with calibrated probabilities, instead of generating text. The first one is TypeSafe's Jev (POST /v1/systemone).

This is the SDK surface only. No provider yet — see Follow-ups.

result, err := sdk.Evaluate(ctx,
    sdk.WithEvaluationModel(model),
    sdk.WithState("Help! My payouts have been failing for 3 days."),
    sdk.WithQuestions(
        sdk.Noul("is_urgent", "Does this convey urgency?").
            WithMeanings("Explicitly time-sensitive", "No urgency expressed"),
        sdk.Choice("department", "Which team should handle this?",
            sdk.Opt("billing", "Payments, invoicing, refunds"),
            sdk.Opt("technical", "Bugs, outages, integrations"),
            sdk.Opt("sales", ""), // no extra detail; sent as null
        ),
        sdk.Score("frustration", "How frustrated is the customer?",
            "Calm", "Frustrated", "Very angry"),
    ),
)

urgent, _ := result.Noul("is_urgent")              // 0.999
team, confidence, _ := result.Choice("department") // "billing", 0.596

Why a new capability instead of a Provider

A System One model has no messages, no tools, no streaming and no text output. Routing it through DoGenerate would mean a provider that rejects almost every GenerateParams field and returns a text result nobody wants. So it gets its own interface, exactly like EmbeddingProvider, SpeechProvider and VideoProvider:

type EvaluationProvider interface {
    ListModels(ctx context.Context) ([]*EvaluationModel, error)
    DoEvaluate(ctx context.Context, params EvaluateParams) (*EvaluateResult, error)
}

Prior art agrees on the diagnosis. maruel/genai opens its TypeSafe notes with "TypeSafe is not a chat provider... it does not fit the rest of the provider interfaces directly" — but then maps it onto its chat entry point anyway, and documents the cost: its scoreboard has to be hand-authored because the generic smoke tests drive providers with text prompts and cannot exercise a provider that only answers questions. twilight has the same generic Test/TestModel/ListModels surface and would pay the same price, so this PR keeps the capability separate.

The interface is deliberately vendor-neutral (EvaluationProvider, not JevProvider): a second System One vendor, or an LLM-backed implementation built on twilight's existing providers, can satisfy it without a change here.

The JSON ordering problem

encoding/json marshals Go maps with their keys sorted. For most APIs that is invisible. Here it is not: a System One model reads the object as content, so the order of Choice options — and of the fields of a structured state — is part of what it is shown. A map[string]string of options silently reorders the question on its way to the wire, which makes results harder to reproduce and is invisible at the call site.

Both Go projects that already integrate this API use map[string]... for choice criteria and lose the order. This PR fixes it in three places:

Where Solution
Choice criteria ChoiceOptions is a slice with a MarshalJSON that emits a JSON object in slice order. Duplicate option names are an error rather than a silently collapsed object.
Noul criteria NoulCriteria is a struct, so true stays before false. A map would sort them the other way.
Structured state / instructions New sdk.JSONObject, an ordered JSON object for values built at runtime. Order is preserved at every nesting level on the way back in too, and numbers decode as json.Number so a document round-trips byte-identically.

A Go struct already marshals in field-declaration order, so a struct state needs no special handling — that is documented on WithState, and JSONObject is there for the case where a map would otherwise be the obvious choice. Score levels are a slice already, and their index is the level number, so they must never be reordered.

Design notes

  • Criteria any mirrors the wire. The API has exactly one criteria field whose shape follows type, so Question does too: *NoulCriteria, ChoiceOptions, or ScoreLevels. Validate type-switches on Kind and reports the mismatch. This keeps Question small enough to pass by value and lets a provider marshal q.Criteria straight through.
  • Confidence is a *float64. The API documents that Noul answers carry no confidence. A plain float64 would read as 0.0, and the confidence-gating pattern the API is designed around (if confidence < 0.5 { route_to_human() }) would silently treat every Noul as maximally uncertain.
  • Typed accessors fail loudly. result.Noul("department") on a Choice answer returns answer for "department" is a choice, not a noul, not a zero value.
  • Documented limits are enforced client-side (MaxChoiceOptions = 255, MinScoreLevels = 2, MaxScoreLevels = 10), so a malformed questionnaire fails before the round trip instead of as an opaque 422.
  • WithQuestions appends, so questions can be assembled across call sites. Asking them together is the point: the state is read once and every question is evaluated against it in parallel, which is what makes batching dramatically cheaper than one call per question.

Tests

29 new unit tests, no API key or network required:

  • ordering: Choice options, nested JSONObject, NoulCriteria field order, struct members. The ordering tests assert against the sorted-map output too, so they fail loudly if they ever stop proving anything.
  • JSONObject round trip including nested objects, arrays and a large integer; non-object input rejected; duplicate keys rejected; nil marshals to null.
  • validation table over all three kinds, both criteria-count bounds, wrong criteria types, duplicate/unnamed options, duplicate question IDs, missing model/provider/state/questions.
  • a fake EvaluationProvider covering parameter pass-through, error propagation, typed accessors, kind-mismatch errors, and Noul's absent confidence.

go build, go vet, golangci-lint run (0 issues) and go test -short -race all pass.

Follow-ups

Not in this PR, kept separate to keep it reviewable:

  1. provider/typesafe/systemone — the actual client. Needs 429/529 Overloaded backoff honouring Retry-After, which internal/utils/fetch.go does not have yet. Note sdk.ClassifyProbeStatus does not recognise 529 either.
  2. Struct-field questionnaires — QuestionsFrom(&struct{ Urgent sdk.Noul; Team sdk.Choice }{...}), so a question and its answer live in one field. Good ergonomics for fixed question sets; the slice API in this PR stays underneath for the runtime-generated ones (ranking 255 options or 182 tools is a primary use case and cannot be expressed as struct fields).
  3. Docs — docs/evaluation.md, plus the capability table in docs/providers.md and the README feature list.
  4. An LLM-backed EvaluationProvider built on the existing OpenAI/Anthropic/Google providers, so the same questions can run against either a real System One model or an LLM for cost/quality comparison. TypeSafe ships exactly this for Python; twilight's multi-provider reach makes it a natural fit.

🤖 Generated with Claude Code

@chen-ran @claude
feat(sdk): add Sy 958D stem One evaluation capability
2ad93bc 
A System One model (TypeSafe's Jev) evaluates a state against typed
questions and returns one typed answer per question. It generates no
text, has no messages, no tools and no streaming, so it does not fit
sdk.Provider. Add it as a capability of its own, alongside embedding,
speech, transcription, image and video.

Questions are built with Noul, Choice and Score, and a single Criteria
field mirrors the wire shape. Answers come back keyed by question ID,
with typed accessors that reject a kind mismatch instead of returning a
zero value. Confidence is a pointer because a Noul answer carries none,
so a confidence gate cannot silently read it as a hard zero.

Add sdk.JSONObject to solve the ordering problem this API exposes:
encoding/json sorts map keys, but a System One model reads the object as
content, so the order of Choice options and of a structured state is part
of what it is shown. ChoiceOptions marshals in slice order, NoulCriteria
is a struct so true stays before false, and JSONObject covers objects
built at runtime, preserving order at every level on the way back in too.

No provider yet; this is the SDK surface only.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@chen-ran chen-ran mentioned this pull request Sep 19, 2026
Draft
@chen-ran
chen-ran added this pull request to stack #52 September 19, 2026 02:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@chen-ran
0