Phase 1 comparative behavioral study of gpt-5.5-2026-04-23, claude-opus-4-7, and the exploratory within-provider drift arm claude-opus-4-8.
The publish-candidate write-up is docs/post.md. The pre-registration/design document is study_design.md. The current Phase 1/2 design baseline is v1.6.0 (c90e846); the final preregistration lock hash is still pending the remaining Phase 2 calibration work described in the design doc. The claude-opus-4-8 arm was added post-hoc as an exploratory replication/control and does not revise the original confirmatory tests.
docs/post.md— Phase 1 blog post draft/publish candidate.study_design.md— study protocol, amendments, analysis plan, and release policy.prompts/— prompt registry and coding fixtures.rubrics/andjudge/rubrics/— human-readable anchors and machine-readable judge schemas.analysis/,scoring/,tools/— feature extraction, summaries, figures, kappa tooling, and correctness scoring.data/figures/— tracked figures used by the post.data/results/— tracked summary metrics used by figures, tables, and scope claims when raw artifacts are restricted.orbench/sample_manifest.json— pinned OR-Bench sample manifest without restricted prompt text.
Raw generations, judge logs, kappa coding files, and derived parquet feature tables are intentionally not committed wholesale. They follow the release policy in study_design.md §10, because some prompts and outputs are sensitive or restricted by provider/API terms.
Tracked public artifacts currently include the code, prompts, rubrics, summary metrics, figure outputs, and OR-Bench sample manifest needed to inspect the Phase 1 claims. The ignored local raw rows and judge logs remain the source artifacts for regeneration; tracked summaries under data/results/ pin the post-facing values and availability claims in a clean checkout.
Install dependencies into a virtual environment, then run the clean-checkout checks:
python3 -m venv .venv
.venv/bin/pip install -r requirements.txt
.venv/bin/python -m pytestThe rendered post figures are tracked under data/figures/. Regenerating them requires local data/judgments/*.jsonl files, which may be absent in a clean public checkout:
.venv/bin/python -m analysis.post_figuresThe figure command refuses to run if the required local judgment files are missing, empty, or have unexpected per-cell counts, to avoid overwriting tracked figures with misleading charts.
This repository is dual-licensed:
- Code (everything except the content listed below) — MIT License.
- Content —
study_design.md,docs/,rubrics/*.md,data/figures/, and the prompt/rubric text — Creative Commons Attribution 4.0 International (CC BY 4.0).
You may copy, modify, and redistribute both (including commercially), provided you keep the MIT notice for code and give appropriate credit for content. The un-committed raw model outputs are not covered by these licenses; their release is governed by the §10 data-release policy in study_design.md and by the applicable OpenAI / Anthropic API terms.