8000 GitHub - pedromnasc/house-style: A pilot comparative behavioral study: do frontier LLMs have a house style? (GPT-5.5 vs Claude Opus 4.7/4.8) · GitHub
[go: up one dir, main page]

Skip to content

About

A pilot comparative behavioral study: do frontier LLMs have a house style? (GPT-5.5 vs Claude Opus 4.7/4.8)

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

House Style

Phase 1 comparative behavioral study of gpt-5.5-2026-04-23, claude-opus-4-7, and the exploratory within-provider drift arm claude-opus-4-8.

The publish-candidate write-up is docs/post.md. The pre-registration/design document is study_design.md. The current Phase 1/2 design baseline is v1.6.0 (c90e846); the final preregistration lock hash is still pending the remaining Phase 2 calibration work described in the design doc. The claude-opus-4-8 arm was added post-hoc as an exploratory replication/control and does not revise the original confirmatory tests.

Repository Contents

  • docs/post.md — Phase 1 blog post draft/publish candidate.
  • study_design.md — study protocol, amendments, analysis plan, and release policy.
  • prompts/ — prompt registry and coding fixtures.
  • rubrics/ and judge/rubrics/ — human-readable anchors and machine-readable judge schemas.
  • analysis/, scoring/, tools/ — feature extraction, summaries, figures, kappa tooling, and correctness scoring.
  • data/figures/ — tracked figures used by the post.
  • data/results/ — tracked summary metrics used by figures, tables, and scope claims when raw artifacts are restricted.
  • orbench/sample_manifest.json — pinned OR-Bench sample manifest without restricted prompt text.

Data Policy

Raw generations, judge logs, kappa coding files, and derived parquet feature tables are intentionally not committed wholesale. They follow the release policy in study_design.md §10, because some prompts and outputs are sensitive or restricted by provider/API terms.

Tracked public artifacts currently include the code, prompts, rubrics, summary metrics, figure outputs, and OR-Bench sample manifest needed to inspect the Phase 1 claims. The ignored local raw rows and judge logs remain the source artifacts for regeneration; tracked summaries under data/results/ pin the post-facing values and availability claims in a clean checkout.

Reproducing Local Checks

Install dependencies into a virtual environment, then run the clean-checkout checks:

python3 -m venv .venv
.venv/bin/pip install -r requirements.txt
.venv/bin/python -m pytest

The rendered post figures are tracked under data/figures/. Regenerating them requires local data/judgments/*.jsonl files, which may be absent in a clean public checkout:

.venv/bin/python -m analysis.post_figures

The figure command refuses to run if the required local judgment files are missing, empty, or have unexpected per-cell counts, to avoid overwriting tracked figures with misleading charts.

License

This repository is dual-licensed:

You may copy, modify, and redistribute both (including commercially), provided you keep the MIT notice for code and give appropriate credit for content. The un-committed raw model outputs are not covered by these licenses; their release is governed by the §10 data-release policy in study_design.md and by the applicable OpenAI / Anthropic API terms.

About

A pilot comparative behavioral study: do frontier LLMs have a house style? (GPT-5.5 vs Claude Opus 4.7/4.8)

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages

0