8000 aligns.md/benchmarks/references.md at main · hmmhmmhm/aligns.md · GitHub
[go: up one dir, main page]

Skip to content

Latest commit

 

History

History
12 lines (9 loc) · 2 KB

references.md

File metadata and controls

12 lines (9 loc) · 2 KB

Public evaluation examples

These repositories informed the reporting choices below. They evaluate different skills and tasks, so their scores are not comparable with ALIGNS results. Links were checked on 2026-10-15; the linked branch contents may change.

Source What it reports Lesson for this benchmark
NVIDIA nvidia-kaggle benchmark With-skill and without-skill evaluation across two agents, with 18 tasks: 12 positive and six negative activation cases. It separates execution, efficiency, accuracy, and goal accuracy. Include routine cases where extra process can hurt, and distinguish procedure use from task outcomes.
Planning with files evaluations The initial evaluation explicitly measures adherence to its 3240 three-file workflow rather than general planning ability. Later sections discuss recovery tests and limitations. Name the measured construct. Creating the expected files alone cannot demonstrate improved user decisions.
Codebase onboarding skill Public example outputs and a comparison across Claude, Codex, and Gemini, with structural and content checks. Its displayed baseline is a Claude run, not a no-skill condition. Publish inspectable outputs, and say exactly what a baseline contains. Cross-model comparisons do not isolate skill effects.
Odoo workflow benchmark Per-case outcomes, grader preferences including losses and ties, tool counts, output tokens, and limits from one-model, one-run coverage. Show cost and unfavorable outcomes beside gains, and keep conclusions within the sample.

These are examples of public project evaluations, not independent validation of ALIGNS. None establishes that ALIGNS improves collaboration or reduces user effort.

0