8000 abhaymettu (Abhay Mettu) · GitHub
[go: up one dir, main page]

Skip to content
View abhaymettu's full-sized avatar

Organizations

@oneirum

Block or report abhaymettu

Block user

Prevent this user from interacting with your repositories and sending you notifications. Learn more about blocking users.

You must be logged in to block users.

Content in all repositories owned by your account will be closed.
Maximum 250 characters. Please don’t include any personal information such as legal names or email addresses. This note will only be visible to you.
Report abuse

Contact GitHub support about this user’s behavior. Learn more about reporting abuse.

Report abuse
abhaymettu/README.md

i build ai agents and the tools that check whether they actually did what they said.

  • agent-run-reliability · reruns 24 agent tasks eight times on two models to see how many runs a ranking needs. with one run per task, two evaluations pick different winners 27.5% of the time.
  • code-freeze-bench · rebuilds the 2025 incident where an agent wiped a production database during a code freeze, on a real postgres server, to find out which guardrails actually stop it.
  • receipts · coding agents often say "tests pass" or "fixed it" without having checked. this reads an agent's session and flags the claims its own tool calls don't back up.
  • benchmeasure · treats ai benchmarks as measurement instruments and asks how much a leaderboard ranking can be trusted. near the top of swe-bench, not much.
  • emotion-label-ceiling · grades the same speech-emotion model against the emotion the actor was asked to play and against the one listeners heard. the two scores are 25.6 points apart.
  • laneproof · runs many coding agents in parallel and only counts work that lands on main with its evidence re-run, so no agent's "done" is taken on trust.

also minecraft-sre (an agent on call for a minecraft server, graded from the world itself rather than its report), index-cards (a memory layer for coding agents over a folder of markdown notes), sayso (your own agent can answer your claude code sessions, and anything risky waits for a yes signed on your phone), because (preregistered experiments on what makes claude code follow its instructions), mcpify (any scriptable mac app as an mcp server), hill-climb (a skill for improving code in measured rounds) and perch (everything running on your mac, in the menu bar). write-ups and papers at abhaymettu.com.

economics at uw-madison, cs and data science minors, may 2026 · previously swe intern at martin (yc s23) · linkedin

Pinned Loading

  1. receipts receipts Public

    Checks what a coding agent says it did against what its own tool calls show. Runs on Claude Code and Codex sessions, or as a Stop hook.

    Python 1

  2. benchmeasure benchmeasure Public

    How far can you trust an AI benchmark's ranking? Psychometric checks for reliability, invariance and validity on any results file, plus the paper

    Python

  3. code-freeze-bench code-freeze-bench Public

    Does a coding agent honour a code freeze, and tell the truth the next morning? Real Postgres, statement log, PITR.

    Python

  4. laneproof laneproof Public

    Run many coding agents in parallel. Count only the work that lands with evidence.

    Python

  5. agent-run-reliability agent-run-reliability Public

    How often you have to rerun an agent benchmark before its ranking means anything: run noise vs model differences, decomposed.

    Python

  6. emotion-label-ceiling emotion-label-ceiling Public

    Speech-emotion scores depend on which label you grade against, the emotion the actor played or the one listeners heard.

    HTML

0