We introduce optstop, an open-source tool for LLM evaluations that keeps running where uncertainty is high, and stops where estimates are precise or stable enough.
Standard evaluations cap how much compute AI agents can use. We show that raising those caps changes measured capability, the difficulty of tasks agents can solve, and how fast the frontier appears to move.
Alongside Irregular, we found evidence demonstrating that evaluators need to use large token budgets to understand the cyber capabilities of recent Large Language Models (LLMs).
We outline a step-by-step pipeline for using our open-source transcript analysis tool, Inspect Scout.
Why we use transcript analysis for our agent evaluations, and results from an early case study.
Calibrating AI risk assessment through rigorous elicitation practices.
Our new framework can assess the reliability of LLM evaluators, while simultaneously answering a primary research question.
HiBayES: a flexible, robust statistical modelling framework that accounts for the nuances and hierarchical structure of advanced evaluations.
A Methodology for Evaluating Scientific Assistants
A common technique for quickly assessing AI capabilities is prompting models to answer hundreds of questions, then automatically scoring the answers. We share insights from months of using this method.