[go: up one dir, main page]

The USQRD Journal

Insights

Notes from the work, not the hype.

Practical thinking from the engineers and operators replacing manual work with systems that actually ship in production.

Featured
Catching Agent Failure in Production Before Your Users Do
Lead storyField Notes

Catching Agent Failure in Production Before Your Users Do

How to detect AI agent failure in production before users notice — drift detection, online monitoring, confidence-based fallbacks, and the telemetry that makes it work.

10 min read
Read
Featuring
Latest26 articles
Human-in-the-Loop That Doesn't Train Reviewers to Rubber-Stamp
01Field Notes

Human-in-the-Loop That Doesn't Train Reviewers to Rubber-Stamp

Approve-every-action gating trains reviewers to click yes. How to gate on uncertainty, blast radius, and reversibility instead — with a decision table.

Read more
The Data Access Layer Is Where Agent Projects Actually Stall
02Field Notes

The Data Access Layer Is Where Agent Projects Actually Stall

Enterprise agent projects don't stall on model choice or prompts. They stall on stale ETL, row-level permissions, and no clean API. A CTO's de-risking checklist.

Read more
Document Chunking for RAG: What We Dropped and What We Ship
03Field Notes

Document Chunking for RAG: What We Dropped and What We Ship

The chunking strategies we tried and abandoned across messy enterprise corpora — and the structure-aware, metadata-rich approach we default to now.

Read more
Prompts Are Artifacts, Not Dashboard Config: How We Version Them
04Playbook

Prompts Are Artifacts, Not Dashboard Config: How We Version Them

How we version, eval-gate, canary, and roll back prompts in production AI systems — and why editable dashboard prompts silently degrade quality across a fleet.

Read more
Your RAG Isn't Hallucinating — It Never Saw the Answer
05Field Notes

Your RAG Isn't Hallucinating — It Never Saw the Answer

Teams grade answer quality while ignoring whether the right passage was even retrieved. How to build a retrieval-only eval and why it comes first.

Read more
Five RAG Failure Modes We Keep Finding in Audits
06Field Notes

Five RAG Failure Modes We Keep Finding in Audits

A candid field guide to the five RAG failure modes we keep diagnosing in client audits — symptom, real root cause, and the cheapest fix for each.

Read more
How We Stress-Test an AI Agent Before It Ships
07Playbook

How We Stress-Test an AI Agent Before It Ships

The harness we use to break an agent before production does: adversarial inputs, failure injection, tail latency, cost per resolved task, and injection probes.

Read more
How We Cut an Agent's Token Bill 60% Without Losing Eval Points
08Case Study

How We Cut an Agent's Token Bill 60% Without Losing Eval Points

An anonymised case study: where an agent's tokens actually went, the changes that cut spend ~60%, and the measurement discipline that proved quality held.

Read more
How We Shipped a Support Agent That Deflected 40% of Tickets
09Case Study

How We Shipped a Support Agent That Deflected 40% of Tickets

An anonymised case study of a customer-support agent that deflected ~40% of inbound tickets without escalation-rage — the failures, the fixes, and the deflection ceiling.

Read more
Catching Agent Failure in Production Before Your Users Do
10Field Notes

Catching Agent Failure in Production Before Your Users Do

How to detect AI agent failure in production before users notice — drift detection, online monitoring, confidence-based fallbacks, and the telemetry that makes it work.

Read more
Workflow or Agent? A Decision Framework for Enterprise AI
11Framework

Workflow or Agent? A Decision Framework for Enterprise AI

Most enterprise tasks handed to us as 'agent' projects ship better as constrained workflows. Concrete criteria for deciding when to give a model agency — and when to take it away.

Read more
Why Your Agent Makes 40 Tool Calls to Do a 6-Call Job
12Field Notes

Why Your Agent Makes 40 Tool Calls to Do a 6-Call Job

Production agents balloon their tool-call counts with redundant retrievals and second-guessing loops. How to instrument, budget, and detect thrashing.

Read more
Why LLM Confidence Scores Lie — and Break Your Human Gate
13Field Notes

Why LLM Confidence Scores Lie — and Break Your Human Gate

LLM self-reported confidence and logprobs are badly calibrated. Here's how that breaks human-in-the-loop gating in production agents — and the cheap fixes that work.

Read more
Golden Eval Datasets Rot: How to Keep Yours Honest
14Field Notes

Golden Eval Datasets Rot: How to Keep Yours Honest

Golden eval sets silently rot within months. How we source real edge cases, version evals alongside prompts, and refresh without invalidating history.

Read more
The Agent That 10x'd Traffic and Blew Its Budget
15Case Study

The Agent That 10x'd Traffic and Blew Its Budget

An anonymised case study of an AI agent that worked in pilot but blew the budget at scale. Where the cost went, the fixes, and the before/after numbers.

Read more
The Model Writes the Bug and the Confidence
16Field Notes

The Model Writes the Bug and the Confidence

AI-generated code ships exploitable flaws with total confidence. Why security review is the one step you can't delegate to the tool that caused the problem.

Read more
Treat AI Agents Like Untrusted Insiders, Not Magic
17Field Notes

Treat AI Agents Like Untrusted Insiders, Not Magic

DeepMind's AI Control Roadmap signals the shift from hoping models stay aligned to securing agents like insider threats. Here's the executive mental model.

Read more
The Eval Harness Is the Deliverable, Not the Agent
18Field Notes

The Eval Harness Is the Deliverable, Not the Agent

For agentic systems, the eval harness is what lets you ship with confidence and iterate safely. Here's what goes into a real one — and why teams pay for skipping it.

Read more
Illustration for "Why Your RAG Demo Works and Production Doesn't"
19Field Notes

Why Your RAG Demo Works and Production Doesn't

The specific failure modes that kill RAG systems after the demo — retrieval drift, chunk boundaries, stale indexes — and how to catch them with evals first.

Read more
Illustration for "Build vs Buy: AI Agents for Enterprise"
20Framework

Build vs Buy: AI Agents for Enterprise

A decision framework for CTOs and Heads of AI in 2026. Four paths — Build, Buy, Partner, Wait — with honest costs, timelines, and failure modes.

Read more
Illustration for "Why Most Enterprise AI Pilots Die in Month 4"
21Field Notes

Why Most Enterprise AI Pilots Die in Month 4

The month-four death has remarkably consistent causes. None of them are technical. Four organisational failure modes that kill AI pilots.

Read more
Illustration for "How Anglo American Cut Fleet Downtime 40% with Predictive Maintenance"
22Case Study

How Anglo American Cut Fleet Downtime 40% with Predictive Maintenance

Case study: 100+ vehicles, 2 years, 40% downtime reduction. What we built, why it worked, and the patterns worth stealing.

Read more
Illustration for "The Anatomy of a 6-Week Enterprise AI Project"
23Playbook

The Anatomy of a 6-Week Enterprise AI Project

There is a specific shape that enterprise AI projects take when they actually ship. Here's the week-by-week breakdown.

Read more
Illustration for "Buy vs Build vs Fractional: A Decision Framework for Enterprise AI"
24Framework

Buy vs Build vs Fractional: A Decision Framework for Enterprise AI

The question is almost never framed correctly. A framework for deciding how to stand up AI capability — honest about where each option breaks.

Read more
Illustration for "AI Automation for Enterprise: What Actually Works in 2026"
25Playbook

AI Automation for Enterprise: What Actually Works in 2026

Most enterprise AI projects fail. Here's what separates the ones that ship from the ones that stall — based on what we've seen across 6 industries.

Read more
Illustration for "Why More Companies Are Hiring a Fractional Head of AI"
26Leadership

Why More Companies Are Hiring a Fractional Head of AI

You need senior AI leadership but aren't ready for a full-time hire. Here's how the fractional model works and when it makes sense.

Read more