What a 50% accuracy score taught us about building AI agents

The necessary AI eval: Building AI agents you can trust

Every team building AI agents will tell you evals matter. Fewer will tell you why they matter more than almost anything else in the system, or what a real eval practice looks like once you get past the slide that just says "evaluation."

The most dangerous failure mode in an AI agent is the one that answers fluently, confidently, and wrong. No hedge, no "I'm not sure." Just a clean, plausible, incorrect answer, delivered with the same tone as a correct one. On the other hand, most teams are looking in the wrong direction; expecting these failure modes to look like just crashes, times out, or visibly refusing to answer. 

This frequently occurs in natural-language interfaces for structured systems. These tools allow users to query databases in plain English and receive direct answers. It's a compelling product on day one. It's also a category where a misread schema, an ambiguous join, or a misunderstood column name can produce an answer that looks completely reasonable and is entirely wrong. The agent has no built-in way to know it got something wrong. That's the whole problem, and it's structural, not a bug you patch once and move past.

Teams evaluating agent reliability often anchor on the wrong signal because of this:

• Smooth day-to-day usage
• High query volume
• No complaints last sprint

This feels like stability, but it isn't. It's the absence of a hard question, not the presence of a good answer, and that gap is exactly where confidently wrong responses hide.

The fix lies in infrastructure, and it's called AI evals. 

What this looked like when we didn't catch it

We ran into this ourselves with an internal tool called Insifi. It connects to Nexus, Aubergine's internal system for timesheets, employee data, and project allocation, and lets anyone ask a plain-English question about that data instead of digging through a spreadsheet.

Early testing looked solid:

• A handful of engineers used it for quick lookups
• The answers checked out, and word spread
• Individual failures that came up (a misread schema, an ambiguous join) were easy to diagnose and fix
• The harder problem was that something working cleanly one week would break the next, with no way to know when or for whom
• Based on usage volume without complaints, the internal read was that accuracy was in good shape

This confidence turned out to be misplaced. Someone from leadership asked how many hours the design team had logged on a specific project the previous month. Insifi returned a clean, specific number, stated exactly like every other answer it had given that week. Checked against the actual timesheet records in Nexus, the number didn't hold up:

• Insifi had matched hours from a similarly named project
• It folded in contributors who weren't on that team at all
• Nothing in the response signaled a problem, no hedge, no lower confidence, no indication the join underneath it was shaky

Individually, the failures behind that answer were mundane. The risk was never any single mistake. It was that the agent never once signaled uncertainty while making them. A wrong answer that sounds unsure is a minor annoyance. A confident but incorrect answer destroys trust. When leadership relies on a tool, repairing that broken trust is far more expensive than funding the evaluation work to prevent it.

The framework: what a real eval loop runs on

This is what we learnt. To start with a golden dataset, not a vibe. 

A golden dataset is:

• A golden dataset of real questions paired with verified ground-truth answers (via manual reviews or policy docs).
• Small enough to be manageable, a few dozen handwritten and synthetically generated questions is a legitimate start
• Built from actual use cases, not imagined edge cases, since imagined cases test assumptions about the product rather than the product itself

We build ours from three sources, and each catches a different kind of failure:

From there, build the simplest harness that scores automatically:

• A script that runs the test set against the agent and checks output against expected answers
• No dashboard required at this stage
• The value sits in the discipline of running it every time, not the tooling around it

Match the eval method to the failure type, deliberately. Not every output can be checked the same way, and treating them identically is where a lot of AI evaluation efforts fail.

A few rules of thumb for choosing between them:

• Structured outputs, a generated query for instance, can be checked deterministically: run it, compare the result set, done
• Open-ended or natural-language outputs usually need an LLM-as-judge pattern, since there's rarely one single correct string to match against
• High-stakes or low-confidence cases need a human in the loop, not either automated method on its own

Design the human-in-the-loop fallback before you need it, not after. Decide in advance:

• Which categories of query should trigger a hand-off to a person
• What confidence threshold should make the agent flag its own uncertainty
• Where it should decline to answer rather than guess

A system that hands off or flags itself is recoverable. This whole practice exists to prevent highly confident, incorrect guesses.

Once we built a golden dataset and ran it properly:

• Roughly 200 to 300 handwritten questions, scored by hand
• The AI scores from that first real run pointed straight at specific join types, naming conventions, and phrasing patterns that were quietly wrong
• Every failure became a fix, and every fix ran back through the same test set before it counted as a real improvement

We'd written earlier about the automation work that led to Insifi in how we used AI-driven workflow automation to improve product workflows. What's described here is the discipline that had to exist before any of that automation could be called dependable.

Some quick tips on building your own golden dataset

• Start with a golden dataset built from real usage, not hypothetical questions
• Build the simplest scoring harness that runs automatically, a script and a spreadsheet is a fine first version
• Match the eval method to the failure type: deterministic checks for structured output, LLM-as-judge for open-ended answers, humans for the calls that genuinely need judgment
• Decide in advance where the agent should hand off to a person instead of guessing
• Treat every real failure as a permanent addition to the eval set
• Re-run the full set on every change, not just the case that triggered it
• Continuously enrich eval suite by fetching production failures from user feedback
• Scope coverage to what the agent is actually meant to handle, not the entire universe of possible questions

Regression eval deserves its own discipline

This is the part most teams underbuild, and it's usually the one that matters most once a tool is live.

• Every change to an agentic system, a prompt tweak, a schema update, a new tool added to what it can call, needs to run against the full historical failure set, not just the new case that prompted the change
• This matters more in agentic systems than in traditional software, not less, because these systems are non-deterministic
• Fixing one failure mode can reintroduce or worsen another with no warning sign
• Without a full regression pass, that only shows up once it hits production again, weeks later, for a different user, debugged as if it were new

Regression testing itself isn't new. Re-running a known set of cases after every change to confirm nothing that used to work has broken is decades-old software discipline. Most agentic products simply haven't imported it yet, and treat regression testing as something to add later, if there's time. There usually isn't.

Eval maturity has to track product maturity

An eval suite isn't something built once and considered finished. It has to mature along two axes at once.

Capability

• A pre-launch eval practice can run on a small, manually scored golden set and still do its job
• As a tool moves into real usage, it needs automated scoring and category-level breakdowns of where accuracy is weak
• Eventually, it needs a pipeline that runs on every change without anyone needing to remember to trigger it

Coverage

• As usage grows, the distribution of real queries shifts with it: new phrasings, new domains, edge cases nobody anticipated at launch
• A golden set that isn't actively growing goes stale, and a stale eval set is worse than none at all, since it gives false confidence instead of an honest signal
• Every production failure is the highest-value test case available, specific, free, and proof of an actual gap rather than a hypothetical one

This is the shift that actually matters: moving failure discovery from "someone noticed and complained" to "the eval caught it before deploy."

The honest answer

Eval infrastructure has a cost, and it's easy to treat that cost as overhead against shipping speed. It isn't. It's the cheaper failure mode, chosen deliberately, over the far more expensive one: a confident, fluent, wrong answer reaching someone who acts on it before anyone catches it.

None of this is unique to any one team:

• Gartner expects more than 40% of agentic AI projects to be shelved by the end of 2027, pointing to unclear business value and inadequate risk controls as leading causes
• MIT's 2025 review of enterprise AI deployments found that the vast majority never produced a measurable business return, largely because the tools in question couldn't adapt to real workflows or retain feedback from actual usage

An eval practice is one of the few parts of this that's fully within a team's control. If someone asks how you trust a system that behaves non-deterministically, the answer starts in the same place every time: with a mindful and well-thought-through eval.

Building AI agents your business will actually depend on

Golden datasets and eval loops aren't a side project thrust onto an agent once it's live. They turn a novelty tool into one leadership trusts for real decisions. If you're building agentic tools internally and want a second set of eyes on how you're evaluating them, or you're early enough in the process to build this in from the start, let's talk.