In brief: an AI pilot needs a business outcome, an invested owner and evidence that it can handle the work people will give it. I use three tests to decide whether to stop, fix or scale. Set a decision date within a month, then fund only the next step the evidence supports.
We expected each person to use an AI agent four or five times a day. What I saw was almost no daily usage and only one or two uses a month.
That was the point at which we knew the pilot had stalled.
The agent hadn't earned its place in people's work. Users weren't getting the outcomes they expected, and some real use cases hadn't been discovered or tested in staging. When those cases failed, trust took another hit. People didn't consistently report the problems, so the team couldn't see everything that needed fixing.

I've also seen an ownership problem up close. A department manager was supposed to benefit from new features, but he wasn't excited about using them. He didn't see a need for custom charts and dashboards until someone noticed him struggling with a task and pointed out that the pilot could help. The company owner had a clear vision for making that job easier, but his calendar was busy and he didn't have time to own the work himself.
The lesson for me was uncomfortable. We hadn't secured the right internal champion and kept that person as invested in the outcome as we were. That is our responsibility too. Naming someone on a project plan is easy. Making sure they believe in the change and have time to lead it takes work.
A stalled AI pilot needs a decision
When a pilot struggles, the conversation can move quickly to a newer model, another vendor or more budget. I understand why. People have tried the tool and found it unreliable. They say it hallucinates, doesn't handle their tasks well, or that they could do the work more accurately themselves.
Those complaints deserve attention. They tell us where to investigate. A model change might help a particular failure, but it won't give a busy department manager time to champion a tool, or explain to users why they should change a habit that already lets them finish their work.
I want a decision on the calendar. Give the team up to a month to examine the pilot, establish what is missing and come back with a recommendation: stop, fix a specific gap or scale under agreed conditions. Some investigations and approvals will take longer. The date is for an honest decision about the next investment, not permission to rush into production.
For me, that decision rests on three tests.

Test 1: is the AI pilot worth putting into production?
Start with the work the business wanted to change. What result was the pilot supposed to produce? Who would experience that result, and what did the work cost or take before the pilot?
A business owner might want a manager to build useful charts without waiting for someone else. Another team might want to resolve a particular category of customer request faster. Those aims give us something to examine. "People liked the demo" does not tell us whether either job became easier.
I'd ask the owner to identify a business metric and an initial baseline, then compare the pilot's actual outcome against it. Include the time spent checking or correcting the output. Our guide to measuring AI ROI covers the baseline in more detail.
Then look at the money. What will the system cost at the expected volume, including testing, infrastructure, review and support? Who has agreed to fund the next phase? If a pilot needs another round of work, I want an estimate for that round and a clear view of what decision it will support.
A missing baseline isn't a reason to invent a return. Recover what you can from the existing workflow, label estimates and decide whether the remaining uncertainty is worth funding. If the team cannot identify a meaningful outcome, I would stop and reconsider the use case.
Test 2: does the pilot have an owner who will champion adoption?
The owner I want is close enough to the work to understand the benefit and has enough authority to make the change happen. They need time to meet the users, make decisions on the pilot's scope and follow up when something breaks. They should also be prepared to say, "This isn't helping my team yet."
Ask the proposed champion to show you where the agent fits into their own day. Which task will they use it for this week? What would make the answer useful? Who will help if it gets something wrong? If the owner struggles to answer, the next meeting should be about that gap, not the launch announcement.
User buy-in deserves the same attention. People need a reason to try the solution, a clear way to use it in an actual task and a reason to return after a bad experience. Show them what the tool can do for their work. Give them a way to report a failure without interrupting their day, then tell them what changed as a result.

Our article on why AI agents fail goes into ownership, evaluation and handoffs in the system. At the pilot review, I want to see the business side of ownership as well: a named champion, time in their calendar and an agreement about who acts on feedback.
If nobody can take that responsibility, pause the rollout. Adding users to an agent that the business isn't ready to support can make it harder to win them back later.
Test 3: is the pilot ready for production?
A demo can be exciting. It can also make a team forget how much diligence sits behind a production system.
I'd like "ready for production" to be agreed before the pilot begins. For the work in scope, the business and technical leads should be able to show:
- A named business stakeholder, an agreed outcome and a metric that can show whether the investment is working
- Discovered use cases, with the intended cases tested and approved, and exclusions or unresolved cases recorded
- A golden dataset of representative questions or inputs with verified expected results, and an evaluation pipeline the team can rerun
- Data sovereignty, privacy and security requirements identified and met for the proposed use
- User acceptance testing on live production workflows for two to four weeks, with appropriate access, human oversight and safeguards for the risk involved
- Observability and audit requirements met, including someone responsible for reviewing what the system did
- A feedback loop for successful and failed cases, with a route to investigate and correct problems
- A chosen model and configuration recorded for the release, with changes assessed and tested before rollout
- Expected production usage and full operating costs projected, with budget approvals in place
These checks draw on the AI 360 Compass. Evidence and controls should fit the use case: an internal drafting aid and an agent changing financial records will need different approvals. Record the model and configuration being approved, and test any change before release.
What Insifi taught us about demo confidence
We learned something important with Insifi, our internal tool for asking questions of company data: a confident answer could still be wrong, as we describe in our article on AI evaluation.
My takeaway is that we treated a demo too much like a working application. We needed to be better prepared with evaluations, a golden dataset and production-like questions that reflected what people would actually ask.
For a sufficiently broad question-answering workflow, I'd want the team to work toward testing at least 500 production-like queries as a planning target, alongside the cases that matter most. That number is not a universal release threshold. The right coverage depends on the scope and risk, and the team should be able to explain why its test set is representative. A smaller set can also reveal valuable problems early. The published Insifi article describes the 200 to 300 handwritten questions used in its initial evaluation.

My AI agent pilot scorecard
I'd put these questions on one page for the decision meeting. A "no" does not automatically mean the idea is dead. It tells us what must change before we can responsibly spend more or widen use.
Review the evidence with the business owner, technical lead and the people who will use the system. Bring in security, compliance or finance when their decisions are needed. A blank answer is an open issue, not a pass.
If you're facing this decision, talk to our team at Aubergine. Bring the pilot's intended outcome, what users are experiencing and the evidence you have. We can work through the gaps and the next decision with you.
Stop, fix or scale: what each decision means
Stop. If there's no credible business outcome, no committed champion or an unacceptable risk that cannot be addressed within the available constraints, stop the pilot. Don't keep pushing a rushed demo into people's work while it damages their trust. Record what would need to change before you revisit the idea.
Fix. If the opportunity remains worthwhile and the gap is specific, pause the affected use and agree on a bounded fix. Perhaps the team needs to test a class of queries it missed or find a champion with the time to work with users. Assign an owner, a budget and a date for reviewing the evidence. Reintroduce the solution only when the agreed conditions are met.
Scale. If the three tests are satisfied for a defined scope, expand deliberately. Record which users and tasks are included, who supports them, how the system is monitored and who can pause it. A successful pilot for one workflow does not automatically approve every other use.
When a pilot has lost trust, has no internal champion and puts a rushed demo into production, I'd stop it to prevent further damage. Fix what it was supposed to do before bringing it back. That can be the right commercial decision even when a team has already invested heavily. The next dollar has to earn its place.
How to make the call within a month
A month is a decision window. It is not a substitute for the two-to-four-week user acceptance testing or approvals the work requires. If a readiness gate cannot be completed safely in that window, the month-end decision can be to fund a defined next stage or stop. Production waits for the evidence.

- Week one: bring the original outcome, expected and actual use, known failures and current costs into one review. Identify the business owner and speak to the people expected to use the pilot.
- Week two: agree on the use cases that matter, recover the baseline where possible and review the test results. Find out which real tasks were missed and whether data, security or access constraints block the intended scope.
- Week three: price the work needed to close the material gaps. Confirm who will champion adoption and who has authority to approve the budget. Start or continue controlled user acceptance testing where the prerequisites are in place.
- Week four: review the three tests with the decision-makers. Stop, fund a specific fix or approve a limited expansion when the agreed readiness criteria are met. Record any outstanding gate, who owns it and the date it returns for a decision.
FAQs
Why do AI pilots stall before production?
A pilot can stall when it lacks a business owner who can lead adoption, fails to deliver a useful outcome or encounters real use cases it was not tested to handle. Cost, security and an unclear path to production can also block progress. Examine the evidence for the specific pilot before choosing a fix.
How do we move an AI pilot to production?
Agree on the business outcome, secure an active champion and define production-readiness criteria. Test the intended use cases, meet the relevant security and data requirements, and complete user acceptance testing with appropriate safeguards. Approve expansion only for the scope supported by the evidence.
Who should own an AI pilot?
A business stakeholder close to the work should own the outcome and champion adoption. A technical lead owns the technical work, while security, finance and other functions approve the parts within their responsibility. The business owner needs time and authority to act.
When should we stop an AI pilot?
Stop when the value cannot justify further investment, an accountable champion cannot be secured or the risks cannot be controlled within the available constraints. If trust has been damaged by a rushed release, pause the affected use and decide what must be fixed before bringing it back.
Is four weeks enough to approve an AI pilot for production?
Four weeks can be a useful period for a stop, fix or scale decision. It is not a guarantee of production readiness. If testing or approvals are incomplete, agree on the next bounded stage and its decision date; do not waive required controls to meet the calendar.


.jpg)


