AI pilot to production
Our AI pilot worked in testing but nobody uses it six months later. What went wrong, and should we fund the next phase?
Last updated
Pilots often stall because they were tested on clean examples instead of live work, sat outside the system staff already use, or had no business owner once the project team moved on. Before funding the next phase, measure real use against a baseline, find out why staff stopped, and fund it only if the cause is fixable and someone in the business will own the tool.
Who this is for
This page is for operations, continuous improvement, IT and finance leads at mid-sized to enterprise organisations whose AI pilot has stalled, whether a Microsoft 365 Copilot rollout or a custom tool on the ERP, field system or inbox. Solve8 runs AI work in fixed-fee phases built to avoid this: discover one process with a measured baseline, prove it in shadow mode on live work, build it into the systems staff already use, such as Business Central, Pronto Xi, SAP or Microsoft 365, on your infrastructure in Australia, then support it or hand it over. We work in Sydney, Brisbane, Melbourne, Perth and across Australia and New Zealand.
How to decide whether to fund the next phase
Pull the usage data
Look at who used the tool, how often and the trend since launch. A sharp drop after the first few weeks usually points to the workflow or to trust rather than the technology.
Ask the intended users
Talk to five to ten people who were meant to use it. Ask what they do instead and what made them stop. Their workaround usually names the real problem.
Retest on recent live work
Run the tool on a fresh sample of real documents or requests, not the pilot test set. Accuracy that holds on live work means the problem is adoption. Accuracy that drops means the build needs work.
Compare against a baseline
Measure time per item, error rate or turnaround for the manual process and for the tool. If no baseline was taken, collect one now from the manual process.
Name the cause
Match what you found to the causes in the table below. Each has a different fix and a different cost.
Decide in writing
Choose fund, fix then fund, or stop. Write down the stop and go criteria, the owner and the date of the next review before any money is approved.
What usually goes wrong between pilot and production
| Cause | Sign | Fix |
|---|---|---|
| Tested on curated data | Results drop on live documents or unusual cases | Retest on live samples and route low-confidence results to a person |
| Outside the daily workflow | Staff had to open another screen and copy results across | Integrate the output into the ERP, inbox or job system they already use |
| No business owner | Errors are not fixed and rules drift as the process changes | Name an owner in the business, not only in IT |
| No agreed measure | People argue about whether it worked | Agree one or two metrics and a baseline before the next phase |
| Low trust | Staff check every output, so no time is saved | Show the checks, run in shadow mode and publish agreement rates |
| Training done once | New starters never learned the tool | Build it into induction and the process documents |
What Australian guidance and trials say
Treasury's evaluation of its part in the Australian Government trial of Microsoft 365 Copilot found the tool useful for some purposes, but its value for more complex work tasks was limited, and privacy and security restrictions limited the product's quality. The lesson for any pilot is that a general tool tested on general tasks can struggle to prove value on a specific process.
The National AI Centre's Guidance for AI adoption includes testing an AI system before deployment and monitoring it after deployment. Its implementation guidance recommends clear acceptance criteria that reflect the intended use, and documented deployment authorisation from the accountable person based on test results. Treat that as the shape of a production decision: agreed criteria, test evidence and a named person who signs off.
When to stop instead of funding more
The process changed
If the work the pilot targeted has moved, shrunk or been redesigned, the business case no longer holds.
Volume is too low
If the process runs a few times a week, the saving may never cover running and support costs.
The data is not available
If the tool needs data that staff cannot or will not capture, fix the data first or stop.
A standard feature now does it
Your ERP or productivity suite may have added a feature that covers the need. Check before rebuilding.
Nobody will own it
Without a business owner, a production tool decays the same way the pilot did.
Who should own it
Process owner
Owns the outcome, the metric and the decision to keep or retire the tool.
Continuous improvement manager
Runs the review, the user interviews and the baseline comparison.
Application or IT manager
Owns integration, access, monitoring and support.
Finance sponsor
Approves funding against the written stop and go criteria.
Related work
AI business case toolkitEstimate benefits honestly and set stop and go criteria.
Purchase orders into ERPAn example of AI built inside the ERP workflow with checks and approval.
Custom AI developmentAI built into the systems your staff already use.
Process automationLess retyping, chasing and manual checking.
Sources
Australian Treasury, Evaluation of generative artificial intelligence: https://evaluation.treasury.gov.au/publications/evaluation-generative-artificial-intelligence
National AI Centre, Guidance for AI adoption: foundations: https://www.ai.gov.au/staying-safe-and-responsible/essential-ai-practices/guidance-ai-adoption-foundations
National AI Centre, Guidance for AI adoption: implementation guidance: https://www.ai.gov.au/staying-safe-and-responsible/essential-ai-practices/guidance-ai-adoption-implementation-guidance
Questions
How do we measure whether staff are actually using an AI tool?
Use the tool's own logs to count active users, items processed and repeat use per week, and compare that with the number of people and items the tool was meant to cover. Then check outcomes, not just logins: how many outputs were accepted, edited or discarded. A tool opened often but whose outputs are always rewritten is not saving time.
What usage level should an AI pilot reach before it moves to production?
There is no universal threshold. Set it from the business case: the share of items the tool must handle, the accuracy it must reach on live work and the time it must save per item for the benefits to cover running costs. Agree those numbers before the next phase so the decision is not argued after the fact.
Should we restart a stalled AI pilot with a different vendor?
Only if the cause was the technology. If the pilot stalled because it sat outside the workflow, had no owner or had no agreed measure, a new vendor will hit the same problems. Retest the current tool on live work first. If accuracy holds, fix adoption. If it does not, compare vendors on your own documents.
How long should an AI tool run in shadow mode before people stop checking every output?
Long enough to cover normal variation in the work, including month-end peaks, unusual customers and new staff. Compare the tool's output with staff decisions item by item and relax checking only for categories where agreement is consistently high. Keep a person reviewing low-confidence items and anything with financial or safety impact.
What should a production business case include that the pilot case did not?
Measured results from live work against a baseline, the full running cost including support, monitoring and model or licence fees, the integration work needed to put the tool inside the daily workflow, a named business owner, and written stop and go criteria with a review date.
Who should own an AI tool once the project team moves on?
The manager who owns the process the tool supports, such as the accounts payable lead or customer service manager. IT owns the platform, access and monitoring, but the business owner decides when rules change, reviews accuracy and decides whether the tool stays.
Related
- AI business case toolkitA one-page business case, a vendor checklist and the security questions IT will ask, so you can take an AI pilot to your COO, CFO and CIO. Copy it, print it or forward it.
- Customers email us PDF purchase orders and staff retype them into the ERP. Can AI do this reliably?How AI reads emailed PDF purchase orders, checks them against customer, item and price data, routes exceptions to a person, and writes orders to your ERP.
- Custom AI developmentAI that reads, checks and drafts the first pass.
- Process automationLess retyping, chasing and manual checks.
Have this problem across your branches?
A 30-minute review with a senior consultant. You leave with a written one-page summary you can share internally.
Fixed-fee phases, agreed before we start. Stop after any phase.