AI Pilot Projects: How to Survive the Go/No-Go Review

An AI pilot project succeeds when it produces a defensible go/no-go decision. That depends on five choices made in the first week: a narrow use case, a measured baseline, a representative cohort, success criteria and kill criteria signed by the executive sponsor. This 90-day playbook is how we approach enterprise AI consulting engagements in Australia, from kickoff to readout.

It is written for the COO, programme manager or CIO at a mid-sized, enterprise or multi-site Australian organisation about to launch its first or next AI pilot. If you have already worked through the board-level business case and the payback maths, this covers the 90 days between the go decision and the go/no-go review. For what happens after a go decision, see operating AI agents in production.

Why do AI pilot projects fail?

Four failure modes account for most cancelled pilots. If your pilot is heading toward one of them, a better model or vendor will not save it.

	Failure Pattern	Root Cause and What Good Looks Like	Impact
Inconclusive pilot	Cannot prove improvement or harm. The steering committee asks 'did it work?' and the answer is 'sort of'.	Root cause: no clean before/after baseline. Good: 4 weeks of pre-pilot measurement on the same metric the AI will move.	Defensible
Successful demo, failed scale	Sandbox results look strong, production results disappoint. The vendor blames data, the business blames the vendor.	Root cause: pilot ran on cherry-picked data with no integration load. Good: production data, real volumes and real exception rates.	Scalable
Adoption collapse	The pilot team likes it, the broader team ignores it after handover.	Root cause: change treated as training instead of job redesign. Good: cohort selected to represent normal operations.	Sticky
Compliance retrofit	Pilot worked, but privacy, Fair Work or regulator review surfaces issues that cannot be retrofitted.	Root cause: Privacy Impact Assessment and security review skipped. Good: PIA completed before production data touches the system.	Defensible

All four share one pattern: the pilot was designed to demonstrate enthusiasm instead of producing a decision the board can accept.

What five design decisions decide an AI pilot's outcome?

These decisions are made, or avoided, in the first seven days. Every failed pilot we have reviewed missed at least two.

1. 1Use case scopingOne process, one data source, one cohort. Narrow enough that movement in the metric is attributable to the pilot.
2. 2Baseline definitionMeasure the same metric for at least 4 weeks before the AI does anything.
3. 3Cohort selectionRepresentative of normal operating conditions, large enough that results generalise.
4. 4Success criteriaPre-committed in writing by the executive sponsor: specific metric, target and window.
5. 5Kill criteriaPre-committed stop conditions covering safety, compliance, cost overrun and adoption floor.

The most common omission is kill criteria. Without them, a pilot runs on "we will continue if it looks promising", and it never gets stopped even when the data says it should. Writing go/no-go thresholds down in advance is consistent with risk frameworks such as the NIST AI Risk Management Framework and Australia's Guidance for AI Adoption, which both emphasise defined accountability and measurement.

For who should staff these roles, see the AI agent staffing gap. For vendor scoping, run the vendor selection question set before kickoff.

What does a 90-day AI pilot framework look like?

Compress a pilot to 30 days and you are running a demo. Stretch it past 120 and the sponsor disengages and the goalposts move.

1. Days 0-7Kickoff and charterCharter signed by the executive sponsor. Governance membership confirmed. Baseline measurement begins, backfilled with 4 weeks of historical data. PIA scoping decided. Kill criteria committed in writing.
2. Days 8-30Build and integratePlatform configured. Production integrations stood up. Testing on real, de-identified data. PIA completed per OAIC guidance. Security review completed. Workforce consultation started where roles change.
3. Days 31-60Parallel runAI runs alongside the existing process. No decision is taken on AI output alone. Daily metric capture, weekly cohort stand-up, fortnightly steering review. Exceptions logged with reason codes.
4. Days 61-75AssessmentCompare parallel-run data to the baseline. Measure daily active use. Calculate fully loaded cost per transaction. Audit a stratified quality sample.
5. Days 76-90Go/no-go decisionExecutive readout with three options: scale, iterate, stop. Each has a budget envelope and a decision owner.

The most important calendar item is the Day 0 baseline. If you cannot state the cycle time, error rate and cost per transaction for the four weeks before the pilot, drawn from operational systems, the pilot cannot produce a defensible decision however well the AI performs.

The parallel run is where our approach differs from many vendor templates: the AI runs alongside your team until it proves accurate, and a person approves before anything reaches a customer or system of record.

Which AI pilot metrics survive a go/no-go review?

Boards accept measurable, attributable change against a baseline.

	Defensible Metric	How to Measure	Impact
Cycle time	Median minutes from input received to output produced.	Pull from the workflow system and compare the cohort to its own 4-week baseline.	Survives
Error rate	Share of outputs needing correction within 30 days.	Stratified audit sample, same method as the baseline. Track AI and human override errors.	Survives
Adoption rate	Daily active use across the cohort.	System telemetry on a 30-day rolling window against a pre-committed floor.	Survives
Cost per transaction	Fully loaded: model, integration, oversight time, exception handling.	Finance reconciles against actual spend and time records.	Survives
Quality / accuracy	Domain-specific, such as classification accuracy.	Held-out test set audited by a subject expert independent of the vendor.	Survives
Time to output	Median and 95th percentile latency at production load.	Measured at peak hour. The 95th percentile matters for service levels.	Survives

	What People Try to Count	Why It Fails Scrutiny	Impact
Users said it was great	Survey, often from the enthusiast cohort.	Not attributable or generalisable.	Reject
Pilot completed on time	Project management metric.	Measures the team. Many on-time pilots are inconclusive.	Reject
Vendor was responsive	Relationship metric.	Useful for vendor selection, irrelevant to go/no-go.	Reject
Demo went well	Presentation outcome.	Demos are curated. Production is not.	Reject
Number of users onboarded	Counts access.	Logins are not adoption. Daily active use is.	Reject

What anti-patterns kill AI pilot projects in Australian organisations?

None of these is about the model.

1. No baseline. Measurement starts on go-live day, so there is nothing to compare against. Fix: four weeks of pre-pilot measurement on the same metric, cohort and definitions.
2. A cohort that is too small or too keen. Three enthusiasts produce results that do not generalise. Fix: a mix of high, mid and low performers across at least two sites or teams. For multi-site organisations, include a site that does the process differently.
3. A "free" vendor pilot with no exit clause. By day 90 the integration is so deep that switching costs lock you in. Fix: exit clauses, data export rights and a switching plan agreed up front. See why DIY without understanding costs more.
4. Synthetic data. Production data has edge cases, formatting variance and personal information that synthetic data lacks. Fix: production data, de-identified where the PIA requires, from week one of build.
5. No change management. Adoption stalls because the workflow was not co-designed with the people who do the work. See AI change management and employee adoption.

What governance does an AI pilot need?

A pilot is not a regulatory exemption. Australia's voluntary guidance from the Department of Industry, Science and Resources (the 2024 Voluntary AI Safety Standard, followed by the Guidance for AI Adoption in October 2025) applies from pilot to production. Deferring compliance "until scale" is a common reason a working pilot cannot be scaled.

1. 1Privacy Impact AssessmentThe OAIC recommends a PIA for projects handling personal information. Complete it by Day 21.
2. 2Security reviewReview against a structured AI security checklist before production data is used. Document gaps and accepted risks.
3. 3Workforce consultationPilots that materially change employees' work can trigger consultation obligations under modern awards and enterprise agreements.
4. 4Audit logging from day oneCapture inputs, outputs, model version and human override decisions from the first transaction.
5. 5Industry overlayFinancial services, healthcare, aged care, government supply and consumer-facing work each add obligations.

Useful references: our 50-point AI security checklist, Fair Work compliance for AI automation, financial services AI compliance, aged care compliance, government contract requirements and ACCC consumer guarantees. Where data crosses borders, our explainer on Privacy Act versus GDPR covers the pilot-stage implications. Running the pilot on private AI infrastructure in Australia removes many cross-border questions at the start.

Is your AI pilot designed to produce a defensible decision?

If you cannot answer "yes" to all five, the pilot will not produce a decision the board accepts.

Can your pilot survive a steering committee challenge?

• Baseline measured for at least 4 weeks on the same metric and cohortYes: defensible direction. No: the result will be inconclusive.
• Success criteria committed in writing by the executive sponsorYes: protects against moving goalposts. No: results get reinterpreted to favour continuation.
• Kill criteria committed and the sponsor will respect themYes: a real decision is possible. No: the pilot drifts indefinitely.
• Cohort large and representative enough to generaliseYes: the scale decision is defensible. No: scale surfaces failures the pilot missed.
• PIA and security review done before production data was touchedYes: results can scale. No: a compliance retrofit will block scale.

If three or fewer are "yes", do not start yet. Spend two weeks fixing the gaps. That costs far less than an inconclusive 90-day pilot.

The 12-question pre-pilot readiness checklist

Use this as the kickoff agenda. If a question cannot be answered, resolve it before proceeding.

1. What single business metric will this pilot move (cycle time, error rate, cost per transaction, conversion)?
2. What is its pre-pilot baseline, drawn from operational systems?
3. What is the target value and date, signed by the executive sponsor?
4. What are the kill criteria (cost, safety, compliance, adoption floor), each with a threshold and owner?
5. Who is the executive sponsor, and what is their fortnightly time commitment?
6. Who is in the cohort, and is the selection representative?
7. What production data will be used, and what is the PIA status?
8. What is the integration plan, and the rollback plan if a production system is destabilised?
9. Who owns change management, and what does the workflow look like after the pilot?
10. What is the audit logging design, and who reviews the logs weekly?
11. What is the vendor exit clause and data export design?
12. What is the Day 90 readout format, and who attends?

What does an AI pilot really cost?

Cost a pilot honestly across all five categories, not just the platform:

• Platform and model usageOften the smallest line
• Internal time: sponsor, owner, subject experts, change leadUsually the largest line
• Integration and data preparationScoped up front
• PIA, security review and compliance workDone before production data
• Change management and trainingBuilt in
• Decision standardFully loaded cost per transaction vs baseline

A pilot budget that leaves out governance, change and baseline measurement is underspending on the work that decides whether the result survives review. For post-decision payback maths, see our payback period calculator article.

Vendor-supplied pilot templates are usually built to demonstrate the vendor's product. The framework above replaces them with a governance-aligned alternative. For the decisions that come before a pilot, see the AI model selection guide and our AI pilot to production guide.

How Solve8 can help

Our consultants design pilots that produce defensible go/no-go decisions, then build and run them on your infrastructure in Australia, alongside your team. Each phase has a fixed fee agreed before we start, and you can stop after any phase. See our AI strategy and managed AI services pages, or talk to us about a pilot design review.