Operating AI agents in production is an ongoing cost that usually exceeds the build. After launch, Australian organisations pay for model fees, monitoring, regression testing, human reviewers, privacy reviews, incident readiness and forced model migrations. Most failures appear between months three and 24, so plan the operating model before you sign a build contract.
Key takeaways
- An AI agent is not deploy-and-forget software. The model, the inputs and the rules around it all keep changing.
- Six failure modes appear after launch: harmful output, binding misstatements, data leakage, damaging actions, drift and discrimination exposure.
- Build cost is a minority of total cost. Model fees, evaluation, human review, governance and migrations make up the rest.
- The operating loop is known. Define, evaluate, deploy, monitor, detect, review, update, using NIST AI RMF, ISO/IEC 42001 and OWASP.
- Decide who carries the burden. Build and run it yourself, partner for operations, or buy a bounded product someone else runs.
Why do AI agents in Australia fail after launch?
Most guidance on AI agents in Australian organisations covers the build: choose a model, connect tools, write prompts, ship. Our post on why DIY AI agents fail without understanding covers week one. This one covers months six, 12 and 24.
An AI agent is a system that keeps changing: the vendor's model changes, the inputs change, and the regulations change. The build cost is the visible tip of a long operating tail. Every incident below is published and verifiable. Where we illustrate operating mechanics, examples are clearly hypothetical.
What are the six failure modes that appear after deployment?
1. Brand-damaging output reaching real users
In January 2024, delivery firm DPD disabled part of its customer chatbot after a customer prompted it to swear and write a poem criticising DPD. The screenshots went viral within hours. Without output guardrails, runtime moderation and adversarial testing, almost any customer-facing agent can be steered into outputs nobody approved. That needs continuous probing, not a one-off pre-launch test.
2. Misstatements the business is bound by
In Moffatt v Air Canada (2024), the British Columbia Civil Resolution Tribunal held Air Canada liable for inaccurate bereavement fare information from its chatbot, rejecting the argument that the chatbot was responsible for its own statements.
Under the Australian Consumer Law, representations made by an automated agent on behalf of a business are the business's representations, and the misleading or deceptive conduct provisions apply. If your agent quotes a price, promises a refund or misdescribes a product, you may be held to it. Every customer-facing output should either be constrained to verified content or routed to a person for any statement that makes a commitment.
3. Data leakage through staff use and prompt injection
In April 2023, Samsung restricted generative AI use after engineers pasted proprietary source code and meeting notes into ChatGPT. We cover the wider pattern in how the wrong AI tools leak business data.
Prompt injection is the main external attack. The OWASP Top 10 for LLM Applications ranks it first (LLM01). In December 2023, a US car dealership's chatbot was manipulated into "agreeing" to sell a vehicle for one US dollar. The offer was unlikely to be enforceable, but the reputational damage was immediate. Treat every prompt-injectable surface as untrusted input, and control what staff can paste into AI tools.
4. Agent actions that change production systems
In July 2025, an agentic coding assistant on Replit deleted a live production database during a session and gave misleading explanations when challenged. Replit's CEO apologised publicly and the company changed its agent's permissions.
Once an agent can act, by writing to a database, sending an email or calling an API that changes state, one bad inference can cause real damage in seconds. That is the line where governance becomes essential. Our post on AI agent ecosystem architecture covers permission boundaries.
5. Drift, model deprecation and prompt regression
A prompt that scored well on your test set in March can score noticeably worse in October with no change on your side. The vendor patched the model, a safety filter tightened, or a retrieval source changed format. Chen, Zaharia and Zou's 2023 study "How Is ChatGPT's Behavior Changing Over Time?" measured drift in maths, reasoning and code tasks between two snapshots three months apart. Vendors also retire model versions on published schedules, forcing migration.
Running a production agent means running a regression test treadmill: a maintained set of test inputs with expected behaviour, automated evaluations on every prompt or model change, and engineering time budgeted for it.
6. Discrimination and privacy exposure
In August 2023, the US Equal Employment Opportunity Commission settled with iTutorGroup over an automated hiring screen that rejected older applicants, its first AI-related settlement.
In Australia, the Sex, Age, Disability and Racial Discrimination Acts apply to decisions made by or with AI, as the Australian Human Rights Commission's 2020 technical paper on algorithmic bias explains. The OAIC's October 2024 guidance on AI and privacy confirms that the Australian Privacy Principles, including APP 11 on security and retention, apply to personal information processed by AI, along with the Notifiable Data Breaches scheme. OAIC reports consistently show malicious attacks as the leading cause of notifiable breaches, with human error second. Agents with action authority increase both risks.
What does an AI agent really cost to operate?
Business cases that use build cost as a proxy for total cost understate it badly. The categories below belong in every 24-month budget, sized to your interaction volumes and risk.
| Typical build pitch | Full 24-month operating picture | Impact | |
|---|---|---|---|
| Initial build | Quoted | Plus rework after the first model migration | Often underestimated |
| Model API fees | Often estimated low | Monthly, growing with adoption | Grows with use |
| Monitoring and tracing | Rarely included | Monthly tooling and storage | Essential |
| Evaluation and regression testing | Not mentioned | Ongoing engineering time | Recurring |
| Human reviewers | Not modelled | Hours per week from your team | Recurring |
| Privacy impact assessment refresh | One-off if mentioned | Annual and on material change | Recurring |
| Incident readiness | Not modelled | Legal, communications and forensic standby | Recurring |
| Model migration and re-prompting | Not mentioned | Forced when versions retire | Per event |
| Audit log retention | Not mentioned | Privacy Act, NDB and sector rules | Monthly |
- Build, including one round of post-migration reworkOne-off
- Model fees at expected monthly volumeMonthly x 24
- Evaluation engineering and human review hoursWeekly x 104
- Governance, privacy reviews and incident readinessAnnual x 2
- Build as a share of the totalUsually the minority
None of this makes AI agents uneconomic. Many pay back strongly when they replace costly manual work. The point is to compare value against the full operating cost.
What does good production AI operations look like?
The operating discipline is now well documented:
- NIST AI Risk Management Framework 1.0 (January 2023) for risk identification and treatment
- ISO/IEC 42001:2023, the AI management system standard (see our ISO 42001 guide)
- OWASP Top 10 for LLM Applications (2025) for security threat modelling
- Australia's AI Ethics Principles and the Department of Industry, Science and Resources Guidance for AI Adoption (October 2025), which builds on the earlier Voluntary AI Safety Standard
- DefineSpecify intended behaviour, prohibited behaviour, and escalation criteria in writing
- EvaluateRun a maintained test set against the agent before any change ships
- DeployRelease to a limited group first with rollback ready
- MonitorTrace inputs, outputs, tool calls, latency, cost, and content flags
- DetectFlag drift, anomalous patterns, jailbreak attempts, policy breaches in near real time
- ReviewWeekly drift triage, monthly accuracy audit, quarterly governance review
- UpdateRe-prompt, refresh retrieval, adjust guardrails, then return to Evaluate
Pre-deployment evaluation. A test set of a few hundred to a thousand representative inputs with expected behaviour. Every change to the prompt, retrieval data, model version or tools triggers a rerun, and a drop below threshold blocks release. This is the most under-invested practice we see.
Scheduled regression testing. The same test set runs on a schedule against production, so silent drift from a vendor update is caught in days, not through complaints.
Written human approval rules. Not everything to a person, and not nothing. A written matrix: actions that always need approval (anything legally binding, anything changing a system of record, anything involving sensitive personal information) and actions the agent can complete alone. A person approves before anything reaches a customer or system of record until the agent proves accurate.
Incident response runbook. Including the path to OAIC notification. Under the Privacy Act, a suspected eligible data breach must be assessed within 30 days, and an eligible breach notified to the OAIC and affected individuals as soon as practicable.
- Stage 0ExperimentalInternal sandboxes, no production data, no external users
- Stage 1Bounded internal useAgent serves staff only, read-only access to non-sensitive data, every output reviewed
- Stage 2Supervised internal actionAgent drafts, suggests and prepares; a person approves before commit
- Stage 3Customer-facing with constraintsExternal users, narrow domain, strong guardrails, full monitoring, named on-call owner
- Stage 4Governed action on production systemsExplicit permission boundaries, continuous evaluation, an AI management system aligned to ISO 42001
Most mid-sized organisations we speak with sit between Stage 1 and Stage 2. Many vendor pitches assume Stage 4 while skipping the discipline Stages 2 and 3 require.
Should you build, partner or buy an AI agent?
The board question is: should we own the burden of running this in production?
Do you have in-house AI engineering, monitoring tooling, and on-call capability for a production system?
- Yes to all three, and AI is core to your competitive advantageBuilding may be defensible. Adopt NIST AI RMF and ISO 42001 from day one and budget the full operating cost.
- Yes to engineering, no to on-call or governanceHybrid. Build the domain capability, partner for operations, monitoring and governance.
- Use case is bounded and a credible product existsBuy. A vendor spreads the operating cost across many customers. Spend on integration and change management.
- Use case is novel, but you lack operations depthPartner-led. A partner builds and operates; you keep the strategy and the data.
- Not yet sure the use case justifies the spendStage 0 or 1 only. Run a 4 to 8 week structured experiment first.
For most mid-sized organisations, full in-house ownership is the wrong answer for most use cases. Either buy a bounded product someone else runs, or have a partner run operations while you keep the strategic and data assets. Our AI pilot to production guide covers making that call on measured numbers.
What 12 questions should be answered before an agent goes live?
- Which tasks can the agent perform alone, and which need approval? Is that written down?
- What is the regression test plan, test set size and pass threshold, and who maintains it?
- When the model is retired, who owns the migration and what is budgeted?
- Who is on call when the agent creates legal, regulatory or reputational exposure, and how fast do they respond?
- Where are prompts, traces and tool calls stored, for how long, and under what legal hold conditions?
- What is the data residency position for every model, retrieval service and monitoring tool involved?
- Has a privacy impact assessment been completed per OAIC guidance, and how often is it refreshed?
- What is the path from anomaly detection to Notifiable Data Breaches assessment and notification?
- How is prompt injection tested continuously, not just at launch?
- Is there an enforced policy on staff pasting sensitive data into unapproved AI tools?
- What is the rollback procedure if a change causes regression?
- What is the 24-month operating budget, and who in finance signed it off?
If any question has no answer, the agent is not ready for production.
How do we know these operating costs are real?
We build and run our own agentic products. RootCauseAI, our investigation product for IT support teams, runs inside the customer's own boundary because the data access risks above made cloud-only deployment unacceptable for its users. Carbonly.ai, our ESG automation product, runs multi-agent document extraction. Both have been through model changes, evaluation rebuilds and the monthly discipline of regression testing. For a real deployment write-up, see an AI agent for IT support investigations, and for governance, AI agent governance, data access and human override.
How Solve8 can help
Our consultants and engineers design, build and run AI agents on your infrastructure in Australia or New Zealand, with evaluation, monitoring and human approval built in from the first release. See our managed AI services and custom AI development pages, or talk to us before you commit to a build.
