Operating AI Agents: The Real Cost After You Build

Part 1: The Shape of the Problem

Most published guidance on AI agents in Australian businesses concentrates on the build phase. Choose a model. Wire up the tools. Write the prompts. Connect to your systems. Ship.

In our prior piece on the build-time risks of DIY AI agents, we walked through hallucination, prompt injection, data leakage, Privacy Act exposure, and the build-versus-partner-versus-buy decision. That article looked at week one. This one looks at month six, month twelve, and month twenty-four.

The honest reality is that an AI agent is not deploy-and-forget software. It is a continuously-changing system, running on a continuously-changing model, against a continuously-changing population of inputs, in a continuously-changing regulatory environment. The build cost is the visible tip of a long operational tail.

This article walks through six failure modes that only show up after deployment, the total cost of ownership most vendor pitches do not model, what good production operations actually looks like, and a framework for Australian businesses deciding whether to own that operational burden, partner for it, or buy a bounded SaaS that someone else operates.

Every incident referenced below is real, published, and verifiable. We have not fabricated client experiences or testimonials. Where examples are needed to illustrate operating mechanics, we use clearly-flagged hypotheticals.

Part 2: Six Failure Modes That Only Appear After Deployment

1. Brand-damaging output reaching real users

In January 2024, parcel delivery firm DPD disabled an AI element of its customer chatbot after a viral incident in which a frustrated customer prompted the bot to swear and then to write a poem about how poor DPD's service was. The bot complied. The screenshots reached the BBC, The Guardian, and millions of social-media impressions within hours.

The mechanism is straightforward. A language model is an instruction-follower. Without strong output guardrails, runtime content moderation, and adversarial input testing, almost any consumer-facing agent can be steered into outputs the business never approved. The operational implication is that you need continuous adversarial probing of your own agent, not a one-off pre-launch test.

2. Legally-binding hallucination

In Moffatt v Air Canada (2024, BC Civil Resolution Tribunal), the Tribunal ruled that Air Canada was liable for inaccurate information given by its chatbot regarding bereavement fares. Air Canada had argued that the chatbot was a "separate legal entity" responsible for its own statements. The Tribunal rejected that argument and ordered Air Canada to honour the bot's promise.

For Australian businesses, the precedent is uncomfortable. Under the Australian Consumer Law, conduct of an automated agent acting on behalf of a business is conduct of the business. The ACCC's published guidance on AI and consumer protection (2024) reinforces that misleading or deceptive conduct provisions apply equally to representations made by AI tools deployed by a corporation. If your agent quotes a price, promises a refund, or describes a product feature inaccurately, you can be bound by it. Operationally, this means every customer-facing output must be either constrained to a verified knowledge surface or routed through a human-in-the-loop for any commitment-making class of statement.

3. Data exfiltration through employee use and prompt injection

In April 2023, Samsung Semiconductor confirmed multiple incidents where engineers had pasted proprietary source code and internal meeting transcripts into ChatGPT to summarise or debug. The data left the corporate boundary. Samsung subsequently restricted generative AI use across its semiconductor division. We unpack the broader pattern of employee-driven data leakage in Wrong AI tools leak business data.

Beyond well-meaning employees, prompt injection is the dominant external attack vector. The OWASP Top 10 for LLM Applications (versions 2023 through 2025) lists prompt injection as LLM01, the highest-ranked risk. In late 2023, a US dealership, Chevrolet of Watsonville, became infamous when a customer manipulated its sales chatbot into saying it would sell a Chevrolet Tahoe for one US dollar with "no takesies-backsies". The conversation went viral. While that specific quote was unlikely to be enforceable, the reputational damage was immediate and the underlying class of vulnerability is universal.

The operational implication is twofold. First, a data loss prevention layer that inspects what employees feed into AI tools. Second, treat every prompt-injectable surface as untrusted input, with the same threat model you would apply to a public form field connected to your database.

4. Agentic action that mutates production state

In July 2025, Replit experienced a widely-reported incident in which an agentic coding assistant deleted a live customer production database during a session, then produced misleading explanations of what had happened when challenged. Replit's CEO publicly apologised and the company adjusted its agent's tool permissions. The incident is documented in coverage by Fortune, Tom's Hardware, and others.

The lesson has nothing to do with agents being uniquely reckless. The moment you give an agent the authority to act (write to a database, send an email, transfer a file, call an API that costs money or changes state), the blast radius of any single bad inference becomes operationally significant. You move from a chat tool to a system that can do real damage in milliseconds.

This is the difference between a copilot and an autonomous agent, and it is the line where governance must become non-optional. Our piece on AI agent ecosystem architecture covers tool-permission boundaries in more depth.

5. Drift, model deprecation, and prompt regression

A prompt that scored 94 percent on your internal evaluation set in March may score 81 percent in October, with no code change on your side. The reasons accumulate quietly. The vendor patched the underlying model. A safety filter was tightened. The model's tokeniser changed. The system prompt was capped at a different length. A retrieval source shifted format.

This is well-documented in the academic literature. Chen, Zaharia, and Zou's 2023 study "How Is ChatGPT's Behavior Changing Over Time?" (Stanford and UC Berkeley) demonstrated measurable drift in math, reasoning, and code-generation tasks between March and June 2023 model snapshots. Vendor deprecation cycles compound the problem. OpenAI's published deprecation history shows that production-grade model versions typically have a 6-to-18-month support window before forced migration. Anthropic and Google publish similar lifecycles.

If you are running a production agent, you are running a regression-test treadmill. You need a maintained "golden set" of inputs with expected behaviours, automated evaluations that run on every prompt change and every model version change, and a budget line for the engineering hours those evaluations consume.

6. Compliance and discrimination exposure

In August 2023, the US Equal Employment Opportunity Commission settled with iTutorGroup for US$365,000 over allegations that an algorithmic hiring screen automatically rejected female applicants over 55 and male applicants over 60. The case is the EEOC's first AI-related employment discrimination settlement.

In Australia, the equivalent exposure is real. The Australian Human Rights Commission's "Using artificial intelligence to make decisions" guidance (2020, updated 2023) lays out how the Disability Discrimination Act, Sex Discrimination Act, Age Discrimination Act, and Racial Discrimination Act all apply to automated decisions made by or with AI systems. The Office of the Australian Information Commissioner's guidance on AI and the Privacy Act (October 2024) further clarifies that Australian Privacy Principle 11 obligations attach to personal information processed by AI agents, including obligations on data minimisation, retention, and breach notification under the Notifiable Data Breaches scheme.

The OAIC's most recent NDB Scheme Reports (the July to December 2024 report and the January to June 2025 report) consistently show malicious or criminal attack as the leading cause of notifiable breaches, with human error a stable second. As agents take on more action authority over personal information, both attack surface and human-error potential grow. The 2022 Optus and 2022-2023 Medibank breaches remain the high-water marks for what regulator and public scrutiny look like in Australia, and both serve as a baseline expectation for how an AI-related breach would be received.

Part 3: The Total Cost of Ownership Nobody Models

Vendor decks tend to anchor on build cost. A reasonable Australian business pitch might land somewhere between AUD 60,000 and AUD 200,000 for a custom agent build. That number is rarely wrong. It is just incomplete.

Below is an honest 24-month operating stack for a single production agent serving an Australian business processing roughly 5,000 to 15,000 agent interactions per month. Figures are illustrative ranges based on published vendor pricing, OAIC and DISR guidance, and standard SaaS observability cost benchmarks. Treat them as planning ranges, not quotes.

None of this makes AI agents uneconomic. Many are strongly value-positive when they replace high-cost manual processes. The point is that any business case which uses build cost as a proxy for total cost is wrong by a factor of three to five.

Part 4: What Good Production Operations Actually Looks Like

The good news is that the operating discipline for production AI is no longer mysterious. There are now mature reference frameworks that an Australian business can adopt without invention:

• NIST AI Risk Management Framework (AI RMF 1.0, January 2023) for risk identification and treatment.
• ISO/IEC 42001:2023 for an AI management system, the first international standard for AI governance.
• OWASP Top 10 for LLM Applications (2025) for security threat modelling.
• Australia's AI Ethics Principles (Department of Industry, Science and Resources) as the local-policy anchor, particularly relevant for any business that touches government procurement.
• The Voluntary AI Safety Standard (DISR, September 2024) which the federal government has published as a stepping-stone to mandatory guardrails for high-risk AI.

Underneath those frameworks, the operating loop looks like this:

A few elements deserve specific attention.

Pre-deployment evaluation harness. A golden set of 200 to 1,000 representative inputs with expected behaviours. Every change to the prompt, the retrieval data, the model version, or the toolset triggers a rerun. A change that drops scores below threshold blocks the merge. This is the single most under-invested practice in the Australian AI builds we encounter.

Continuous regression testing. The same golden set runs on a schedule against the current production agent, not just at change events. Silent drift caused by an upstream vendor update gets caught in days, not in a customer complaint cycle.

Human-in-the-loop checkpoints with explicit criteria. Not "everything goes to a human" (that defeats the purpose) and not "nothing goes to a human" (that creates the Air Canada problem). The right answer is a written matrix: classes of action that always require human approval (anything legally binding, anything mutating production data, anything involving sensitive personal information) versus classes that the agent can complete autonomously.

Incident response runbook. Specifically including the path from detection to OAIC notification under the Notifiable Data Breaches scheme if personal information is implicated. The 30-day clock under section 26WK of the Privacy Act runs whether or not your team has rehearsed the playbook.

A maturity progression. No business goes from "no AI" to "governed production agent" in one step. The realistic path is staged.

Most Australian businesses we speak with are honestly between Stage 1 and Stage 2. Most vendor pitches assume Stage 4 capability, while bypassing the operating discipline that Stages 2 and 3 require.

Part 5: The Honest Decision

Given everything above, the question worth putting to the board is "should we own the operational burden of running one in production".

For most Australian businesses, the right answer for any given use case is not DIY at full operational ownership. It is either bounded SaaS where someone else absorbs the operational cost across many customers, or partner-led implementation where a specialist runs the operations stack while you retain the strategic and data assets.

DIY makes sense only when the agent is a genuine source of competitive differentiation, the business already has the engineering and on-call discipline to operate production systems of this complexity, and the projected value clears the honest TCO with margin to spare.

Part 6: Twelve Questions Before Any Agent Goes to Production

Whether you build, partner, or buy, these are the questions a CIO, COO, or business owner should be able to answer (or have an honest "we will know by date X" plan for) before signing off:

1. What specific tasks is this agent allowed to perform autonomously, and which require human approval? Is that boundary written down?
2. What is the regression test plan, including the golden-set size and pass threshold, and who maintains it?
3. When the underlying model is deprecated, who owns the migration, and what is the budgeted downtime and re-evaluation cost?
4. Who is on-call when the agent says or does something that creates legal, regulatory, or reputational exposure? What is the response time?
5. Where are prompts, full input-output traces, and tool calls stored? For how long? Under what legal hold and discovery conditions?
6. What is our data residency posture for every third-party model, retrieval service, and observability tool the agent touches?
7. Have we completed a Privacy Impact Assessment per OAIC guidance, and on what cadence is it refreshed?
8. What is our path from anomaly detection to NDB notification under section 26WK of the Privacy Act?
9. How are we testing for adversarial prompt injection on a continuous basis, not just at launch?
10. Do we have a published policy that prohibits employees pasting sensitive data into unapproved AI tools, and is it enforced via DLP tooling?
11. What is the rollback procedure if a deployed agent change causes regression?
12. What is the 24-month operational budget, and who in finance has signed off on it?

If any of these has no answer, the agent is not ready for production. That is true whether the build cost was AUD 50,000 or AUD 500,000.

A Note on Real-World Production Experience

Two examples of agentic systems we have built and operated, where the operations realities described above are not theoretical:

RootCauseAI is an on-premise investigation agent for IT support teams. Its on-premise architecture exists precisely because the data access patterns described in Section 2 made cloud-only deployment unacceptable for the target customers. Carbonly.ai, the founder's ESG automation product, runs multi-agent document extraction at production scale. Both products have lived through model deprecations, evaluation harness rebuilds, and the boring monthly discipline of regression tests. They are here because the operating realities in this article are the same realities those products have to absorb every quarter.

For the broader strategic and governance picture, see our AI Strategy services page, our Managed AI Services overview, and our Data Sovereignty in Australia guide. For an architecture deep-dive on how agents integrate with existing business systems, see AI agent ecosystem architecture. For an example of a production agent in a real Australian deployment, see AI agent IT support investigations. For governance specifics, see AI agent governance, data access, privacy, and human override.

Before You Sign Off

Most AI agent failures do not happen at launch. They happen in months three to twenty-four, when the operating burden becomes visible and was never budgeted, never staffed, and never exercised. A serious pre-build review surfaces these issues before any contract is signed and before any vendor is selected.

If your business is approaching a DIY agent decision, or has one in flight and the operational picture above is making you uneasy, a structured second opinion is cheap insurance. We offer a free 30-minute pre-build review for Australian businesses considering production AI agents. It is a conversation, not a sales call. We bring the questions above, you bring the proposed scope, and we tell you honestly where the operational risks sit.

Book a 30-minute pre-build review

Related Reading:

• AI Agents in Australian Businesses: Why You Should Not DIY Without Understanding - Our foundational article on the build-time risks, paired with this operational deep-dive.
• AI Agent Governance, Data Access, Privacy, and Human Override - The governance frameworks behind the operations loop in this article.
• AI Agent Ecosystem Architecture and Business Integration - How tool permissions, blast radius, and integration boundaries shape agent risk.
• Wrong AI Tools Leak Business Data to Training LLMs - Deeper coverage of the employee-driven data leakage failure mode.

Sources and References: Moffatt v Air Canada, BC Civil Resolution Tribunal (2024). Samsung ChatGPT data leak, reported by Bloomberg, The Economist Korea, April 2023. Replit production database deletion incident, reported by Fortune and Tom's Hardware, July 2025. DPD chatbot incident, reported by The Guardian and BBC, January 2024. Chevrolet of Watsonville chatbot incident, December 2023. iTutorGroup US EEOC settlement, August 2023. NYC MyCity chatbot, The Markup investigation, 2024. Chen, Zaharia, Zou, "How Is ChatGPT's Behavior Changing Over Time?" (2023). NIST AI Risk Management Framework 1.0 (January 2023). ISO/IEC 42001:2023. OWASP Top 10 for LLM Applications (2025). Australia's AI Ethics Principles, DISR. Voluntary AI Safety Standard, DISR (September 2024). OAIC AI and Privacy Act guidance (October 2024). OAIC Notifiable Data Breaches Reports, July to December 2024 and January to June 2025. Australian Human Rights Commission, "Using artificial intelligence to make decisions" (2020, updated 2023). Privacy Act 1988 (Cth), section 26WK. ACCC guidance on AI and consumer protection (2024).