10 Questions to Ask Before Choosing an AI Vendor
Vendor Selection Is the Second-Most Consequential Decision in Your AI Program
The first was deciding not to build it yourself. We covered that in Why Midsize Australian Businesses Should Not DIY AI Agents, with the operational reality detailed in Operating AI Agents in Production, and the workforce gap in The AI Agent Staffing Gap. If you have read those, you already know that running agentic AI in production is harder, costlier, and more regulated than vendor decks imply.
This article answers the next question. Once you have decided to buy rather than build, how do you actually evaluate AI vendors so you do not end up locked into a 36-month contract that destroys value?
Gartner's 2025 forecast estimates that over 40 percent of agentic AI projects will be cancelled by the end of 2027, with escalating costs, unclear business value, and inadequate risk controls cited as the dominant drivers (Gartner, "Predicts 2025: AI Agents Will Reshape Enterprise Software", June 2025). Most of that value destruction is locked in at contract signature, before a single token is consumed. The vendor selection process is where you avoid becoming part of that statistic.
This is a procurement and governance piece written for CIOs, COOs, CFOs, and Heads of Procurement at Australian midsize businesses (50 to 500 employees). The aim is to give your team a vendor evaluation framework that holds up under legal, security, and finance review, not a list of demo questions.
The Three-Layer Evaluation Framework
Every AI vendor decision sits on top of four layers. Most procurement processes only test the first.
AI Vendor Evaluation Framework
Technical Fit is where most vendor demos shine. The differentiation, and the risk, lives in the bottom three layers. Australian midsize buyers in particular tend to under-weight the AU-Specific layer because most vendors are global platforms with thin local adaptation.
The 10 Questions
Question 1: Where will our data be processed and stored, under what jurisdiction, and what is your sub-processor list?
This is a legal question first, a technical one second.
Why it matters: Australian Privacy Principle 8 (cross-border disclosure of personal information) makes the disclosing entity responsible for any breach by an overseas recipient unless specific exemptions apply (OAIC, "Chapter 8: APP 8 Cross-border disclosure of personal information", 2024). When your AI vendor sends a customer record to an LLM hosted in the United States, your business carries the residual privacy liability, not the vendor.
What good looks like: A named primary processing region (for example, AWS Sydney ap-southeast-2 or Azure Australia East), a documented sub-processor list including the underlying foundation-model provider, contractual data-processing terms aligned with the Privacy Act 1988, and a written commitment that customer data is not used to train any vendor or third-party model.
Red flag answers: "We use industry-standard cloud infrastructure." "Data is processed in multiple regions for resilience." "We do not currently publish a sub-processor list." Any of these mean the vendor is either unwilling or unable to give your privacy team what they need.
Question 2: What is the genuine total cost of ownership over 36 months?
License price is the smallest line in the TCO stack for any non-trivial agentic deployment.
Why it matters: Foundation-model token usage scales with adoption. Integration costs front-load. Professional services hide in change requests. Eval and prompt maintenance is a continuous expense that vendors rarely surface during procurement. Gartner's 2025 cost guidance for enterprise GenAI deployments estimates that licence and tokens typically account for 30 to 45 percent of three-year TCO, with the balance in integration, professional services, and ongoing operations (Gartner, "Forecast Analysis: Generative AI Cost Optimization", 2025).
What good looks like: A line-itemised 36-month forecast covering platform licence, token consumption at modelled volumes, professional services for implementation, integration build and maintenance, ongoing prompt and eval maintenance, support tiers, and exit-related costs. Currency stated explicitly (AUD or USD), GST treatment confirmed.
Red flag answers: "We can provide pricing once we understand the use case better." "Token usage depends on your volumes, we cannot estimate." Honest vendors will model scenarios. Evasive vendors will not.
Question 3: What is your model deprecation and version-change policy?
Foundation models deprecate. Frequently.
Why it matters: OpenAI publishes a public deprecation schedule for its models (OpenAI, "Deprecations" page, platform.openai.com/docs/deprecations). Anthropic and Google have similar lifecycle patterns. When the model your agent was tested and certified against is sunset, your prompts, evals, and guardrails may all need re-validation. If your vendor passes that risk through without notice or rebate, your operations team carries the cost.
What good looks like: A documented model-change notification window (90 days or more), a regression-testing commitment by the vendor against your existing eval suite, and a contractual right to rebate or service credit if the migration breaks production behaviour.
Red flag answers: "We always run the latest model so you do not need to worry about this." That is the answer that should worry you most.
Question 4: How do we exit the contract and extract our data, prompts, fine-tunes, embeddings, and eval suite?
Exit is where vendor lock-in is engineered.
Why it matters: The investment that compounds inside an AI platform is not the licence, it is the prompts, the eval suite, the fine-tuned model weights, the embedding indexes, and the integration code. If you cannot extract those in a portable form, you cannot leave. Forrester's 2024 enterprise AI research identifies vendor lock-in as the single most-cited concern among AI buyers (Forrester, "The State Of AI In Australia", 2024).
What good looks like: Contractual exit terms that include export of prompts, evals, embeddings, fine-tune training data, and any custom configuration in documented portable formats. A defined transition period with continued read access. Documented evidence of data destruction within 30 days of termination.
Red flag answers: "Your customisations are platform-specific and non-transferable." "Embeddings are stored in our proprietary vector format." "We do not provide eval-suite export."
Question 5: What is your incident response posture under the Notifiable Data Breach scheme?
The Notifiable Data Breach (NDB) scheme is not optional.
Why it matters: Under Part IIIC of the Privacy Act 1988, entities covered by the Act must notify the OAIC and affected individuals of an eligible data breach as soon as practicable (OAIC, "Notifiable Data Breaches scheme", oaic.gov.au). The Privacy and Other Legislation Amendment Act 2024 has materially increased penalties for serious or repeated interference with privacy. If your vendor experiences a breach involving your data, the question of who notifies, on what timeline, and bearing what costs needs to be settled in the contract, not after.
What good looks like: A 24 to 72 hour breach-notification clause from vendor to customer, with the vendor providing the technical and forensic detail the customer needs to meet its own NDB obligations. Named incident-response contacts in Australia or APAC.
Red flag answers: "We follow global incident response best practice." "Notification timelines depend on the nature of the incident." Both are non-answers.
Question 6: What is your SLA, your support hours in AEST, and what counts as "available"?
Most global vendor SLAs were not written with AEST in mind.
Why it matters: A "24x5 business-hours" SLA usually means US or European business hours. A critical production agent breaking at 10am Sydney time on a Tuesday can sit in a queue until US morning if the vendor has no APAC follow-the-sun coverage. For a midsize business with a customer-facing agent, that is a revenue-impacting window.
What good looks like: Explicit AEST/AEDT coverage hours, a named regional support team (Sydney, Melbourne, Singapore, or equivalent), response-time SLAs broken down by severity, and a clear definition of availability that excludes only documented maintenance windows.
Red flag answers: "Our global team provides 24-hour coverage." Press for the AEST-specific response-time data from the past 12 months.
Question 7: How do you handle prompt injection and OWASP LLM Top 10 risks?
This is the security question your CISO will care about most.
Why it matters: The OWASP Top 10 for Large Language Model Applications (current version: OWASP, "OWASP Top 10 for LLM Applications", genai.owasp.org) enumerates the dominant attack surfaces for agentic systems: prompt injection, insecure output handling, training data poisoning, model denial of service, supply-chain vulnerabilities, sensitive information disclosure, insecure plugin design, excessive agency, overreliance, and model theft. NIST's AI Risk Management Framework (NIST AI RMF 1.0, January 2023, and the Generative AI Profile released July 2024) gives the assurance vocabulary that mature vendors should speak.
What good looks like: A documented security architecture mapped to OWASP LLM Top 10 and NIST AI RMF, third-party penetration-testing reports on the agentic layer (not just the underlying cloud), input and output filtering, tool-use sandboxing, and rate-limiting controls.
Red flag answers: "Our models are trained with safety in mind." "We use guardrails." These are marketing words, not controls.
Question 8: Who has access to our inputs and outputs? Are they used to train your or any third party's models?
The training-data question has to be answered in writing.
Why it matters: Several enterprise incidents in 2023 and 2024 stem from staff pasting sensitive information into consumer-grade AI tools where it was used for model training. We covered this risk in detail in How the Wrong AI Tools Leak Your Business Data into Training LLMs. The vendor's standard terms must explicitly carve out customer inputs and outputs from training use, by them and by any downstream model provider they pass data to.
What good looks like: A clear "no-training" clause covering inputs, outputs, fine-tune data, and embeddings. Documented sub-processor agreements that flow the same restriction through to underlying foundation-model providers. SOC 2 Type II or ISO 27001 attestation covering the relevant scope.
Red flag answers: "We may use de-identified or aggregated data to improve our services." Define de-identified. Define aggregated. Get it in the contract.
Question 9: What is your evidence base? Reference customers we can speak to, third-party security attestations, AU government panel listings.
Demos sell. Evidence proves.
Why it matters: Australia's Digital Transformation Agency and AusTender panels (BuyICT, Cloud Marketplace, Data and Analytics Panel) act as proxy due-diligence for vendor maturity. SOC 2 Type II and ISO 27001 are baseline. IRAP assessment is the standard for vendors serving government and regulated sectors (Australian Cyber Security Centre, "Information Security Registered Assessors Program (IRAP)", cyber.gov.au).
What good looks like: Two or three Australian customer references of similar size and industry, available for unscripted conversation. Current SOC 2 Type II and/or ISO 27001 reports. IRAP assessment if you operate in a regulated industry. Listing on relevant DTA or state-government panels.
Red flag answers: "Our Australian customers prefer to remain confidential." "References available after contract signing." Both are signs of either thin local footprint or unhappy customers.
Question 10: What is the lawful basis on which your agent acts on behalf of our business?
This is the question almost no procurement process asks. It is the one that matters most.
Why it matters: In Moffatt v Air Canada (2024 BCCRT 149, British Columbia Civil Resolution Tribunal), Air Canada was held liable for refund commitments made by its chatbot, even though the bot had given incorrect information. The tribunal rejected Air Canada's argument that the chatbot was a "separate legal entity" responsible for its own outputs. That case is now widely cited as a precedent for AI-agent liability. The Australian Competition and Consumer Commission has signalled that misleading representations made via AI agents will be treated as misleading conduct under sections 18 and 29 of the Australian Consumer Law, with the deploying business liable (ACCC, "Targeting scams and digital deception", multiple public statements 2024 to 2025).
What good looks like: Contractual clarity that the vendor is the data processor and the customer is the data controller. Vendor warranties on output accuracy where the agent is making representations to consumers. Clear allocation of liability for outputs that breach the Australian Consumer Law. Insurance arrangements that cover AI-specific liability.
Red flag answers: "Our model is generally accurate." "Liability for outputs sits with the customer." Some of that is reasonable. All of it is not.
Vendor Red Flags vs What Good Looks Like
Vendor Answers: Red Flag vs Good Answer
| Metric | Vendor red flag | What good looks like | Improvement |
|---|---|---|---|
| Data location | We use industry-standard cloud infrastructure | Primary processing in AWS Sydney, sub-processor list provided, contractual data-residency commitment | APP 8 cross-border disclosure liability sits with you |
| TCO | Pricing depends on your use case, we will scope later | 36-month forecast covering tokens, integration, services, ongoing eval maintenance, exit | Most AI value destruction is contractual, not technical |
| Model changes | We always run the latest model so you do not need to worry | 90-day deprecation notice, regression testing, rebate if migration breaks production | Your eval suite is the asset, not the vendor's roadmap |
| Exit | Customisations are platform-specific and non-transferable | Prompts, evals, embeddings, fine-tune data exportable in portable formats, 30-day transition period | Lock-in is engineered at signature, not at termination |
| NDB scheme | We follow global incident response best practice | 24 to 72 hour breach notification with forensic detail, named AU/APAC contacts | Your OAIC clock starts the moment you become aware |
| AEST support | Our global team provides 24-hour coverage | Named APAC support team, 12 months of AEST response-time SLA data | Severity 1 at 10am Tuesday cannot wait for US morning |
| Security | Our models are trained with safety in mind | OWASP LLM Top 10 and NIST AI RMF mapping, third-party pen-test on agentic layer | Marketing words are not controls |
| Training data use | We may use de-identified data to improve our services | Explicit no-training clause covering inputs, outputs, embeddings, flowed to sub-processors | Your data leaks to the model permanently if this is wrong |
The 36-Month True TCO Stack for a Midsize Australian Deployment
The TCO question deserves its own breakdown. Below is a realistic three-year cost stack for a midsize Australian business deploying a moderate-complexity agent (for example, an internal IT support assistant or a customer-service triage agent) covering 200 to 500 staff. Ranges are conservative and drawn from Gartner cost-optimisation guidance, vendor public pricing pages (AWS Bedrock, Azure OpenAI, Anthropic), and Hays and Robert Half 2025 Salary Guides for Australian AI engineering rates.
36-Month True TCO Stack (Midsize AU Deployment)
The point is not that AI is expensive. It is that the vendor's licence price typically represents 25 to 35 percent of true TCO. Procurement decisions made on licence price alone systematically under-fund the rest of the program. A vendor that helps you model the full stack is worth more than one offering a cheaper headline rate.
The Australian-Specific Evaluation Layer
This is the layer global vendors most often miss, and where Australian midsize buyers carry the largest residual risk.
Privacy and personal information. Privacy Act 1988 (Cth), Australian Privacy Principles, APP 8 cross-border disclosure obligations. The 2024 amendments add a statutory tort for serious invasions of privacy and increase OAIC enforcement powers. Vendors processing personal information of Australian individuals need to be able to speak this vocabulary.
Australian Consumer Law. Sections 18 and 29 of the Competition and Consumer Act 2010 (Cth) cover misleading and deceptive conduct, and false representations about services. The ACCC's public statements through 2024 and 2025 confirm that AI-generated representations are within scope. Your contract needs to allocate the risk of inaccurate agent outputs.
Notifiable Data Breach scheme. Part IIIC of the Privacy Act 1988. Notification obligations, timelines, and definitions of "eligible data breach" need to be settled in the vendor contract, not figured out during the incident.
Data hosting region. AWS Sydney (ap-southeast-2), Azure Australia East and Australia Southeast, Google Cloud Sydney and Melbourne, are the relevant in-country regions. Confirm in writing which region your data and processing live in, and whether any inference, embedding generation, or logging crosses borders. See our Data Sovereignty in Australia Guide for the detail.
ATO record-keeping. Where the agent touches financial data or business records, the ATO's record-keeping requirements (generally 5 years for tax records, longer in some cases) apply. Vendor data-retention policies need to align.
Sector-specific regulation. APRA CPS 230 (Operational Risk Management) and CPS 234 (Information Security) for APRA-regulated entities. AHPRA and TGA frameworks for healthcare. ASIC guidance for financial services. The Department of Industry, Science and Resources Voluntary AI Safety Standard (September 2024) sets baseline expectations even outside regulated sectors.
Government panel listings. DTA Cloud Marketplace, BuyICT, state government Whole-of-Government panels, IRAP assessment status. Even if you are not selling to government, these listings act as proxy due-diligence.
For deeper coverage of the governance side, see AI Agent Governance: Data Access, Privacy, and Human Override.
Which Vendor Type Fits Your Use Case?
There is no universally right vendor for AI. The right vendor depends on the use case, your data constraints, your existing platform commitments, and your budget.
Which AI Vendor Type Fits?
The last option is the one most midsize buyers should run first. Skipping it is how organisations end up in the Gartner 40 percent.
A Realistic AU Midsize Vendor Evaluation Timeline
Vendor selection done well takes roughly 12 to 16 weeks from kick-off to signature. Compressing it below 8 weeks materially increases the risk of contracting in problems you will pay for over 36 months.
Typical AU Midsize Vendor Evaluation
Procurement Readiness Checklist (Run This Before the First Vendor Demo)
If your team cannot answer yes to all of these before scheduling vendor demos, the demos will not produce a sound decision. They will produce a sales experience.
- Have we defined the use case, success metrics, and exit criteria in writing?
- Has the executive sponsor signed off on a maximum 36-month TCO envelope?
- Has Legal reviewed our standard MSA template for AU jurisdiction and Australian Consumer Law compatibility?
- Has Privacy completed (or scoped) a Privacy Impact Assessment for the data the agent will touch?
- Has Security threat-modelled the use case against the OWASP LLM Top 10?
- Have we documented which existing systems the agent must integrate with, including authoritative data owners?
- Have we identified the regulatory regimes that apply (Privacy Act, APRA, AHPRA, TGA, ASIC, sector-specific)?
- Have we defined what "exit" means in this contract, and which assets we must be able to extract?
- Have we identified the named accountable executive for the agent's outputs in production?
- Have we agreed how we will run the pilot, what success looks like, and what triggers a no-go to production?
For more on the governance and operational discipline behind this list, see Operating AI Agents in Production and our AI Strategy service.
A Note on E-E-A-T and the Evidence Behind This Article
Solve8 is a vendor-neutral AI consultancy. We have no vendor partnerships or referral arrangements, which is why this article does not name any vendor as recommended. The framework draws on enterprise-platform evaluation patterns developed across BHP, Rio Tinto, and Senex Energy data and integration programs, and on Solve8's own product builds, including the multi-agent architecture in Carbonly.ai and the on-premise investigation tool RootCauseAI. Both products gave us a builder's view of what enterprise customers actually need to see in a vendor contract.
For ongoing operations once the vendor is selected, see our Managed AI Services page.
Before Your Next Vendor Demo
Most AI vendor regret is locked in before pilot. The 10 questions above are designed to surface that risk while you can still walk away.
If your team is preparing to enter a vendor evaluation, we run a 30-minute pre-procurement review covering the four-layer framework, the AU-specific layer, and red-flag screening against your shortlist. It is vendor-neutral. No referral fees. No platform we are trying to sell you.
Book a 30-minute pre-procurement review
Related Reading:
- Why Midsize Australian Businesses Should Not DIY AI Agents - The foundational build-vs-buy decision this article follows from
- Operating AI Agents in Production - The operational reality and TCO context for vendor decisions
- The AI Agent Staffing Gap in Australian Midsize Business - Why the in-house team you would need usually does not exist
- AI Agent Governance: Data Access, Privacy, and Human Override - The governance posture you need before any vendor signs
Sources: Research synthesised from OAIC guidance on APP 8 and the Notifiable Data Breach scheme (2024), the Privacy and Other Legislation Amendment Act 2024, OWASP Top 10 for LLM Applications, NIST AI Risk Management Framework 1.0 and Generative AI Profile (2024), Gartner forecasts on agentic AI and GenAI cost optimisation (2025), Forrester State of AI in Australia (2024), Moffatt v Air Canada 2024 BCCRT 149, ACCC public statements on AI and the Australian Consumer Law (2024 to 2025), Department of Industry, Science and Resources Voluntary AI Safety Standard (September 2024), and the Australian Cyber Security Centre IRAP framework.