How to Select an AI Vendor: 10 Questions to Ask

To select an AI vendor, test four layers: technical fit, commercial fit (true three-year cost and exit terms), operational fit (Australian-hours support and incident response) and the Australian layer (Privacy Act, APP 8, Notifiable Data Breaches, Australian Consumer Law). The ten questions below form AI vendor evaluation criteria that hold up under legal, security and finance review.

Why vendor selection matters so much

Gartner predicted in June 2025 that over 40 per cent of agentic AI projects will be cancelled by the end of 2027, citing escalating costs, unclear business value and inadequate risk controls. Much of that is decided at procurement.

This guide is for CIOs, COOs, CFOs and procurement leads at mid-sized and enterprise Australian organisations who have decided to buy rather than build. If you are still weighing that, read why mid-sized businesses should not DIY AI agents and the AI staffing gap.

What are the right AI vendor evaluation criteria?

1. 1Technical FitModel quality, integrations, data handling, scalability
2. 2Commercial FitTrue 36-month cost, contract terms, exit clauses
3. 3Operational FitAustralian-hours support, SLAs, incident response
4. 4Australian LayerPrivacy Act, ACL, Notifiable Data Breaches, ATO records, sector regulators

Demos shine on technical fit. The risk lives in the other three layers, and Australian buyers tend to under-weight the local layer because most vendors are global platforms with thin local adaptation.

How to select an AI vendor: the 10 questions

1. Where will our data be processed and stored, and who are your sub-processors?

Why it matters: Under Australian Privacy Principle 8, an organisation that discloses personal information to an overseas recipient generally remains accountable for that recipient's handling of it (OAIC APP Guidelines, Chapter 8).

Good answer: A named processing region (for example AWS Sydney or Azure Australia East), a published sub-processor list including the foundation-model provider, Privacy Act-aligned data processing terms, and a written commitment that your data is not used to train any model.

Red flags: "Industry-standard cloud infrastructure." "Multiple regions for resilience." "We don't publish a sub-processor list."

2. What is the genuine total cost of ownership over 36 months?

Why it matters: Token usage scales with adoption, integration front-loads, services hide in change requests, and prompt and evaluation maintenance never stops.

Good answer: A line-item forecast covering licence, tokens at modelled volumes, implementation services, integration build and upkeep, evaluation maintenance, support tiers and exit costs, with currency and GST stated.

Red flags: "We can price once we understand the use case better." Honest vendors model scenarios.

3. What is your model deprecation and version-change policy?

Why it matters: Model providers retire models on published schedules. When the model your agent was tested on is sunset, prompts, evaluations and guardrails may need revalidation.

Good answer: A notice window of 90 days or more, regression testing against your evaluation suite, and service credits if migration breaks production.

Red flags: "We always run the latest model so you don't need to worry."

4. How do we exit and extract our data, prompts, embeddings and evaluation suite?

Why it matters: The asset that compounds is your prompts, evaluations, fine-tuned weights, embeddings and integration code. If you cannot export them, you cannot leave.

Good answer: Contractual export in documented portable formats, a transition period with read access, and evidence of data destruction after termination.

Red flags: "Customisations are platform-specific." "Embeddings are in our proprietary format."

5. What is your incident response under the Notifiable Data Breaches scheme?

Why it matters: Under Part IIIC of the Privacy Act 1988, covered entities must notify the OAIC and affected individuals of an eligible data breach. The Privacy and Other Legislation Amendment Act 2024 strengthened enforcement. Who notifies, when and at whose cost must be in the contract.

Good answer: A short, defined vendor-to-customer notification window with the forensic detail you need, and named contacts in Australia or APAC.

Red flags: "We follow global best practice." "Timelines depend on the incident."

6. What are your SLAs and support hours in Australian time?

Why it matters: "Business hours" often means US or European hours. A production agent failing at 10am Sydney time can sit in a queue until US morning.

Good answer: Explicit AEST/AEDT coverage, a regional support team, severity-based response times and a clear availability definition.

Red flags: "Our global team covers 24 hours." Ask for the last 12 months of Australian-hours response data.

7. How do you handle prompt injection and the OWASP LLM Top 10?

Why it matters: The OWASP Top 10 for LLM Applications covers risks such as prompt injection, sensitive information disclosure, supply chain weaknesses, improper output handling and excessive agency. NIST's AI Risk Management Framework (2023) and its Generative AI Profile (2024) give mature vendors a shared assurance vocabulary.

Good answer: A security architecture mapped to OWASP and NIST, third-party penetration testing of the agent layer, input and output filtering, tool sandboxing and rate limits.

Red flags: "Our models are trained with safety in mind." "We use guardrails."

8. Are our inputs and outputs used to train your or anyone else's models?

Why it matters: Staff pasting sensitive information into consumer AI tools has caused real incidents. See how the wrong AI tools leak business data.

Good answer: A no-training clause covering inputs, outputs, fine-tune data and embeddings, flowed through to sub-processors, plus SOC 2 Type II or ISO 27001 attestation for the relevant scope.

Red flags: "We may use de-identified or aggregated data to improve our services." Define both terms in the contract.

9. What evidence can you show?

Why it matters: Demos sell; evidence proves. SOC 2 Type II and ISO 27001 are baseline. IRAP assessment matters for vendors serving government and regulated sectors.

Good answer: Two or three Australian references of similar size and industry available for unscripted calls, current attestation reports, and IRAP where relevant.

Red flags: "References after signing." "Our Australian customers prefer to stay confidential" with no alternative evidence offered.

10. Who is liable when your agent says something wrong to our customers?

Why it matters: In Moffatt v Air Canada (2024 BCCRT 149), Air Canada was held liable for incorrect refund information its chatbot gave, and the tribunal rejected the argument that the chatbot was responsible for its own statements. In Australia, the Australian Consumer Law prohibitions on misleading or deceptive conduct apply to representations made through AI tools as they do to staff.

Good answer: Clear processor and controller roles, output accuracy warranties where the agent makes representations to consumers, liability allocation for ACL breaches, and insurance that covers AI-specific risk.

Red flags: "Our model is generally accurate." "All output liability sits with you."

Red flag answers versus good answers

	Vendor red flag	What good looks like	Impact
Data location	We use industry-standard cloud infrastructure	Named Australian region, sub-processor list, contractual residency commitment	APP 8 accountability sits with you
Cost	Pricing depends on your use case, we'll scope later	36-month forecast covering tokens, integration, services, evaluation upkeep and exit	Value is lost in the contract
Model changes	We always run the latest model	90-day notice, regression testing, credits if production breaks	Your evaluation suite is the asset
Exit	Customisations are non-transferable	Prompts, evaluations, embeddings and fine-tune data exportable	Lock-in is set at signature
Breach response	We follow global best practice	Defined notification window with forensic detail, named regional contacts	Your OAIC obligations start when you become aware
Support	Our global team covers 24 hours	Named APAC team and Australian-hours response data	Severity 1 can't wait for US morning
Security	Trained with safety in mind	OWASP and NIST mapping, third-party testing of the agent layer	Marketing words are not controls
Training use	We may use de-identified data	No-training clause flowed to sub-processors	Leakage into a model is permanent

Red flags when selecting AI consultants in Australia

The same discipline applies when you hire an AI consultancy instead of buying a product. Watch for:

• No discovery before a quote. Firm numbers before anyone has seen your data and systems.
• Model talk, no integration evidence. They cannot show an integration shipped into a system you run.
• Undefined success. No baseline, no measure, no stop point.
• Vendor incentives undisclosed. Ask directly whether they earn referral fees or resale margin on what they recommend.
• Data handling vague. They cannot say where your data will be processed or by which sub-processors.
• No handover plan. The engagement is designed for permanent dependence.

Our guide to choosing an AI consultancy in Australia goes further.

AI prompts for vendor evaluation

AI tools can speed up evaluation if you keep vendor documents inside an approved, enterprise-grade tool. Useful prompts:

• "From this vendor's data processing addendum, list every sub-processor, the country of processing and whether customer content is used for training. Quote the clause for each."
• "Compare these three RFP responses against our ten questions. For each question, mark answered, partially answered or not answered, and quote the supporting text."
• "Identify every clause in this MSA that limits our ability to export prompts, embeddings or evaluation data on exit."
• "List all commitments in this proposal that are not backed by a contractual clause."
• "Draft follow-up questions for any answer that uses the words 'industry standard', 'best practice' or 'generally'."

Always have a person verify quotes against the source document before they influence a decision.

What does a vendor really cost over three years?

For a moderate-complexity agent serving a 200 to 500-staff organisation, the cost lines usually look like this:

• Platform licence and foundation-model tokensScales with adoption
• Integration build (year 1)Front-loaded
• Professional services and change managementOften under-scoped
• Ongoing prompt, evaluation and guardrail maintenanceContinuous
• Security, audit and governance overheadRecurring
• Exit and migration reserveSet aside at signing

Decisions made on licence price alone systematically under-fund the rest. A vendor that helps you model the full stack is worth more than a cheaper headline rate.

The Australian evaluation layer

• Privacy Act 1988 and APPs, including APP 8 and the 2024 amendments, which added a statutory tort for serious invasions of privacy and strengthened OAIC powers.
• Australian Consumer Law: misleading or deceptive conduct and false representations, including those made by AI agents.
• Notifiable Data Breaches scheme under Part IIIC of the Privacy Act.
• Hosting region: AWS Sydney and Melbourne, Azure Australia East and Southeast, Google Cloud Sydney and Melbourne. Confirm where inference, embeddings and logs run. See our data sovereignty guide.
• ATO record-keeping: generally five years for tax records. Retention policies must align.
• Sector rules: APRA CPS 230 and CPS 234, AHPRA and TGA for health, ASIC for financial services, and the Voluntary AI Safety Standard (2024) as a general baseline.

See also AI agent governance, data access and human override.

Which vendor type fits your use case?

What does your use case look like?

• Bounded internal productivity, already on Microsoft 365 or Google WorkspacePlatform-native tools such as Microsoft 365 Copilot or Gemini for Workspace
• Customer-facing or revenue-impacting agent, regulated industry, data must stay in AustraliaAustralian implementation and managed service partner building in your own environment
• Deeply domain-specific workflow with an established specialist vendorSpecialist SaaS with documented Australian residency and sub-processor flow-down
• Strategic differentiation on proprietary dataCustom build on your own cloud with an implementation partner
• Unsure of use case maturityBounded proof of value before any 36-month commitment

A realistic vendor evaluation timeline

1. Weeks 1 to 2Use-case scoping and RFIDefine use case, success and exit criteria. Scope the privacy impact assessment. RFI to 5 to 8 vendors.
2. Weeks 3 to 5RFP and shortlistRFP covering the 10 questions and the Australian layer. Shortlist 2 or 3.
3. Weeks 6 to 9Proof of value and referencesTime-boxed POV on real data. Reference calls. Review penetration tests, SOC 2, ISO 27001, IRAP.
4. Weeks 10 to 13Commercial and legalCost model agreed. Redlines on privacy, breach notification, ACL, exit, deprecation, sub-processors.
5. Weeks 14 to 16Pilot to productionBounded pilot with go/no-go criteria. Production only on pilot success.

Procurement readiness checklist

1. Use case, success metrics and exit criteria defined in writing?
2. Executive sponsor signed off a maximum 36-month cost envelope?
3. Legal reviewed the MSA for Australian jurisdiction and ACL?
4. Privacy impact assessment completed or scoped?
5. Use case threat-modelled against the OWASP LLM Top 10?
6. Systems to integrate and their data owners documented?
7. Applicable regulatory regimes identified?
8. "Exit" defined, with the assets you must extract?
9. Named executive accountable for production outputs?
10. Pilot design and no-go triggers agreed?

How Solve8 can help

Our consultants run vendor-neutral pre-procurement reviews: we apply this framework to your shortlist, read the data processing terms and help you design the proof of value. Where building in your own environment fits better, we do that too, on your infrastructure in Australia. See our AI strategy service or talk to our team.