To choose an AI model for an Australian organisation, build a shortlist that passes five gates (task fit, data class, residency, latency and cost), then test the shortlisted models against a set of examples drawn from your own work. Off-the-shelf frontier models now perform similarly on many tasks, so fit to your data, hosting rules and budget decides the winner.
Key takeaways
- "Which model should we use?" is the wrong question. Ask which shortlist passes your gates and wins your evaluation this quarter.
- Data class and residency narrow the shortlist before capability comparisons start.
- "Hosted in Australia" can mean several different things. Check inference, logging, caching and telemetry separately.
- Most business AI traffic is routine and runs well on mid-tier models. Save frontier models for hard reasoning.
- Your evaluation test set is the durable asset. Models, prices and hosting options change every few months.
Why is "pick a model" the wrong question?
The most common request from CIOs and data leads is "just tell us which model to use". Asked that way, the discussion becomes a brand argument about GPT, Claude or Gemini, and skips what decides whether a project ships, performs and survives an audit.
The better question: which shortlist of models passes our task fit, data class, residency, latency and cost gates, and which one wins our evaluation this quarter?
The Stanford HAI 2025 AI Index Report found that performance gaps between leading models narrowed substantially across major benchmarks. For many business workloads, capability is no longer the differentiator. Fit to your use case, data, systems and risk posture is.
For the surrounding constraints, see why you should not DIY AI agents, the AI agent staffing gap and operating AI in production.
What five dimensions decide AI model selection?
If you cannot answer each of these for a use case, you are not ready to pick a model.
- Task fitReasoning, coding, language, multimodal, structured output
- Data classPersonal, sensitive, regulated, IP, public
- ResidencyAustralia only, offshore acceptable, or hybrid
- Latency and throughputInteractive, batch, or agent loop
- Cost profileUsage cost, hosting cost, total cost of ownership
Task fit has several axes. A model strong at agent tool use can be weaker at long-document summarisation. Public benchmarks such as MMLU-Pro, GPQA, SWE-bench and ARC-AGI measure different things. Use the one closest to your task, then verify on your own data.
Data class drives almost everything else. Public marketing copy can go to any frontier API. Personal information is governed by the Privacy Act and the Australian Privacy Principles, especially APP 8 when it leaves Australia. Regulated data (APRA-supervised entities, health records, government work) adds hosting, audit and breach-notification requirements that shrink the shortlist.
Residency has more than one answer. Where a model was trained and where it is served are separate questions. An Australian endpoint that calls an offshore subprocessor for any step is, under APP 8, a cross-border disclosure.
Latency and throughput depend on the workload. A customer-facing chat needs fast first-token times. A nightly invoice batch can wait. An agent loop with ten tool calls multiplies single-call latency by ten.
Cost means total cost of ownership. Open-weight models have no per-token fee but real operating cost. Frontier APIs have visible usage cost and little operating cost. A routing setup adds development and monitoring work.
What AI model families should you compare?
Five families cover the practical market. We avoid naming a single best in each because the leader changes every few months. Your shortlist usually takes one candidate from each relevant row.
| Family | Profile | Impact | |
|---|---|---|---|
| Frontier closed | OpenAI's flagship GPT models, Anthropic Claude Opus and Sonnet, Google Gemini Pro | Top-end reasoning and coding. Australian inference options through Azure Australia East, AWS Bedrock Sydney and Vertex AI Sydney, model by model. Premium usage pricing. | Capability ceiling |
| Mid-tier closed | Smaller GPT models, Claude Haiku class, Gemini Flash class | Strong general capability at a fraction of frontier cost. Often the right default for production traffic. | Cost and capability balance |
| Open-weight large | Llama family, Mistral Large, DeepSeek | Close to frontier on many tasks, with weights you can run yourself or through managed inference. Licence terms vary. | Control and IP |
| Open-weight small | Small Llama, Mistral, Phi and Gemma variants | Fits on a single modest GPU. Good for fine-tuning, edge use, or as a router in front of a larger model. | Specialisation and edge |
| Specialist | Embeddings, speech recognition, vision encoders, retrieval-tuned models | Works alongside a general model. Often the unsung half of a retrieval or multimodal pipeline. | Pipeline component |
We do not quote per-token prices because they change too often. Use the vendors' official pricing pages and rebuild your cost model at each re-evaluation.
The OpenAI model deprecation history shows frontier models have lifecycles measured in quarters. Anthropic, Google and Mistral follow similar patterns. Assume whatever you pick will be deprecated, repriced or superseded within a year or two.
What does "hosted in Australia" actually mean for AI models?
"Available in the cloud", "served from an Australian region", "no offshore data flow anywhere in the pipeline" and "IRAP-assessed" are four different things.
| Hosting path | What it does and does not give you | Impact | |
|---|---|---|---|
| AWS Bedrock Sydney | Selected models in-region; others through cross-region inference profiles | Check which profile you use: Australia-only, Asia Pacific or global. Confirm whether features such as Guardrails, Knowledge Bases or Agents call out of region. | Profile matters |
| Azure OpenAI, Australia East | OpenAI-family deployments in Microsoft's Australian region | Standard (regional) deployments process in region. Global and Data Zone deployments do not guarantee Australian processing. Check the regional availability matrix. | Deployment type matters |
| Google Vertex AI Sydney | Gemini and some partner models at the Sydney endpoint | Google notes that endpoint location alone does not guarantee in-region processing. Check the data residency page per model. | Verify per model |
| Direct vendor API | Calls to the vendor's general endpoints | Usually processed outside Australia, so APP 8 applies. Fine for prototypes, generally unsuitable for production personal information. | Convenience only |
| Self-hosted open weights in Australia | Your own cloud account in an Australian region or your own servers | Full residency control and full operating responsibility: GPUs, patching, monitoring and model lifecycle. | Full control |
IRAP. The Australian Signals Directorate's IRAP program assesses cloud services against the Information Security Manual. IRAP coverage differs across individual cloud AI services, so verify per service. This matters most for government and financial services work.
APP 8. OAIC guidance on cross-border disclosure means you generally stay accountable for how overseas recipients handle personal information. Australian inference with evaluation logs flowing to an offshore monitoring tool still triggers APP 8. See our data sovereignty guide.
Should you use open-weight models or closed APIs?
A common pattern: an engineer prototypes on a frontier API, then argues for self-hosted open weights to save cost. Sometimes right, often wrong, because the comparison counts usage cost and ignores operating cost.
| Closed frontier API | Self-hosted open weights | |
|---|---|---|
| Unit cost | Visible, metered per call | No per-call fee; cost is GPU hours plus engineering time |
| Capability ceiling | Highest available at any moment | Strong, usually behind the top frontier models on the hardest tasks |
| Operating burden | Vendor runs inference, scaling and updates | You run the inference server (vLLM, TGI, Ollama or similar), monitoring and lifecycle |
| Control over data and IP | Governed by enterprise terms | Complete; weights and data stay inside your boundary |
| Lock-in | Migration is a project; behaviour drifts between versions | Weights are portable between open models |
| Team needed | Software engineering, prompting and evaluation discipline | All of that plus model serving, GPU operations and observability |
Open weights make sense at high, sustained volumes, where data class makes self-hosting mandatory, or where the team already runs GPU workloads. For most mid-sized organisations, the staffing needed to run inference in production is the binding constraint. A hybrid, frontier API for hard tasks and open-weight models for routine traffic, becomes attractive as volume grows. Our managed AI services can run that split for you.
When do smaller, cheaper models win?
Most business AI traffic is routine: classification, extraction, structured output, short summaries and routing. For those tasks, mid-tier closed models and small open-weight models often match frontier quality at a fraction of the cost and latency.
Frontier models earn their premium on reasoning-heavy work: multi-step analysis, complex coding, agent loops with tools and long-context synthesis. These are usually a minority of traffic but a large share of value.
A model router (or cascade) sends each request to the smallest model that can handle it, escalating hard requests to a frontier model. Cloud providers and frameworks such as LangChain support this pattern. Measure the savings on your own traffic, because they depend on your task mix.
How do you evaluate AI models before choosing one?
Picking a model from a demo is the most damaging mistake. Demos show strong cases. Production traffic differs. Evaluate candidates against representative examples from your own workload, the kind of measurement discipline the NIST AI Risk Management Framework calls for.
- Step 1Define the use caseTask, inputs, expected outputs, success criteria, latency budget and data class on one page.
- Step 2Build a test set100 to 500 representative examples from real or realistic data, including edge cases, with expected outputs annotated.
- Step 3Run candidatesThree to five shortlisted models. Capture accuracy, latency, cost per call and failure modes.
- Step 4ScoreA simple scorecard weighted to your priorities. The right balance wins.
- Step 5Run alongside productionSend a copy of live traffic to the top two candidates and compare, with no user impact.
- Step 6Choose and documentRecord the decision, results and rationale as your audit trail.
- Step 7Re-evaluate quarterlyRe-run the suite every quarter and on every model version change.
The test set is the artefact you keep. Models, prompts and hosting will change. Our AI vendor selection questions sit upstream, and the evaluation makes the selection real.
Which AI model family fits your use case?
Use this to set the shortlist, then evaluate.
What is the binding constraint for this workload?
- Data must not leave Australia or pass through any offshore subprocessorOpen-weight models self-hosted in Australia (large for hard tasks, small for routine).
- Regulated data with strict residency, offshore subprocessors acceptable under contractFrontier closed models through an Australian region deployment. Verify IRAP status for government work.
- Standard business data, hard reasoning or coding, low to moderate volumeFrontier closed models through an Australian region deployment.
- Standard business data, routine tasks, high volumeMid-tier closed models through an Australian region deployment. Often the best default.
- Mixed task profile, want lower cost without losing quality on hard casesRouter: mid-tier for routine traffic, frontier for escalation. Needs evaluation discipline.
- Prototype or internal-only, low data sensitivityUse whatever gets you to working code fastest. Revisit before production.
Three implications. An Australian region deployment is the right default for most production traffic. The router design and evaluation suite usually matter more than frontier versus open weights. And the security posture of the hosting path matters as much as the model.
What questions should every AI model selection answer?
- What is the task family, and which public benchmark comes closest?
- What is the data class of every field passing through the model, including logged context?
- What is the residency requirement at the inference, logging, caching and telemetry layers?
- What is the latency budget per call and per agent loop?
- What volume is expected at launch and at 12 months?
- What is total cost of ownership, including integration, monitoring and re-evaluation?
- What is the vendor's deprecation history for this model family?
- Who operates it in production when a model is deprecated or quality regresses?
- Who owns the evaluation suite, and how often is it re-run?
- What is the exit path if the model, platform or vendor becomes unsuitable?
What does a worked model selection look like?
For example, a hypothetical 220-person professional services firm builds an internal document review assistant. Inputs are client matters: sensitive, not APRA- or health-regulated. Output is a structured summary with highlighted risk clauses. Interactive latency is required, with volume growing over the first year.
- Task fit: long-context summarisation, structured output, clause-level reasoning. Most frontier and near-frontier models qualify.
- Data class: confidential client data under the Privacy Act and professional obligations.
- Residency: Australian inference to meet client expectations and APP 8 risk.
- Latency: interactive, which rules out the slowest reasoning models for the live path.
- Cost: mid-tier pricing is comfortable at projected volume; frontier is not.
The shortlist becomes mid-tier and upper-mid-tier models served from an Australian region. The test set is 200 redacted historical matters with expert-annotated summaries. Three candidates run through it, and the winner meets the accuracy bar inside the latency budget at the lowest cost per review. Model names will change. The method does not.
How Solve8 can help
Our consultants and engineers run model selection as part of our AI strategy and private AI infrastructure services: shortlisting, building your evaluation set and deploying on your infrastructure in Australia. We built RootCauseAI, our own on-premises investigation product, with the same conservative posture. Talk to us before you commit to a model.
Sources: Stanford HAI AI Index Report 2025; NIST AI Risk Management Framework; OAIC APP 8 guidance; ASD Information Security Manual and IRAP program; Microsoft, AWS and Google regional availability documentation; OpenAI deprecations page.
