AI Readiness Assessment for Australian Midsize
The pilot was unready before it started
Walk back through the post-mortems of failed AI programs at Australian businesses and the same finding keeps appearing. The pilot did not fail because the model was wrong, the vendor was incompetent, or the use case was silly. It failed because the organisation was unready before the pilot was approved. The data was the wrong shape, in the wrong place, with the wrong access controls. Ownership was unclear. The team that knew the data left two years ago. No executive sponsor was on the hook for the outcome. The kill criteria were never written down.
This is the picture painted by MIT Sloan's State of AI in Business 2025 research, which found that the overwhelming majority of enterprise generative AI initiatives produced no measurable profit-and-loss impact. It is reinforced by Gartner's data quality research, which estimates that poor data quality costs organisations an average of $12.9 million annually. And it is consistent with what every practitioner across our staffing-gap analysis and production-reality piece reports: the work that determines pilot success happens before the pilot.
This article gives you the diligence framework. It is the step that sits before our AI pilot project success factors playbook. If the pilot framework is the 90-day operational plan, this is the gate the pilot must pass through before anyone touches a model. AI readiness is mostly a data-and-org question, and the cost of skipping the diligence is paid in dollars, time, and board credibility.
Part 1: The six dimensions of AI readiness
Before any pilot is approved, six dimensions must each clear a defined threshold. Skip any one and the pilot is borrowing against a debt it will eventually have to pay, usually at a much higher rate.
The six AI readiness dimensions
The order matters. You cannot assess data quality on data you do not have access to. You cannot govern data that is not catalogued. You cannot staff capability you have not scoped. And you cannot assign executive sponsorship to a problem nobody on the leadership team owns. Each dimension feeds the next, and the readiness assessment must walk through them in this order to produce a credible decision.
Two of these dimensions warrant cross-references to prior articles in this series. Org capability is handled in depth in our staffing-gap analysis. Compliance posture overlaps heavily with the ACCC consumer guarantees article, the APRA and ASIC compliance piece for financial services, and the 50-point AI security checklist. The remaining four dimensions, particularly data quality and governance, are where this article does the most work.
Part 2: The DAMA and ISO 25012 lens on data quality
When practitioners argue about what "good data" means, they are usually arguing because they have not agreed on a shared definition. Two canonical references resolve this. DAMA-DMBOK 2 (the Data Management Body of Knowledge, published by the Data Management Association) defines data quality through a set of measurable dimensions. ISO/IEC 25012:2008 (the international standard for data quality) overlaps strongly with DAMA and is the basis for the Australian-adopted AS ISO 8000 series on data quality.
Together they give you eight measurable dimensions. Each one has a definition, a measurement method, a target threshold, and a common failure pattern at Australian businesses. The table below is the practitioner's working set.
Data quality dimensions for AI pre-pilot assessment
| Metric | Definition and measurement | Common AU failure pattern |
|---|---|---|
| Accuracy | Data matches reality; measured by sampling against ground truth | Customer addresses with invalid postcodes; ABN fields holding free text |
| Completeness | Required fields populated; measured as % non-null on critical attributes | Time-series gaps where systems were offline; optional fields treated as mandatory by AI use case |
| Consistency | Same entity represented the same way across systems | One customer in Xero, MYOB, CRM, and ERP with four different IDs and three different name spellings |
| Uniqueness | No duplicate records; measured by entity-resolution match rates | Merged businesses doubled customer records; never reconciled |
| Timeliness | Data current enough for the decision being made | Inventory updated nightly but used for real-time pricing decisions |
| Validity | Conforms to format, range, and business rules | Date fields holding strings; postcodes outside Australia; negative prices |
| Accessibility | Data is reachable by the systems and people that need it | Data locked in a legacy system with no API; one person knows the export procedure |
| Integrity | Relationships and constraints preserved across the data set | Orphan invoice lines; foreign-key relationships broken during migrations |
Where this matters for AI pilots: a generative AI use case can tolerate weaker completeness if humans verify outputs, but it cannot tolerate weak consistency, because the model will learn or retrieve contradictory facts and produce confidently wrong answers. A classification model cannot tolerate weak validity because invalid inputs crash the pipeline. An agent workflow cannot tolerate weak accessibility because the agent will be unable to do its job without a workaround the operator has to maintain. The dimensions are not all equal, and the pre-pilot assessment must weight them against the specific use case under consideration.
Part 3: The data inventory and lineage exercise
A pre-pilot data assessment is a structured four-to-six-week discovery exercise that ends in a documented decision: proceed, remediate, or halt. No platform purchase substitutes for it. DAMA-DMBOK 2 is the canonical reference for the activities, and the EDM Council's Cloud Data Management Capabilities (CDMC) framework is the canonical reference for the cloud governance overlay that most Australian programs now require.
Four-to-six-week pre-pilot data assessment
The output of this exercise is three documents: a data inventory (what we have), a quality scorecard (how good it is), and a gap analysis (what is missing for the use case). These become the evidence that supports either the pilot business case or the decision to defer. A small team produces them, typically a data analyst, a business SME, and an architect walking the data with the people who own it. No tool does this part.
A common failure here is to treat the assessment as a documentation exercise instead of a measurement exercise. If your quality scorecard does not contain numbers, you have not measured anything. Sampling 100 records and counting how many violate each dimension is more useful than a 20-page narrative about how the data feels.
Part 4: The Australian-specific readiness layer
Generic data readiness frameworks are written for global audiences. They do not address the specific obligations an Australian business carries. The readiness assessment must include an Australian layer on top of the generic framework. The table below covers what AU-specific compliance adds.
Australian-specific readiness obligations on top of generic data readiness
| Metric | Obligation | What the readiness assessment must verify |
|---|---|---|
| Privacy Act APP 1 | Open and transparent management of personal information | Documented privacy management plan covers the AI use case |
| Privacy Act APP 10 | Quality of personal information | Personal information used by AI is accurate, current, complete |
| Privacy Act APP 11 | Security of personal information | Encryption, access control, retention and deletion procedures defined |
| Privacy Act APP 8 | Cross-border disclosure of personal information | Vendor data processing locations identified, contractual protections in place |
| Notifiable Data Breach scheme | Mandatory breach notification | Breach response plan extended to cover AI vendor incidents |
| APRA CPS 230 and CPS 234 (finserv) | Operational risk and information security | AI vendor included in operational risk register and material service provider list |
| AASB S2 climate disclosure | Mandatory climate-related financial disclosures | Emissions and energy data quality sufficient for disclosure (live for large entities) |
| Modern Slavery Act 2018 | Annual supply chain statement | AI vendor due diligence captured in modern slavery reporting |
| DISR Voluntary AI Safety Standard 2024 | Ten guardrails for safe AI deployment | Guardrail 4 (data governance) demonstrably met |
| Fair Work and state surveillance laws | Workforce monitoring and decision-making constraints | Workforce data used by AI complies with surveillance and decision-making rules |
For a deeper treatment of these obligations, our Privacy Act vs GDPR comparison, APRA and ASIC compliance article, and Fair Work compliance guide cover the regulatory detail. The point for the readiness assessment is that compliance posture is a yes-or-no question. Either the use case has been mapped against the applicable obligations and gaps remediated, or it has not. Pilots that proceed without this mapping inherit a compliance debt that almost always surfaces at the worst possible moment.
Part 5: The 30-point AI readiness checklist
The 30 questions below are organised into five sections of six. Each item is scored 0 to 3, where 0 is "not started," 1 is "documented intent," 2 is "partially in place," and 3 is "fully in place and evidenced." Total maximum: 90.
Section A: Data foundation (six questions)
Section A: Data foundation
| Metric | Question | Evidence of 3 / fully in place |
|---|---|---|
| A1 Inventory | Is there a documented inventory of every data source relevant to the use case? | Catalogue with owner, technology, volume, refresh cadence, access path for each source |
| A2 Lineage | Is the lineage of critical data elements documented end-to-end? | Origin system, transforms, destinations, and accountable owner per critical element |
| A3 Quality scoring | Has data been scored against ISO 25012 dimensions for the use case? | Quality scorecard with measured percentages, not narratives |
| A4 Access | Can the AI system access the data through a supported integration path? | API or warehouse contract documented; no manual exports |
| A5 Retention | Are retention and deletion rules defined and enforceable? | Policy documented; technical enforcement in place |
| A6 Residency | Are data residency requirements known and met? | Storage and processing locations documented; vendor contracts aligned |
Section B: Use case definition (six questions)
Section B: Use case definition
| Metric | Question | Evidence of 3 / fully in place |
|---|---|---|
| B1 Bounded scope | Is the use case bounded to one process, one team, one population of data? | Scope statement signed by sponsor; out-of-scope items listed |
| B2 Measurable outcome | Is there a quantitative success metric tied to a business outcome? | KPI with target, measurement method, and reporting cadence |
| B3 Baseline | Is the current state baselined with measurements (cost, time, error rate)? | Baseline measurements taken in the four weeks before pilot start |
| B4 Kill criteria | Are kill criteria written down and signed by the sponsor? | Specific conditions under which the pilot is stopped, with timing |
| B5 Executive sponsor | Is there a named accountable executive sponsor? | Sponsor named in the business case with decision rights |
| B6 Regulatory class | Is the use case classified for regulatory and ethical risk? | Risk classification against DISR Voluntary AI Safety Standard 2024 |
Section C: Org readiness (six questions)
Section C: Org readiness
| Metric | Question | Evidence of 3 / fully in place |
|---|---|---|
| C1 Roles | Are the build, run, and oversight roles defined and resourced? | Named individuals or vendor contract for each role through the pilot |
| C2 Training plan | Is there a training plan for users, operators, and reviewers? | Training content, schedule, and competency check defined |
| C3 Change management | Is there a change management plan for affected staff? | Communication plan, feedback mechanism, escalation path |
| C4 On-call | Is operational coverage defined for when the AI misbehaves? | On-call roster, escalation paths, vendor SLAs documented |
| C5 Internal communication | Are internal stakeholders informed and aligned? | Stakeholder map, communication schedule, FAQ |
| C6 Success criteria | Are success and failure criteria shared with the broader organisation? | Decision criteria published before the pilot starts |
For depth on the role definitions and the realistic operating cost of an AI agent in production, see our staffing-gap article and operating reality article. The change management for AI adoption guide covers C3 in detail.
Section D: Compliance posture (six questions)
Section D: Compliance posture
| Metric | Question | Evidence of 3 / fully in place |
|---|---|---|
| D1 Privacy Impact Assessment | Is a PIA completed for the use case? | PIA completed per OAIC guidance, signed and filed |
| D2 Security review | Has the vendor or build been security-reviewed? | Threat model, control mapping, penetration test results |
| D3 Board sign-off | Has the board or executive committee approved the use case? | Minuted approval with risk register entry |
| D4 Contract review | Have vendor contracts been reviewed for data, liability, exit? | Legal review complete; DPA, SLA, exit clauses verified |
| D5 Insurance review | Have cyber and professional indemnity policies been reviewed for AI coverage? | Broker confirmation that AI use case is covered |
| D6 ACL exposure | Has consumer-facing exposure been assessed under Australian Consumer Law? | Misleading-conduct risk reviewed for AI-generated outputs to consumers |
The 50-point security checklist covers D2 in depth. The ACCC consumer guarantees article covers D6.
Section E: Technical posture (six questions)
Section E: Technical posture
| Metric | Question | Evidence of 3 / fully in place |
|---|---|---|
| E1 Hosting | Is the hosting model decided and documented? | Cloud region, tenancy, network topology specified |
| E2 Integration | Are integrations with source systems specified and tested? | API contracts, error handling, retry behaviour documented |
| E3 Observability | Are logs, metrics, traces, and evaluation hooks in place? | Dashboards live; alerts routed; sample-and-review process defined |
| E4 Evaluations | Is there a documented evaluation plan with regression sets? | Eval suite covering accuracy, safety, and edge cases; runs on every change |
| E5 Exit plan | Can the organisation exit the vendor or solution within a defined window? | Exit clause, data extraction procedure, fallback process |
| E6 Retention and audit logs | Are AI inputs, outputs, and decisions logged for audit? | Immutable audit log with retention aligned to regulatory needs |
The AI vendor selection article and AWS, Azure, GCP AI services comparison cover Section E in technical depth.
Part 6: The readiness scorecard
Score each of the 30 items 0 to 3. Total the score. Match it against the bands below to determine the next action. Score honestly: the scorecard exists to inform the decision, and inflating it only moves the cost later.
Readiness scorecard bands
Two notes on using the scorecard. First, a high overall score with one or more zeros is not "ready." Zeros are blockers regardless of total. Second, the scorecard is for a specific use case, not for the organisation in general. The same business may score 85 for an internal knowledge assistant and 35 for a customer-facing pricing agent. Always score against the use case in front of you.
Part 7: Decision tree, what gap to close first
When the scorecard reveals gaps, the temptation is to address them all in parallel. That is rarely the right move for a business with finite bandwidth. The decision tree below sequences the gaps by what unlocks the most downstream value first.
What gap should be closed first?
The order is deliberate. Data accessibility is the precondition for measuring anything. Use case definition is the precondition for scoring anything. Executive sponsorship is the precondition for funding anything. Compliance is the precondition for deploying anything. Operating capability is the precondition for sustaining anything. Skip any layer and the next one collapses.
Part 8: The 12-question pre-pilot sign-off checklist for the CIO or CDO
When the readiness assessment is complete and the scorecard supports a "proceed to pilot" decision, the CIO or CDO signs off with the following 12 questions answered yes in writing. Anything short of yes is either an open risk to be accepted by name, or a reason to delay.
- Is the use case scope statement signed by the executive sponsor with documented kill criteria?
- Is the data inventory complete with lineage documented for every critical element?
- Is the data quality scorecard above the threshold required for the use case?
- Are data residency, retention, and access controls aligned to Australian obligations?
- Has the Privacy Impact Assessment been completed and accepted by the privacy officer?
- Has the security review been completed with no unresolved high or critical findings?
- Is the vendor contract signed with data, liability, exit, and SLA terms reviewed by legal?
- Are the build, run, and oversight roles staffed (internal or vendor) through the pilot?
- Is the change management plan published, with training scheduled and feedback channels open?
- Is the observability stack live with dashboards, alerts, and evaluation hooks ready?
- Is the baseline measured in the four weeks before pilot start, with the KPI calculation method agreed?
- Has the board or executive committee accepted the residual risk in minuted form?
This list is short on purpose. If a CIO or CDO cannot answer all 12 affirmatively, the pilot is not ready to start. It does not mean the pilot is a bad idea; it means the next four to eight weeks should be spent closing the gaps before any model is touched.
For the operational playbook that takes over the moment the sign-off is granted, our AI pilot project success factors article is the next read. For the financial framing the business case will need to land, our automation payback period article, business case template, and AI business case for the board article cover the maths. For the ethical framing that increasingly appears in board questions, the AI ethics framework article is the canonical Solve8 reference.
The Solve8 view
Across the work documented in our Carbonly.ai case study and RootCauseAI case study, a consistent pattern repeats: the technical lift of building an AI system is rarely the binding constraint. The binding constraint is the data and organisational foundations underneath it. A business that gets the readiness work right can move quickly, with high confidence, and recover from the inevitable mid-pilot surprises. A business that skips it spends three times as long, learns the lessons in production, and frequently abandons the program after the first significant incident.
The 30-point framework surfaces work that has to happen anyway, before the surprises become expensive. Most clients find that the readiness exercise itself produces value: a cleaner data inventory, clearer ownership, sharper use case definition, and a sponsor who actually understands what they are sponsoring.
If the assessment in this article reveals that your organisation sits below 75 on the scorecard, the right move is almost always to spend two to six months on foundations before the pilot. The cost of doing so is small. The cost of not doing so is the picture painted in our production-reality article: a system that limps along, drains operating capacity, and never delivers the outcomes the board signed off on. Our DIY caution piece and the Australian data sovereignty guide are companion reading for the foundations conversation, and our AI strategy service and managed AI services describe the engagement models we use to support that work.
If you want help running this assessment for a specific use case, or you want a second opinion on a scorecard your team has already produced, we run pre-pilot readiness reviews as a fixed-scope engagement. Book a 30-minute consultation here: https://calendly.com/solve8/30min.
Related Reading:
- AI Pilot Project Success Factors: The 90-Day Playbook for Australian Businesses - The operational playbook that begins the moment readiness sign-off is granted
- The AI Agent Staffing Gap at Australian Businesses - Sizes the org capability dimension of the readiness scorecard
- Operating AI Agents in Production: The Reality for Australian Business - Why unready pilots fail expensively in production
- AI Business Case Template for Australian Board Presentations - The financial framing that complements the readiness scorecard
Sources: Research synthesised from DAMA-DMBOK 2 (Data Management Body of Knowledge), ISO/IEC 25012:2008 Data Quality model, AS ISO 8000 series, EDM Council Cloud Data Management Capabilities (CDMC) framework, OAIC Privacy Impact Assessment guidance, DISR Voluntary AI Safety Standard 2024, NIST AI Risk Management Framework 1.0, Australia's AI Ethics Principles (DISR 2019), Privacy Act 1988 (APP 1, 8, 10, 11), Notifiable Data Breaches scheme, APRA CPS 230 and CPS 234, AASB S2 Climate-related Disclosures, Modern Slavery Act 2018, MIT Sloan State of AI in Business 2025, and Gartner data quality research.