Offline AI for Australian Business: Local LLM Guide

To use AI offline, install a local runner such as Ollama or LM Studio, download an open-weight small language model (for example Llama 3.1 8B, Mistral 7B or Phi-4-mini), and run it on your own laptop or server. Prompts and documents never leave the device, which is why many Australian organisations that block ChatGPT allow approved local AI.

Many organisations restrict public AI websites to protect confidential information. The good news is that capable models now run entirely on a modern business laptop, with no internet connection after download and no API calls to external servers.

This guide covers how local AI works, which tools and models to use, hardware requirements, step-by-step setup, realistic use cases, and how to approach IT. It is for staff and IT teams at Australian organisations that need AI without sending data offshore.

---

Why do companies block cloud AI tools?

When you type into a public AI website, your text travels over the internet to servers usually in the US or EU, may be logged or retained, and the response comes back.

1. 1You TypeEnter your question
2. 2Data SentOver the internet
3. 3External ServerOften processed offshore
4. 4Stored or LoggedPotentially retained
5. 5ResponseSent back to you

Data leakage is real. Group-IB reported in 2024 that more than 225,000 sets of OpenAI ChatGPT credentials were found for sale, harvested from employee devices by infostealer malware, which would expose chat histories to whoever logged in. In 2023 Samsung staff uploaded proprietary source code to ChatGPT, and several banks restricted the tool while they assessed it.

Compliance obligations apply. Depending on your sector and data:

Intellectual property. Terms on training and retention vary by product and can change. With local AI, prompts stay on hardware you control. Our piece on how the wrong AI tools leak business data goes deeper.

---

What is local AI and how does it work offline?

Local AI runs the model directly on your computer.

1. 1You TypeEnter your question
2. 2Your CPU/GPUProcesses locally
3. 3Local ModelStored on your drive
4. 4ResponseNever leaves the device

• Privacy: data stays on the device.
• No per-use fees: open-weight models are free to download; you pay for hardware.
• Works offline: on a plane, at a remote site, in a secure facility.
• No rate limits: throughput is limited only by hardware.
• Control: you choose the model and its version.

The trade-off: local models are smaller than the largest cloud models. For routine office work (drafting, summarising, meeting notes, formulas) they are more than capable.

---

Which local AI tools should you use?

	Ease of Use	Flexibility	Impact
LM Studio	Easiest (desktop app)	High	Best for beginners
Ollama	Easy (command line)	Highest	Best for power users and integration
Jan	Very easy	Medium	Polished open-source app
GPT4All	Easy	Medium	Simple desktop experience

LM Studio: best for beginners

A free desktop app with a chat interface. Download, install, search for a model, download it and start chatting. It supports file uploads, runs on Windows, macOS and Linux, is free for work use, and offers an OpenAI-compatible local API. Version 0.3.36 was current as of January 2026. See our LM Studio beginner's guide.

Ollama: best for power users

A command-line tool that makes running local models as simple as ollama run llama3.2. It has a large model library, runs as a local API server for integration, and works on macOS, Windows and Linux. Many other tools build on it. See why to run Ollama locally.

Jan: polished open source

An open-source desktop assistant that runs offline, with a clean interface and optional connections to cloud APIs if you choose.

GPT4All: simple desktop app

From Nomic AI, designed for consumer hardware, with a basic but functional interface and models typically a few gigabytes in size.

---

Which small language model should you download?

What's your primary use case?

• General tasks (email, summaries, Q&A)Llama 3.1 8B or Mistral 7B
• Coding and technical workDeepSeek-R1-Distill-Qwen-7B or a Qwen coder model
• Limited hardware (8GB RAM or less)Phi-4-mini (3.8B), Llama 3.2 3B or Qwen 2.5 3B
• Complex reasoning and analysisA 32B to 70B model, if hardware allows

What is a small language model?

A small language model has roughly 1 to 10 billion parameters, small enough to run on a laptop. They handle structured, everyday language tasks well and use far less memory than frontier cloud models.

All-round models

• Llama 3.1 8B (Meta): about 5GB download; 8GB RAM minimum, 16GB recommended. A good balance of capability and hardware.
• Mistral 7B (Mistral AI): about 4GB; 8GB RAM. Fast and dependable for summaries and drafting.

Small but capable

• Phi-4-mini (Microsoft, 3.8B): about 2.5GB; runs on 4 to 8GB RAM. Strong for its size.
• Llama 3.2 3B (Meta): the default llama3.2 in Ollama, about 2GB. Good on older laptops.
• Qwen 2.5 3B (Alibaba): about 2GB; strong multilingual support.

Maximum capability

• DeepSeek-R1-Distill-Qwen-32B: about 20GB; 32GB+ RAM. Strong reasoning and coding.
• Llama 3.1 70B: about 40GB; 64GB+ RAM or a large GPU.

---

Can your laptop run a local LLM?

Minimum for small models (up to about 8B): 8GB RAM (16GB recommended), 10GB free storage (50GB for flexibility), any processor from the last five years. A GPU helps but is not required.

	Example models	Hardware Needed	Impact
3-8B parameters	Phi-4-mini, Mistral 7B, Llama 3.1 8B	8-16GB RAM, no GPU needed	Most business laptops
13-32B parameters	Qwen 2.5 14B/32B, DeepSeek 32B	32GB RAM or a capable GPU	High-end laptops
70B parameters	Llama 3.1 70B	64GB+ RAM or large GPU	Workstations and servers

• Windows: most business laptops from 2022 onward with 16GB RAM handle 7 to 8B models comfortably.
• Mac: Apple Silicon (M1 to M4) is well suited because of unified memory. 16GB is a good minimum.
• Linux: same RAM guidance; NVIDIA GPUs with CUDA give the best acceleration.

A 2019 laptop with 8GB RAM and no GPU will struggle with anything beyond the smallest models.

---

How do you install Ollama and run AI locally?

1. 2 minsDownload and InstallGet Ollama on your system
2. 1 minVerify InstallationCheck it's working
3. 5 minsDownload a ModelGet your first model
4. 2 minsStart UsingAsk your first question

macOS: install with Homebrew, or download from ollama.com and drag to Applications.

brew install ollama
ollama --version

Windows: download the installer from ollama.com, run it, then open Command Prompt and check the version.

ollama --version

Linux:

curl -fsSL https://ollama.com/install.sh | sh
ollama --version

Download and run a model:

ollama pull llama3.2
ollama run llama3.2

At the >>> prompt, type your request, for example Summarise this email in 3 bullet points: [paste text]. Type /bye to exit.

Once a model is downloaded, you can disconnect from the internet and it keeps working.

---

How do you install LM Studio?

1. Go to lmstudio.ai and download for your operating system.
2. Install (drag to Applications on Mac; run the installer on Windows) and launch.
3. Open the Discover tab, search for "llama" or "mistral", and click Download.
4. Open the chat view, select the model and type your question.

You now have a private assistant on your laptop.

---

What can you use offline AI for at work?

1. Email drafting: "Write a polite email declining a meeting this week but offering next week."
2. Document summaries: "Summarise this document in 5 key points." Break long documents into chunks; many local setups use 8K to 32K token context windows.
3. Meeting notes: "Turn these notes into attendees, decisions, action items with owners, and next steps."
4. Formula and code help: "Explain this Excel formula and suggest improvements."
5. Data analysis scripts: "Write a Python script to total monthly sales by region from this CSV."
6. Translation: Qwen and Mistral models handle common languages well; check important translations with a fluent speaker.
7. Report drafts: "Draft an executive summary from these points."

---

How does local AI compare with cloud AI?

	Local AI	Cloud AI (frontier models)	Impact
Privacy	Data stays on the device	Data sent to provider servers	Local wins
Cost model	Hardware you already own	Per-user subscription or per-token fees	Depends on volume
Speed	Slower on laptops	Fast	Cloud wins
Capability	Good for routine tasks	Stronger reasoning and writing	Cloud wins
Availability	Works offline	Needs internet, subject to outages	Local wins
Context length	Typically shorter	Much longer	Cloud wins

Use local AI for first drafts, formatting, meeting notes, formula and code help, and anything involving sensitive data.

Use approved cloud AI for complex multi-step analysis, very long documents and tasks that need current information, where policy allows.

For organisation-wide use, a middle path is private AI infrastructure: larger open models running on your own servers or your cloud in Australia, shared securely across teams. Our data sovereignty guide explains the options.

---

How should you approach IT about local AI?

Do not say: "ChatGPT is blocked and I need it unblocked."

Do say: "Can we assess local AI tools that run entirely offline, with no data leaving the device, against our security requirements?"

Points for IT:

1. No network calls once models are downloaded.
2. No API keys to manage.
3. Works in air-gapped environments.
4. Ollama, Jan and GPT4All are open source and auditable.
5. Uses existing hardware.

Your organisation will still need a software approval process, an acceptable use policy for AI, rules on what data may be processed, and a list of approved models. A sensible start is a small pilot group with defined use cases for 30 days, written guidelines, approved models only, and feedback before expanding.

• Data leaving the deviceNone after model download
• Per-use feesNone
• Setup time for one laptopUnder an hour
• Works without internetYes

---

What should you do this week?

1. Install LM Studio or Ollama.
2. Download Llama 3.1 8B or Mistral 7B (or Phi-4-mini on older hardware).
3. Try summarising a non-sensitive document or drafting an email.
4. Compare a couple of models on your real tasks.
5. Talk to IT about formalising approved use.

---

How Solve8 can help

Setting up one laptop is straightforward. Rolling local or private AI out across hundreds of staff needs approved models, IT policy, access controls and hardware sizing. Our engineers design and deploy private AI that runs on your infrastructure in Australia. See our private AI infrastructure service, or contact us to discuss your requirements.