Local AI models are having a moment. New open models land every few weeks, and “local vs cloud” is a running debate in hobbyist forums. This guide shows you how to install Ollama, chat with a small model in your terminal, call it from code, and give it a custom personality. Every command below comes from Ollama’s official docs.
What you need
Ollama is a free tool that downloads open models and runs them on your own machine. It gives you a command line and a local web API. Once a model is downloaded, your chats are processed on your computer.
Here is the checklist:
- A supported operating system. The official docs list macOS Sonoma (v14) or newer, and Windows 10 22H2 or newer. Linux is supported through an install script.
- Disk space. The Windows docs say the app needs at least 4GB for the binary install, and that models “can be tens to hundreds of GB in size”. Small models are far smaller than that.
- Enough memory for the model you pick. A model has to fit in your GPU’s memory (VRAM) or your regular RAM. Each model page on the Ollama library lists its download size, which is a good first guide. Pick a model that is clearly smaller than the memory you have free.
- A GPU helps, but isn’t required. On Macs, Apple M-series chips get CPU and GPU support, while Intel Macs run on the CPU only. On Windows and Linux, Ollama supports many NVIDIA and AMD cards. Check the GPU docs for your card.
Running on the CPU works. It’s just slower.
Step 1: Install Ollama
The download page offers a one-line installer for each system.
On macOS or Linux, open a terminal and run:
curl -fsSL https://ollama.com/install.sh | sh
On Windows, open PowerShell and run:
irm https://ollama.com/install.ps1 | iex
If you’d rather not pipe a script into your shell, you can download the app instead: Ollama.dmg for Mac or OllamaSetup.exe for Windows, both linked from the download page. On a Mac, drag the app into Applications. The first time it starts, it checks that the ollama command is in your PATH. If it isn’t, it asks permission to create a link in /usr/local/bin.
Check that it worked:
ollama -v
You should see a version number. On Windows, Ollama runs in the background after install. The ollama command works in cmd, PowerShell or any terminal.
Step 2: Pick a small model
The library has hundreds of models, and many are big. For a first try, use Llama 3.2. Its tags page lists llama3.2:3b at 2.0GB and llama3.2:1b at 1.3GB. The plain llama3.2 name points to the 3B version. It’s also the model Ollama uses in many of its own API examples.
If your machine has little memory, start with llama3.2:1b. It’s smaller and lighter, with simpler answers.
Step 3: Run your first chat
One command downloads the model (if needed) and opens a chat:
ollama run llama3.2
The first run shows a download progress bar. Then you get a prompt. The output looks roughly like this (abbreviated):
pulling manifest
pulling ... 100%
success
>>> Explain what a CPU does in one sentence.
A CPU is the part of a computer that carries out instructions ...
>>> /bye
Type /bye to leave the chat. To paste text across several lines, wrap it in triple quotes (""").
Step 4: Manage your models
These four commands cover almost everything you’ll do day to day.
Download a model without starting a chat:
ollama pull llama3.2:1b
See which models are on your disk (ollama ls does the same thing):
ollama list
See which models are loaded in memory right now:
ollama ps
The PROCESSOR column is worth a look. According to the FAQ, 100% GPU means the model is fully on the GPU. 100% CPU means it’s in system memory. A split like 48%/52% CPU/GPU means it’s partly in each. By default, models stay loaded for 5 minutes after you last use them. ollama stop llama3.2 unloads a model right away.
Delete a model you no longer need:
ollama rm llama3.2:1b
Step 5: Talk to it through the local API
While Ollama is running, it serves an API at http://localhost:11434. Any app on your computer can use it. Here is a chat request, following the API reference:
curl http://localhost:11434/api/chat -d '{
"model": "llama3.2",
"messages": [
{ "role": "user", "content": "Why is the sky blue?" }
],
"stream": false
}'
Without "stream": false, you get a stream of small JSON chunks instead of one reply. With it, you get a single object shaped like this (trimmed):
{
"model": "llama3.2",
"created_at": "...",
"message": {
"role": "assistant",
"content": "The sky looks blue because ..."
},
"done": true
}
There is also /api/generate, which takes a single "prompt" string instead of a messages list. It returns the text in a "response" field.
Many tools already speak the OpenAI API format. Ollama’s OpenAI compatibility docs describe an endpoint at http://localhost:11434/v1/. You can call it like this:
curl http://localhost:11434/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "llama3.2",
"messages": [{ "role": "user", "content": "Hello!" }]
}'
OpenAI client libraries need an API key. The docs set it to ollama and note that it is “required but ignored”.
Step 6: Give the model a personality with a Modelfile
A Modelfile is a short text file that builds a custom model on top of an existing one. The Modelfile reference lists the instructions. FROM (required) sets the base model, PARAMETER changes settings, and SYSTEM sets the system message.
Create a file called Modelfile with this content:
FROM llama3.2
PARAMETER temperature 0.7
SYSTEM """You are a patient tutor. Explain things in plain English, in short paragraphs, and give one example."""
Build it and run it:
ollama create tutor -f ./Modelfile
ollama run tutor
Your new tutor model now shows up in ollama list.
Step 7: Change where models are stored, or who can reach the server
According to the FAQ, models are stored here by default:
- macOS:
~/.ollama/models - Linux:
/usr/share/ollama/.ollama/models - Windows:
C:\Users\%username%\.ollama\models
To use another drive, set the OLLAMA_MODELS environment variable to the folder you want. How you set it depends on your system:
- macOS app: run
launchctl setenv OLLAMA_MODELS "/path/to/folder", then restart Ollama. - Linux (systemd): run
systemctl edit ollama.serviceand addEnvironment="OLLAMA_MODELS=/path/to/folder"under[Service]. Then runsystemctl daemon-reloadandsystemctl restart ollama. Theollamauser also needs read and write access to the folder. - Windows: quit Ollama, then search Settings for “environment variables”. Add
OLLAMA_MODELSto your user account and start Ollama again.
OLLAMA_HOST works the same way. By default, Ollama only listens on 127.0.0.1 port 11434, so other devices can’t reach it. Setting OLLAMA_HOST to 0.0.0.0:11434 opens it to your network. Do that only on a network you trust. Anything on your network could then send it requests.
A quick word on privacy
The Ollama FAQ says it plainly: “Ollama runs locally. We don’t see your prompts or data when you run locally.” Downloading models still needs the internet. Ollama also offers optional cloud models, and those run on Ollama’s servers. If you want to stay fully offline, the FAQ explains how to turn the cloud features off. You can set OLLAMA_NO_CLOUD=1, or put "disable_ollama_cloud": true in ~/.ollama/server.json.
That makes local models a good fit for private notes and drafts.
Common errors and fixes
- “command not found” right after installing. Open a new terminal window so it picks up the updated PATH. On a Mac, make sure you accepted the prompt to create the link in
/usr/local/bin. - “connection refused” on localhost:11434, or the server isn’t running. The desktop app has to be open (look for it in the menu bar or taskbar). On Linux, run
systemctl status ollama, or start the server by hand withollama serve. - Port 11434 already in use. This usually means Ollama is already running, for example the app plus a manual
ollama serve. Quit one of them. If another program really owns that port, useOLLAMA_HOSTto make Ollama listen on a different one. - Model too big or out of memory. Switch to a smaller tag like
llama3.2:1b, close other heavy apps, and checkollama psto see where the model loaded. - Very slow replies. If
ollama psshows100% CPU, the model is running without your GPU. Check that your card and driver are on the supported list in the GPU docs, or use a smaller model. - Need more detail? The troubleshooting page explains where the logs are. On a Mac, check
~/.ollama/logs/server.log. On Linux, runjournalctl -u ollama. On Windows, open%LOCALAPPDATA%\Ollama.
Wrapping up
You now have a working local AI setup. It can chat in the terminal, answer API calls on localhost, and run a custom tutor you built yourself. Start small, watch ollama ps, and try larger models from the library only once you know how much memory your machine has spare.
Sources and official docs
Links last checked on 15 November 2026.





Write a response