Running a large language model locally means your prompts, code, and documents never leave your machine — no API bills, no rate limits, no data sent to a third party. On Linux, the fastest path from zero to a working local LLM is Ollama: a single binary that manages model downloads, GPU acceleration, and an HTTP API, wrapped in a systemd service.
This guide walks through the full setup on a real Linux system: installation, hardware sizing, GPU support, systemd configuration, the REST API, model customization, and the troubleshooting steps you will actually need.
Hardware: What You Actually Need
The single number that matters most is memory. A model must fit in VRAM (GPU) or RAM (CPU) to run, and quantization — storing weights at 4-bit instead of 16-bit precision — is what makes consumer hardware viable. Rough sizing for the default 4-bit quantized models Ollama ships:
| Model size | Approx. memory needed | Realistic hardware |
|---|---|---|
| 1–3B params | 2–4 GB | Any modern laptop, even CPU-only |
| 7–8B params | 5–8 GB | 8 GB GPU, or 16 GB RAM on CPU |
| 13–14B params | 10–12 GB | 12–16 GB GPU |
| 70B+ params | 40 GB+ | Multi-GPU or workstation cards |
No GPU? Ollama falls back to CPU automatically. An 8B model on a recent multi-core CPU is slow but usable for short tasks. If the model does not fully fit in VRAM, Ollama splits layers between GPU and CPU — it works, but throughput drops sharply, so pick a model that fits.
Installing Ollama
The official install script detects your distribution and GPU, installs the binary to/usr/local/bin, creates anollama system user, and registers a systemd service:
$ curl -fsSL https://ollama.com/install.sh | sh
# Verify the service is running
$ systemctl status ollama
# Verify the CLI
$ ollama --version
Prefer to read a script before piping it into a shell? Good instinct — download it first, inspect it, then run it. You can also grab the standalone binary from the Ollama GitHub releases page and write your own unit file.
After installation, the API listens on 127.0.0.1:11434. It is bound to localhost by default, which is exactly what you want until you deliberately decide otherwise (more on that below).
Running Your First Model
ollama run pulls the model on first use and drops you into an interactive chat. Start small to confirm everything works:
# Pull and chat with a small model
$ ollama run llama3.2
# One-shot prompt, no interactive session
$ ollama run llama3.2 "Explain what /etc/fstab does in two sentences"
# Pipe a file in — great for scripting
$ cat error.log | ollama run llama3.2 "Summarize these errors"
Day-to-day model management uses a Docker-like verb set:
$ ollama list # installed models and sizes
$ ollama ps # which models are loaded, CPU vs GPU split
$ ollama pull qwen3 # download without running
$ ollama show llama3.2 # context length, parameters, license
$ ollama rm mistral # delete a model, reclaim disk
Model files are large — check ollama list and prune what you do not use. When running as the systemd service, models live under/usr/share/ollama/.ollama/models; runningollama serve manually stores them in~/.ollama/models. A surprise second copy of a 40 GB model is a classic way to fill a root partition.
GPU Acceleration: NVIDIA and AMD
Ollama uses CUDA on NVIDIA cards and ROCm on supported AMD cards. On NVIDIA, you only need the proprietary driver — Ollama bundles its own CUDA runtime libraries:
# Confirm the driver sees your GPU
$ nvidia-smi
# Ubuntu/Debian: install the recommended driver
$ sudo ubuntu-drivers autoinstall
# After a chat, confirm the model is on GPU
$ ollama ps
# PROCESSOR column should say "100% GPU"
For AMD, install ROCm and make sure the user running Ollama is in therender andvideo groups. Support is limited to specific GPU generations — check the Ollama documentation for the current list before buying hardware.
If Ollama silently falls back to CPU, the logs say why:journalctl -u ollama -f shows GPU detection at startup, including missing libraries or an unsupported compute capability.
Configuring the Systemd Service
Ollama is configured through environment variables on its unit. Never edit the unit file directly — use a drop-in override so upgrades do not clobber your changes:
$ sudo systemctl edit ollama
# Add in the editor:
[Service]
Environment="OLLAMA_KEEP_ALIVE=30m"
Environment="OLLAMA_MODELS=/data/ollama/models"
$ sudo systemctl daemon-reload && sudo systemctl restart ollama
The variables you will actually reach for:
OLLAMA_KEEP_ALIVE — how long a model stays in memory after the last request (default 5m). Raise it to avoid reload latency; set -1 to keep it loaded forever.
OLLAMA_MODELS — model storage path. Point it at a big data disk; remember the ollama user needs write access.
OLLAMA_HOST — bind address. 0.0.0.0 exposes the API to your network — see the security note below.
OLLAMA_NUM_PARALLEL — concurrent requests per model. Each parallel slot multiplies context memory use.
OLLAMA_MAX_LOADED_MODELS — how many models may sit in memory at once.
Security note: the Ollama API has no authentication. If you setOLLAMA_HOST=0.0.0.0, anyone who can reach port 11434 can run inference and delete your models. Expose it only behind a reverse proxy with auth, a firewall rule scoped to trusted IPs, or a VPN/tailnet — never directly on the internet. Our server hardening checklist applies here too.
Using the API
Everything the CLI does goes through the local REST API, which makes Ollama scriptable from anything that can speak HTTP:
# Chat endpoint (streaming off for readable output)
$ curl http://localhost:11434/api/chat -d '{
"model": "llama3.2",
"stream": false,
"messages": [{"role": "user", "content": "Why is the sky blue?"}]
}'
Ollama also serves an OpenAI-compatible endpoint at/v1/chat/completions. Most OpenAI SDK clients work by changing the base URL to http://localhost:11434/v1 and passing any placeholder API key. That means editors, chat UIs like Open WebUI, and existing scripts can switch to a local model with a one-line config change.
This is where local LLMs earn their keep for admins: pipe journalctl output into a summarization prompt, generate commit messages in a git hook, or draft firewall rule explanations — all offline, all free per token.
Customizing Models with a Modelfile
A Modelfile is to models what a Dockerfile is to images: it layers a system prompt and parameters on top of a base model and saves the result under a new name:
# Modelfile
FROM llama3.2
PARAMETER temperature 0.2
PARAMETER num_ctx 8192
SYSTEM "You are a Linux sysadmin assistant. Answer concisely with commands first, explanation second. Never invent flags."
$ ollama create sysadmin -f Modelfile
$ ollama run sysadmin "show me open ports"
Lower temperature makes output more deterministic — good for command generation. Raisingnum_ctx (context window) lets the model read longer logs or files, at the cost of more memory. This is often the difference between a toy and a genuinely useful local assistant.
Troubleshooting
Model runs on CPU despite a GPU — check journalctl -u ollama for GPU detection errors; verify nvidia-smi works as the ollama user; restart the service after driver updates.
"connection refused" on port 11434 — the service is not running (systemctl status ollama), or you started ollama serve manually in another terminal and it has since exited.
Out-of-memory or model crashes mid-generation — the model plus context does not fit. Use a smaller model or quantization, reduce num_ctx, or lower OLLAMA_NUM_PARALLEL.
First response is slow, later ones fast — that is model load time from disk. Raise OLLAMA_KEEP_ALIVE so the model stays resident between requests.
Disk filling up — old model versions accumulate. ollama list then ollama rm what you no longer use, and remember the two possible storage locations noted above.
Slow generation after an upgrade — check ollama ps: if the PROCESSOR column shows a CPU/GPU split that used to be 100% GPU, the new model version may simply be bigger than your VRAM.
Best Practices
1. Start with an 8B-class model — validate the workflow before downloading 40 GB of weights.
2. Keep the API on localhost — put a reverse proxy with authentication in front if others need access.
3. Put models on a data disk — set OLLAMA_MODELS before you accumulate hundreds of gigabytes on the root partition.
4. Pin versions for automation — pull models by tag, not just name, so scripted pipelines do not change behavior when a default tag moves.
5. Match the model to the task — a small fast model for log summaries and shell help, a larger one for code review and reasoning.
6. Review AI-generated commands before running them — a local model hallucinates flags just as confidently as a cloud one.
7. Monitor like any other service — it is a systemd unit with logs in journald; alert on failures the same way you do for nginx or postgres.
Frequently Asked Questions
Do I need a GPU to run Ollama on Linux?
No. Ollama falls back to CPU automatically. Small quantized models (1B–8B) are usable on modern CPUs, but a GPU with enough VRAM makes responses dramatically faster.
Where does Ollama store models on Linux?
Under /usr/share/ollama/.ollama/models when running as the systemd service, or ~/.ollama/models when you run ollama serve yourself. Relocate with OLLAMA_MODELS.
Is my data private with a local LLM?
Yes — inference happens entirely on your machine, and the API binds to localhost by default. Ollama only touches the network when pulling models.
How much memory do I need?
For 4-bit quantized models: roughly 5–8 GB for 7–8B models, 10–12 GB for 13–14B, 40 GB+ for 70B-class. The model should fit entirely in VRAM for full GPU speed.
Can I use Ollama with OpenAI-compatible tools?
Yes — point the client's base URL at http://localhost:11434/v1 with any placeholder API key.