Ollama is the fastest way to run a local model, and it uses llama.cpp underneath. Building llama.cpp yourself is what you do when you want the layer below that: a specific compute backend, an OpenAI-compatible server you fully control, your own quantizations, and a real benchmarking tool.
This guide covers building each backend on Linux, running and serving GGUF models, converting and quantizing your own weights, and the compile errors you will actually hit.
Prerequisites
llama.cpp builds with CMake. The core dependencies are small — a C/C++ toolchain, CMake, git, and libcurl if you want the built-in model downloader:
# Debian / Ubuntu
$ sudo apt update
$ sudo apt install -y build-essential cmake git libcurl4-openssl-dev
# Fedora / RHEL-family
$ sudo dnf install -y gcc-c++ cmake git libcurl-devel
# Arch
$ sudo pacman -S base-devel cmake git curl
$ git clone https://github.com/ggml-org/llama.cpp
$ cd llama.cpp
Everything below runs from that clone. Builds go into build/ and binaries land in build/bin/ — llama.cpp does not install system-wide by default, which makes it easy to keep several backend builds side by side.
Choosing a Backend
Pick the backend that matches your hardware. This is the single decision that determines your performance:
| Backend | CMake flag | Use when |
|---|---|---|
CPU | (default) | No supported GPU, or small models |
CUDA | -DGGML_CUDA=ON | NVIDIA GPU — fastest, most tested path |
HIP | -DGGML_HIP=ON | AMD GPU with ROCm installed |
Vulkan | -DGGML_VULKAN=ON | Any vendor — AMD, Intel, NVIDIA, no CUDA/ROCm needed |
SYCL | -DGGML_SYCL=ON | Intel GPUs via oneAPI |
If you have an AMD or Intel GPU and ROCm/oneAPI setup is fighting you, try Vulkan first. It needs only a working Mesa or vendor driver plus the Vulkan headers, and it gets you GPU acceleration in minutes instead of hours.
Building for CPU
The default build already enables the CPU feature detection that matters (AVX2, AVX-512, and friends are selected at build time for your machine):
$ cmake -B build
$ cmake --build build --config Release -j $(nproc)
# Binaries are now in build/bin/
$ ls build/bin/ | head
For CPU-only inference you can optionally link an optimized BLAS library, which mainly speeds up prompt processing (the batch phase), not token generation:
$ sudo apt install -y libopenblas-dev
$ cmake -B build -DGGML_BLAS=ON -DGGML_BLAS_VENDOR=OpenBLAS
$ cmake --build build --config Release -j $(nproc)
Portability warning — a native build may use instructions your other machines lack. If you are building on one host and running on another, that binary can die with an illegal instruction — build on the target, or in a container matching it.
Building with CUDA (NVIDIA)
You need the CUDA toolkit — specifically nvcc — not just the driver. If nvcc --version fails, install the toolkit first (see our NVIDIA driver and CUDA guide):
$ cmake -B build -DGGML_CUDA=ON
$ cmake --build build --config Release -j $(nproc)
# Faster builds: compile only for your GPU's architecture
# (86 = Ampere consumer, 89 = Ada, 90 = Hopper — check your card)
$ cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86
Restricting CMAKE_CUDA_ARCHITECTURES to your own card cuts build time dramatically, because the default builds fat binaries covering many architectures. The tradeoff is that the binary will not run on a different GPU generation.
Building with ROCm / HIP (AMD)
With ROCm installed, point CMake at the ROCm clang and name your GPU target. rocminfo reports the target string (for example gfx1100):
$ rocminfo | grep gfx
$ HIPCXX="$(hipconfig -l)/clang" HIP_PATH="$(hipconfig -R)" \
cmake -B build -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1100 -DCMAKE_BUILD_TYPE=Release
$ cmake --build build --config Release -j $(nproc)
If your card is not on ROCm's supported list, the HSA_OVERRIDE_GFX_VERSION workaround often makes it run by presenting a closely related target — covered in our ROCm setup guide.
Building with Vulkan (Any GPU)
Vulkan is the pragmatic path for AMD and Intel hardware. It needs the loader, headers, and the shader compiler:
# Debian / Ubuntu
$ sudo apt install -y libvulkan-dev glslc vulkan-tools
# Confirm the GPU is visible to Vulkan
$ vulkaninfo --summary
$ cmake -B build -DGGML_VULKAN=ON
$ cmake --build build --config Release -j $(nproc)
Running a Model
llama.cpp runs GGUF files. Download one from Hugging Face, then run it. The flag that matters most is -ngl — how many layers to offload to the GPU:
# Interactive chat, all layers on GPU, 8K context
$ ./build/bin/llama-cli -m models/model.gguf -ngl 99 -c 8192
# One-shot prompt with a token limit
$ ./build/bin/llama-cli -m models/model.gguf -ngl 99 -p "Explain inodes briefly" -n 200 -no-cnv
# CPU-only with an explicit thread count
$ ./build/bin/llama-cli -m models/model.gguf -t $(nproc) -c 4096
-ngl N — layers offloaded to GPU. 99 means "all of them". Lower it when VRAM is tight.
-c N — context window in tokens. Larger contexts consume noticeably more memory.
-t N — CPU threads. Physical cores usually beats logical cores here.
-n N — maximum tokens to generate.
On startup llama.cpp prints which backend it loaded and how many layers went to the GPU. That output is your proof the build did what you wanted — read it before you start blaming the model for being slow.
Serving an OpenAI-Compatible API
llama-server is the piece that makes llama.cpp genuinely useful in a homelab: a small HTTP server with a built-in web UI and an OpenAI-compatible endpoint, so existing clients work unchanged.
$ ./build/bin/llama-server -m models/model.gguf -ngl 99 -c 8192 --host 127.0.0.1 --port 8080
# Web UI: http://localhost:8080
# API: http://localhost:8080/v1/chat/completions
$ curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
"messages": [{"role": "user", "content": "Summarize what cgroups do"}]
}'
Like Ollama, llama-server has no authentication by default. Keep it bound to 127.0.0.1 and put a reverse proxy with auth in front of it before exposing it to a network — the same rule from our server hardening checklist.
Converting and Quantizing Your Own Models
This is the capability Ollama does not give you: take any Hugging Face model, convert it to GGUF, and quantize it to whatever precision fits your card.
$ python3 -m venv .venv && source .venv/bin/activate
$ pip install -r requirements.txt
# 1. Convert HF weights to GGUF at 16-bit
$ python3 convert_hf_to_gguf.py /path/to/hf-model --outfile model-f16.gguf --outtype f16
# 2. Quantize to a smaller, faster file
$ ./build/bin/llama-quantize model-f16.gguf model-Q4_K_M.gguf Q4_K_M
# List every available quantization type
$ ./build/bin/llama-quantize --help
Q4_K_M is the usual default: a large size reduction for a small quality cost. Move up to Q5_K_M or Q6_K if you have VRAM to spare, or down to Q3_K_M when you are desperate to fit a bigger model.
Benchmarking
llama-bench measures prompt processing and token generation separately, and repeats runs so the numbers mean something. Use it to compare backends or -ngl settings on your hardware rather than trusting anyone else's figures:
# Compare GPU offload levels
$ ./build/bin/llama-bench -m models/model.gguf -ngl 0,20,99
# Compare thread counts on CPU
$ ./build/bin/llama-bench -m models/model.gguf -ngl 0 -t 4,8,16
The output splits results into pp (prompt processing, the batch phase) and tg (token generation, the streaming phase). They scale differently — a change that helps one can do nothing for the other.
Troubleshooting
nvcc: command not found — the CUDA toolkit is not installed or not on PATH. Add /usr/local/cuda/bin to PATH, or install the toolkit.
CMake version too old — llama.cpp tracks a recent CMake. Install a newer one via pip install cmake or your distro backports rather than fighting an old package.
Build succeeds but runs on CPU — you configured an existing build directory without the backend flag. CMake caches configuration — delete build/ and configure again.
CUDA error: out of memory — lower -ngl so fewer layers sit in VRAM, reduce -c, or use a smaller quantization.
Illegal instruction (core dumped) — the binary was built for CPU features this machine lacks. Rebuild on the target host.
HIP error / no ROCm devices — the user is not in the render and video groups, or the GPU target is unsupported. Check rocminfo first.
Gibberish output — usually a mismatched or truncated GGUF download, or a chat template mismatch. Re-download and verify the file size.
Best Practices
1. Keep separate build dirs — use -B build-cuda and -B build-cpu so you can A/B backends without reconfiguring.
2. Pin a commit for production — llama.cpp moves fast. Note the commit you deployed so you can reproduce it.
3. Re-quantize after upgrades — quantization formats evolve; regenerating from f16 costs minutes and avoids compatibility surprises.
4. Benchmark before optimizing — llama-bench takes seconds and prevents hours of tuning the wrong variable.
5. Run llama-server under systemd — it is a long-lived daemon; give it a unit, logging, and restart policy like any other service.
6. Watch VRAM, not just tokens/sec — use nvidia-smi or rocm-smi alongside your test to see how close to the edge you are running.
Frequently Asked Questions
Should I use llama.cpp or Ollama?
Ollama if you want models running in one command with automatic management. llama.cpp if you need a specific compute backend, want to quantize your own models, need llama-bench for measurements, or want to control the server yourself. Ollama is built on llama.cpp, so the inference engine is the same.
Do I need the CUDA toolkit or just the NVIDIA driver?
To build llama.cpp with CUDA you need the toolkit, because compilation requires nvcc. Running a prebuilt CUDA binary needs only the driver. This differs from PyTorch, whose pip wheels bundle the CUDA runtime and need only the driver.
What does the -ngl flag actually do?
It sets how many model layers are offloaded to the GPU. Layers that do not fit stay on the CPU, and the model is split across both. Setting -ngl 99 offloads everything; lowering it trades speed for less VRAM use.
Which quantization should I pick?
Q4_K_M is the common default — a large size reduction for a small quality cost. Use Q5_K_M or Q6_K when you have spare VRAM, and Q3_K_M only when you must fit a larger model into limited memory.
Can llama.cpp use an AMD or Intel GPU?
Yes. AMD works through the HIP/ROCm backend, Intel through SYCL, and both work through the vendor-neutral Vulkan backend. Vulkan is usually the quickest to get running because it needs only a working driver plus Vulkan headers.