These four projects all "serve an LLM", but they are built for different situations. Picking wrong wastes either hardware or weeks: vLLM on an 8 GB consumer card will simply refuse to load a model that llama.cpp runs happily, and llama.cpp serving fifty concurrent users will crawl where vLLM would be idle.
The deciding question is almost always the same: how many concurrent requests do you have, and does the model fit entirely in VRAM?
The Comparison
| vLLM | llama.cpp | TGI | Ollama | |
|---|---|---|---|---|
| Primary target | GPU servers | Anything | GPU servers | Workstations |
| Model must fit VRAM | Effectively yes | No — CPU/GPU split | Effectively yes | No |
| Concurrency | Excellent | Limited | Excellent | Limited |
| Weights format | HF safetensors, AWQ, GPTQ | GGUF | HF safetensors, AWQ, GPTQ | GGUF |
| CPU-only | Not practical | Yes | No | Yes |
| Setup effort | Moderate | Build from source | Container | One command |
What Makes vLLM Fast
Two mechanisms, and understanding them tells you when vLLM helps and when it does not:
PagedAttention — manages the KV cache in fixed-size pages like virtual memory instead of one contiguous block per request. That removes the memory fragmentation which otherwise limits how many requests fit at once.
Continuous batching — new requests join the running batch as soon as slots free, rather than waiting for the whole batch to finish. The GPU stays saturated instead of idling between batches.
Both are throughput optimizations for concurrent load. For a single user asking one question at a time, they buy you very little — which is exactly why a workstation user often sees no benefit from vLLM at all.
Running Each One
vLLM — installs as a Python package and ships an OpenAI-compatible server:
$ python3 -m venv .venv && source .venv/bin/activate
$ pip install vllm
$ vllm serve meta-llama/Llama-3.1-8B-Instruct \
--host 127.0.0.1 --port 8000 \
--max-model-len 8192 \
--gpu-memory-utilization 0.90
$ curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
"model": "meta-llama/Llama-3.1-8B-Instruct",
"messages": [{"role": "user", "content": "Explain cgroups briefly"}]
}'
llama.cpp — build it, then run llama-server (full details in our build guide):
$ ./build/bin/llama-server -m models/model.gguf -ngl 99 -c 8192 \
--host 127.0.0.1 --port 8080
# Same OpenAI-compatible path
# http://localhost:8080/v1/chat/completions
TGI — Hugging Face's server, normally run as a container:
$ docker run --gpus all --shm-size 1g -p 8080:80 \
-v ~/models:/data \
ghcr.io/huggingface/text-generation-inference:latest \
--model-id meta-llama/Llama-3.1-8B-Instruct
Ollama — one command, GGUF, automatic model management. See our Ollama guide.
The Decision Guide
| Your situation | Use |
|---|---|
| One person, consumer GPU or no GPU | Ollama, or llama.cpp for more control |
| Model bigger than your VRAM | llama.cpp (CPU/GPU split) — the others cannot |
| Serving an app with real concurrent users | vLLM |
| Already on the Hugging Face stack | TGI |
| CPU-only server | llama.cpp |
| Kubernetes deployment | vLLM or TGI — both containerize cleanly |
| Raspberry Pi or ARM SBC | llama.cpp |
A pattern worth stealing: run Ollama on the workstation for development and vLLM on the shared GPU box for anything user-facing. Both expose the same OpenAI-compatible API, so application code does not change between them.
Memory Planning
The mistake that wastes the most time is assuming weights are the whole cost. You also pay for the KV cache, which grows with context length and with the number of concurrent requests:
Weights — roughly parameters × bits-per-weight ÷ 8. A 7B model at 4-bit is a few gigabytes; at 16-bit it is far larger.
KV cache — scales with context length × concurrent sequences. This is why a server that works for one user OOMs at ten.
Overhead — activations and framework overhead. Never plan to use 100% of VRAM.
vLLM's knob — --gpu-memory-utilization reserves a fraction of VRAM up front. Lower it if other processes share the GPU, raise it on a dedicated box.
Operating Them in Production
Bind to localhost, proxy in front — none of these ship meaningful authentication. Put nginx or a gateway in front and require a credential — see our nginx config generator.
Run under systemd — they are long-lived daemons. Give them a unit with Restart=on-failure and journald logging like any service.
Set explicit context limits — an unbounded max context lets one request consume the memory of many.
Monitor GPU, not just CPU — export nvidia-smi or DCGM metrics; queue depth and VRAM headroom are the signals that matter.
Pin versions — these projects move quickly and performance characteristics change between releases. Record what you deployed.
Measure with your own traffic — published throughput numbers assume a batch profile that is probably not yours. Load test with realistic prompt and output lengths.
Frequently Asked Questions
Is vLLM faster than llama.cpp?
For many concurrent requests on a GPU where the model fits in VRAM, yes — substantially, due to PagedAttention and continuous batching. For a single user, or when the model does not fit in VRAM, llama.cpp is often the better or only option since it can split layers between GPU and CPU.
Can vLLM run on CPU or split a model across CPU and GPU?
Not practically. vLLM is designed around GPU memory management and expects the model to fit in VRAM. If your model is larger than your VRAM, or you have no GPU at all, llama.cpp is the tool that handles that case.
What is the difference between Ollama and llama.cpp?
Ollama is a management layer built on llama.cpp. It adds model downloading, storage, a daemon, and a simple CLI. The inference engine underneath is the same, so choose Ollama for convenience and llama.cpp when you need specific backends, custom quantization, or benchmarking.
Do these servers work with OpenAI client libraries?
Yes. vLLM, llama-server, TGI, and Ollama all expose OpenAI-compatible chat completion endpoints, so most OpenAI SDK clients work by changing the base URL and passing a placeholder API key.
Why does my server run out of memory only under load?
The KV cache scales with both context length and the number of concurrent sequences. Weights are a fixed cost, but each additional simultaneous request adds cache. Cap the maximum context length and limit concurrency to make memory use predictable.