Most published local-LLM benchmark numbers are not comparable to anything. They omit the quantization, the context length, the batch size, and whether the GPU was throttled — any one of which can move the result by more than the hardware difference being claimed.
Measuring properly is not hard. It requires knowing which four numbers matter and controlling the handful of variables that dominate them.
The Four Metrics
| Metric | What it measures | Matters for |
|---|---|---|
| TTFT | Time to first token | Interactive feel — this is what users notice |
| Prompt processing (pp) | Tokens/sec ingesting the prompt | Long prompts, RAG, document input |
| Token generation (tg) | Tokens/sec producing output | Perceived speed of a streaming reply |
| Throughput | Total tokens/sec across all requests | Servers with concurrent users |
Generation and prompt processing scale differently — prompt processing is compute-heavy and parallel; generation is sequential and memory-bandwidth bound. A change that doubles one can leave the other untouched — which is why a single "tokens/sec" figure is close to meaningless.
Latency and throughput trade off — batching raises total throughput while making each individual response slower. Optimizing one without naming the other is how vendors publish flattering numbers.
Establish a Clean Baseline First
Before measuring anything, confirm the machine is in a comparable state. Skipping this is the single biggest source of bad data:
# Is the GPU throttled? Any "Active: Yes" invalidates the run
$ nvidia-smi -q -d PERFORMANCE | grep -A12 "Clocks Event Reasons"
# Current vs max clocks, and power headroom
$ nvidia-smi --query-gpu=clocks.current.sm,clocks.max.sm,temperature.gpu,power.draw,power.limit --format=csv
# Is anything else on the GPU?
$ nvidia-smi
# Is anything else on the CPU?
$ uptime
Run the workload once to warm up and discard that result — the first run pays model load, allocator warm-up, and clock ramp. See our GPU monitoring guide for reading these numbers.
llama-bench: The Right Tool for llama.cpp
llama-bench exists precisely for this. It separates prompt processing from generation, repeats runs, and reports variation:
# Compare GPU offload levels on one model
$ ./build/bin/llama-bench -m models/model.gguf -ngl 0,20,99
# Compare thread counts on CPU
$ ./build/bin/llama-bench -m models/model.gguf -ngl 0 -t 4,8,16
# Compare quantizations of the same model
$ ./build/bin/llama-bench -m models/model-Q4_K_M.gguf -m models/model-Q8_0.gguf -ngl 99
# More repetitions for a stabler number
$ ./build/bin/llama-bench -m models/model.gguf -ngl 99 -r 10
The output splits pp and tg rows with a standard deviation per measurement. If the deviation is large, the number is not usable — something on the machine is interfering. Build details are in our llama.cpp guide.
Measuring a Server Endpoint
For Ollama, vLLM, or llama-server, measure the API as your application will use it. Most OpenAI-compatible servers return timing and token counts in the response:
# Ollama reports durations in nanoseconds when streaming is off
$ curl -s http://localhost:11434/api/generate -d '{
"model": "llama3.2",
"prompt": "Explain what cgroups do in three sentences.",
"stream": false
}' | jq '{
load_ms: (.load_duration/1e6),
prompt_tokens: .prompt_eval_count,
prompt_tps: (.prompt_eval_count / (.prompt_eval_duration/1e9)),
gen_tokens: .eval_count,
gen_tps: (.eval_count / (.eval_duration/1e9))
}'
Use the server's own counters — they exclude network and client overhead, so they isolate inference. Wall-clock from the client measures your whole stack — useful, but a different question.
Measure TTFT with streaming on — the time until the first chunk arrives is the number users feel. It is invisible when streaming is disabled.
Fix the output length — pin generated tokens (for example num_predict) so runs are comparable. Otherwise you are comparing different amounts of work.
Measuring Under Concurrency
Single-request numbers tell you nothing about a server with users. This is where vLLM normally pulls ahead of llama.cpp — see our server comparison:
#!/usr/bin/env bash
# Fire N concurrent requests and report wall-clock and per-request latency
set -euo pipefail
URL="http://localhost:8000/v1/chat/completions"
MODEL="my-model"
CONCURRENCY="${1:-8}"
payload='{"model":"'"$MODEL"'","max_tokens":128,"messages":[{"role":"user","content":"Write a short summary of what systemd does."}]}'
start=$(date +%s.%N)
for i in $(seq 1 "$CONCURRENCY"); do
( /usr/bin/time -f "%e" curl -s -o /dev/null -H "Content-Type: application/json" -d "$payload" "$URL" ) 2>> /tmp/latencies.txt &
done
wait
end=$(date +%s.%N)
echo "concurrency=$CONCURRENCY wall=$(echo "$end - $start" | bc)s"
sort -n /tmp/latencies.txt | awk '{a[NR]=$1} END {
printf "p50=%.2fs p95=%.2fs max=%.2fs
", a[int(NR*0.5)], a[int(NR*0.95)], a[NR]
}'
Sweep concurrency — 1, 2, 4, 8, 16 — and plot throughput against p95 latency. The point where latency starts climbing steeply while throughput flattens is your real capacity limit.
The Confounders
Quantization — the largest single factor. A Q4 and a Q8 version of the same model are different workloads. Always record it.
Context length — generation slows as context grows, because attention costs rise with sequence length. Benchmarking at 512 tokens says little about behaviour at 32K.
GPU/CPU split — a model partly on CPU performs completely differently. Verify with ollama ps or the llama.cpp load log.
Thermal state — a cold GPU beats the same GPU ten minutes into a sustained run. Report sustained numbers, not the first burst.
Power limits — laptops on battery, and servers with conservative power caps, run slower. Check power.limit.
Background load — another process on the GPU or a busy CPU distorts everything. Benchmark on an otherwise idle machine.
Cold cache — the first run includes loading multi-gigabyte weights from disk. Warm up and discard.
Batch size — raises throughput, lowers per-request speed. Never compare a batched number with an unbatched one.
Recording Results So They Stay Useful
A number without its conditions is not a measurement. Record all of this alongside every result:
# Capture the environment with the result
$ cat > run-metadata.txt <<'META'
model: Llama-3.2-3B-Instruct
quantization: Q4_K_M
runtime: llama.cpp (commit abc1234), CUDA backend
context: 4096
ngl: 99
concurrency: 1
META
$ nvidia-smi --query-gpu=name,driver_version,memory.total,power.limit --format=csv >> run-metadata.txt
$ uname -r >> run-metadata.txt
Pin the runtime version — llama.cpp, vLLM, and Ollama all change performance between releases. A result without a version cannot be reproduced.
Report a range, not one number — give median and spread across repeats. A single figure hides whether the measurement was stable.
Benchmark your own prompt shape — if your workload sends 4K-token prompts and gets 100-token answers, measure that — not a short question with a long essay reply.
Frequently Asked Questions
What is a good tokens per second for a local LLM?
There is no universal figure, because it depends on model size, quantization, context length, hardware, and whether requests are batched. The useful comparison is always against your own baseline on the same machine with one variable changed.
What is the difference between TTFT and tokens per second?
TTFT is the delay before the first token appears and is what makes an interface feel responsive. Tokens per second describes how fast the rest streams. They are affected by different things — TTFT is dominated by prompt processing, generation speed by memory bandwidth.
Why do my benchmark results vary between runs?
Common causes are thermal or power throttling, a cold first run that includes model loading, background load on the GPU or CPU, and unpinned output lengths. Warm up, discard the first result, and check llama-bench's standard deviation.
How do I benchmark an LLM server with multiple users?
Sweep concurrency levels and record both total throughput and p95 latency at each. Throughput alone is misleading, since batching raises it while making individual responses slower. The knee where latency climbs sharply is your practical capacity.
Does quantization make inference faster?
Usually yes, because token generation is largely limited by memory bandwidth and fewer bits per weight means fewer bytes read. The size of the gain depends on hardware and whether the model fits entirely in VRAM.