Skip to content

Benchmarking LLM Inference on Linux: Measuring Tokens per Second Properly

Measure local LLM performance you can trust — TTFT vs tokens/sec, llama-bench, throughput under concurrency, and the confounders that make most published numbers useless.

11 min read

Most published local-LLM benchmark numbers are not comparable to anything. They omit the quantization, the context length, the batch size, and whether the GPU was throttled — any one of which can move the result by more than the hardware difference being claimed.

Measuring properly is not hard. It requires knowing which four numbers matter and controlling the handful of variables that dominate them.

The Four Metrics

MetricWhat it measuresMatters for
TTFTTime to first tokenInteractive feel — this is what users notice
Prompt processing (pp)Tokens/sec ingesting the promptLong prompts, RAG, document input
Token generation (tg)Tokens/sec producing outputPerceived speed of a streaming reply
ThroughputTotal tokens/sec across all requestsServers with concurrent users

Generation and prompt processing scale differentlyprompt processing is compute-heavy and parallel; generation is sequential and memory-bandwidth bound. A change that doubles one can leave the other untouched — which is why a single "tokens/sec" figure is close to meaningless.

Latency and throughput trade offbatching raises total throughput while making each individual response slower. Optimizing one without naming the other is how vendors publish flattering numbers.

Establish a Clean Baseline First

Before measuring anything, confirm the machine is in a comparable state. Skipping this is the single biggest source of bad data:

# Is the GPU throttled? Any "Active: Yes" invalidates the run

$ nvidia-smi -q -d PERFORMANCE | grep -A12 "Clocks Event Reasons"

# Current vs max clocks, and power headroom

$ nvidia-smi --query-gpu=clocks.current.sm,clocks.max.sm,temperature.gpu,power.draw,power.limit --format=csv

# Is anything else on the GPU?

$ nvidia-smi

# Is anything else on the CPU?

$ uptime

Run the workload once to warm up and discard that result — the first run pays model load, allocator warm-up, and clock ramp. See our GPU monitoring guide for reading these numbers.

llama-bench: The Right Tool for llama.cpp

llama-bench exists precisely for this. It separates prompt processing from generation, repeats runs, and reports variation:

# Compare GPU offload levels on one model

$ ./build/bin/llama-bench -m models/model.gguf -ngl 0,20,99

# Compare thread counts on CPU

$ ./build/bin/llama-bench -m models/model.gguf -ngl 0 -t 4,8,16

# Compare quantizations of the same model

$ ./build/bin/llama-bench -m models/model-Q4_K_M.gguf -m models/model-Q8_0.gguf -ngl 99

# More repetitions for a stabler number

$ ./build/bin/llama-bench -m models/model.gguf -ngl 99 -r 10

The output splits pp and tg rows with a standard deviation per measurement. If the deviation is large, the number is not usable — something on the machine is interfering. Build details are in our llama.cpp guide.

Measuring a Server Endpoint

For Ollama, vLLM, or llama-server, measure the API as your application will use it. Most OpenAI-compatible servers return timing and token counts in the response:

# Ollama reports durations in nanoseconds when streaming is off

$ curl -s http://localhost:11434/api/generate -d '{

"model": "llama3.2",

"prompt": "Explain what cgroups do in three sentences.",

"stream": false

}' | jq '{

load_ms: (.load_duration/1e6),

prompt_tokens: .prompt_eval_count,

prompt_tps: (.prompt_eval_count / (.prompt_eval_duration/1e9)),

gen_tokens: .eval_count,

gen_tps: (.eval_count / (.eval_duration/1e9))

}'

Use the server's own countersthey exclude network and client overhead, so they isolate inference. Wall-clock from the client measures your whole stack — useful, but a different question.

Measure TTFT with streaming onthe time until the first chunk arrives is the number users feel. It is invisible when streaming is disabled.

Fix the output lengthpin generated tokens (for example num_predict) so runs are comparable. Otherwise you are comparing different amounts of work.

sponsored

Measuring Under Concurrency

Single-request numbers tell you nothing about a server with users. This is where vLLM normally pulls ahead of llama.cpp — see our server comparison:

#!/usr/bin/env bash

# Fire N concurrent requests and report wall-clock and per-request latency

set -euo pipefail

URL="http://localhost:8000/v1/chat/completions"

MODEL="my-model"

CONCURRENCY="${1:-8}"

payload='{"model":"'"$MODEL"'","max_tokens":128,"messages":[{"role":"user","content":"Write a short summary of what systemd does."}]}'

start=$(date +%s.%N)

for i in $(seq 1 "$CONCURRENCY"); do

( /usr/bin/time -f "%e" curl -s -o /dev/null -H "Content-Type: application/json" -d "$payload" "$URL" ) 2>> /tmp/latencies.txt &

done

wait

end=$(date +%s.%N)

echo "concurrency=$CONCURRENCY wall=$(echo "$end - $start" | bc)s"

sort -n /tmp/latencies.txt | awk '{a[NR]=$1} END {

printf "p50=%.2fs p95=%.2fs max=%.2fs

", a[int(NR*0.5)], a[int(NR*0.95)], a[NR]

}'

Sweep concurrency — 1, 2, 4, 8, 16 — and plot throughput against p95 latency. The point where latency starts climbing steeply while throughput flattens is your real capacity limit.

The Confounders

Quantizationthe largest single factor. A Q4 and a Q8 version of the same model are different workloads. Always record it.

Context lengthgeneration slows as context grows, because attention costs rise with sequence length. Benchmarking at 512 tokens says little about behaviour at 32K.

GPU/CPU splita model partly on CPU performs completely differently. Verify with ollama ps or the llama.cpp load log.

Thermal statea cold GPU beats the same GPU ten minutes into a sustained run. Report sustained numbers, not the first burst.

Power limitslaptops on battery, and servers with conservative power caps, run slower. Check power.limit.

Background loadanother process on the GPU or a busy CPU distorts everything. Benchmark on an otherwise idle machine.

Cold cachethe first run includes loading multi-gigabyte weights from disk. Warm up and discard.

Batch sizeraises throughput, lowers per-request speed. Never compare a batched number with an unbatched one.

Recording Results So They Stay Useful

A number without its conditions is not a measurement. Record all of this alongside every result:

# Capture the environment with the result

$ cat > run-metadata.txt <<'META'

model: Llama-3.2-3B-Instruct

quantization: Q4_K_M

runtime: llama.cpp (commit abc1234), CUDA backend

context: 4096

ngl: 99

concurrency: 1

META

$ nvidia-smi --query-gpu=name,driver_version,memory.total,power.limit --format=csv >> run-metadata.txt

$ uname -r >> run-metadata.txt

Pin the runtime versionllama.cpp, vLLM, and Ollama all change performance between releases. A result without a version cannot be reproduced.

Report a range, not one numbergive median and spread across repeats. A single figure hides whether the measurement was stable.

Benchmark your own prompt shapeif your workload sends 4K-token prompts and gets 100-token answers, measure that — not a short question with a long essay reply.

Frequently Asked Questions

What is a good tokens per second for a local LLM?

There is no universal figure, because it depends on model size, quantization, context length, hardware, and whether requests are batched. The useful comparison is always against your own baseline on the same machine with one variable changed.

What is the difference between TTFT and tokens per second?

TTFT is the delay before the first token appears and is what makes an interface feel responsive. Tokens per second describes how fast the rest streams. They are affected by different things — TTFT is dominated by prompt processing, generation speed by memory bandwidth.

Why do my benchmark results vary between runs?

Common causes are thermal or power throttling, a cold first run that includes model loading, background load on the GPU or CPU, and unpinned output lengths. Warm up, discard the first result, and check llama-bench's standard deviation.

How do I benchmark an LLM server with multiple users?

Sweep concurrency levels and record both total throughput and p95 latency at each. Throughput alone is misleading, since batching raises it while making individual responses slower. The knee where latency climbs sharply is your practical capacity.

Does quantization make inference faster?

Usually yes, because token generation is largely limited by memory bandwidth and fewer bits per weight means fewer bytes read. The size of the gain depends on hardware and whether the model fits entirely in VRAM.

sponsored

Related Tools