When an AI workload is slower than expected, the GPU tells you why — but only if you read the right numbers. The two most common mistakes are trusting the utilization percentage as a measure of efficiency, and never checking whether the card is thermally throttled.
This covers the tools per vendor, what each metric actually means, and how to log GPU data for later analysis.
The Tools
| Tool | Vendor | Strength |
|---|---|---|
nvidia-smi | NVIDIA | Ships with the driver; scriptable query mode |
nvtop | NVIDIA, AMD, Intel | htop-style interactive view, multi-GPU |
rocm-smi | AMD | Ships with ROCm; equivalent role to nvidia-smi |
intel_gpu_top | Intel | From intel-gpu-tools; engine-level breakdown |
DCGM exporter | NVIDIA | Prometheus metrics for servers and clusters |
Install nvtop first if you do nothing else — it works across vendors, shows history graphs, and lists the processes using each GPU in one screen.
nvidia-smi Beyond the Default Table
The plain command prints a snapshot. The useful modes are the ones people never reach for:
# Refresh every second
$ nvidia-smi -l 1
# Machine-readable — the right way to script it
$ nvidia-smi --query-gpu=timestamp,name,utilization.gpu,utilization.memory,memory.used,memory.total,temperature.gpu,power.draw \
--format=csv
# Per-process GPU memory
$ nvidia-smi pmon -c 1
# Device monitoring: SM, memory, encoder, power, clocks
$ nvidia-smi dmon
# Multi-GPU topology — how cards are interconnected
$ nvidia-smi topo -m
# Everything about one GPU, including throttle reasons
$ nvidia-smi -q -d PERFORMANCE
--query-gpu is the important one — it emits stable CSV you can log, graph, or alert on. Parsing the human-readable table with awk is a trap — the layout changes between driver versions.
What the Metrics Actually Mean
utilization.gpu — the percentage of time at least one kernel was running — not how much of the GPU's capability you used. A tiny kernel looping forever reports 100%. Treat it as "busy or idle", not as efficiency.
utilization.memory — percentage of time memory was being read or written, not how full VRAM is. People confuse this with memory.used constantly.
memory.used — the number that matters for fitting a model. Compare it against memory.total when sizing.
power.draw — the best rough proxy for real work being done. A GPU near its power limit is genuinely working.
temperature.gpu — matters because of what it causes — throttling. Check throttle reasons rather than guessing from the temperature.
clocks.current.sm — if this sits well below the maximum under load, you are throttled — thermally, by power limit, or by a driver cap.
For LLM inference specifically, generation is largely memory-bandwidth bound. You can be at 100% reported utilization with compute units mostly idle, waiting on memory — which is why quantization speeds things up even though it does not reduce the number of operations. See our quantization guide for that relationship.
Checking for Throttling
This is the check that explains most "it got slower over time" reports — especially on laptops and small-form desktops:
# Throttle reasons — look for Active: Yes
$ nvidia-smi -q -d PERFORMANCE | grep -A12 "Clocks Event Reasons"
# Current vs maximum clocks
$ nvidia-smi --query-gpu=clocks.current.sm,clocks.max.sm,temperature.gpu,power.draw,power.limit --format=csv
# AMD equivalent
$ rocm-smi --showtemp --showpower --showclocks
Any benchmark taken while throttled is meaningless. Establish that clocks are stable before comparing hardware or configurations — see our benchmarking guide.
AMD and Intel
# --- AMD (ROCm) ---
$ rocm-smi # overview
$ rocm-smi --showmeminfo vram # VRAM detail
$ rocm-smi --showuse # utilization
$ watch -n1 rocm-smi
# --- Intel ---
$ sudo apt install intel-gpu-tools
$ sudo intel_gpu_top # per-engine utilization
intel_gpu_top splits activity by engine (render, compute, video), which is more informative than a single percentage — you can see immediately whether your workload is landing on the compute engine at all. Setup for both vendors is covered in our ROCm and Intel guides.
Logging GPU Data Over Time
Spot checks miss the interesting moments. Log to CSV during a long run and look afterwards:
# Sample every 5 seconds into a CSV
$ nvidia-smi --query-gpu=timestamp,utilization.gpu,memory.used,temperature.gpu,power.draw,clocks.current.sm \
--format=csv -l 5 > gpu-log.csv
# Peak VRAM used during the run
$ awk -F', ' 'NR>1 {gsub(/ MiB/,"",$3); if ($3+0>m) m=$3+0} END {print "peak VRAM:", m, "MiB"}' gpu-log.csv
Peak VRAM is the figure to capture — it tells you how close a workload came to OOM, which an average never reveals. Run it as a background job alongside your training or inference test.
Server and Cluster Monitoring
For anything beyond one machine, export metrics instead of reading terminals. NVIDIA's DCGM exporter publishes Prometheus metrics — utilization, memory, temperature, power, and errors — per GPU:
# Run the exporter as a container on each GPU node
$ docker run -d --gpus all --rm -p 9400:9400 \
nvcr.io/nvidia/k8s/dcgm-exporter:latest
$ curl -s localhost:9400/metrics | grep DCGM_FI_DEV_GPU_UTIL
Alert on VRAM headroom — not on utilization. High utilization is normal for an inference server; low free memory is what precedes failures.
Alert on ECC and XID errors — these indicate real hardware trouble and are far more actionable than a utilization spike.
Persistence mode on servers — nvidia-smi -pm 1 keeps the driver loaded so the first request after an idle period does not pay initialization cost.
Troubleshooting
No processes listed, but memory is used — common in containers — the host cannot always see PIDs inside a container's namespace. The memory figure is still accurate.
nvidia-smi works on host, not in container — the NVIDIA Container Toolkit is missing or the runtime is not configured. See our GPU containers guide.
Utilization 100% but work is slow — you are likely memory-bandwidth bound, or running a small kernel in a tight loop. Check power draw: low power with high utilization means waiting, not computing.
VRAM stays allocated after a job ends — a process still holds a context. Find it with nvidia-smi pmon or fuser on the device node.
Numbers differ from what the framework reports — frameworks report their own allocator's view; nvidia-smi reports what the driver has reserved. Caching allocators hold memory they are not actively using.
rocm-smi shows no GPU — a permissions or support problem rather than a monitoring one — check render/video group membership and rocminfo.
Frequently Asked Questions
Does 100% GPU utilization mean the GPU is fully used?
No. utilization.gpu reports the fraction of time at least one kernel was running, not how much of the GPU's capability was used. A small kernel in a tight loop shows 100%. Power draw and clock speeds are better indicators of real work.
How do I see which process is using GPU memory?
Run nvidia-smi pmon -c 1, or read the process table at the bottom of plain nvidia-smi output. Inside containers the host may not resolve PIDs, but the memory totals remain correct.
How can I tell if my GPU is throttling?
Run nvidia-smi -q -d PERFORMANCE and look for throttle reasons marked Active: Yes, or compare clocks.current.sm against clocks.max.sm under load. Any benchmark taken while throttled is not comparable.
What is the best GPU monitor for AMD and Intel on Linux?
rocm-smi for AMD and intel_gpu_top for Intel, both installed with their respective stacks. nvtop is the best cross-vendor interactive option and covers NVIDIA, AMD, and Intel in one htop-style view.
How should I log GPU metrics during a long run?
Use nvidia-smi --query-gpu with --format=csv and -l to sample on an interval, redirecting to a file. That output is stable across driver versions, unlike the human-readable table, and peak VRAM is the most useful figure to extract from it.