Containers are the cleanest way to manage AI dependencies on Linux: the host carries only the GPU driver, and every CUDA version, Python version, and framework lives inside an image. No more LD_LIBRARY_PATH archaeology, and no more two projects fighting over one CUDA installation.
The mechanism that makes it work is the NVIDIA Container Toolkit, which injects the host driver's libraries and device nodes into the container at runtime.
The Model: Driver Out, Runtime In
| Lives on the host | Lives in the image |
|---|---|
| NVIDIA kernel driver | CUDA runtime and libraries |
nvidia-smi | cuDNN, framework builds |
| Container toolkit | Python and your application |
One consequence matters most: the image's CUDA version must be supported by the host driver, not the other way around. A newer driver runs older CUDA images fine; an old driver cannot run a newer CUDA image. That single rule explains most container GPU failures.
Installing the Toolkit
You need a working host driver first — verify with nvidia-smi before touching containers (see our driver guide). Then:
# Add NVIDIA's container-toolkit repository for your distro, then:
$ sudo apt install -y nvidia-container-toolkit # Debian/Ubuntu
# sudo dnf install -y nvidia-container-toolkit # Fedora/RHEL
# Wire it into Docker and restart the daemon
$ sudo nvidia-ctk runtime configure --runtime=docker
$ sudo systemctl restart docker
# The canonical smoke test
$ docker run --rm --gpus all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi
If that last command prints the same table as the host, you are done — everything else is usage.
Selecting GPUs
# All GPUs
$ docker run --rm --gpus all <image>
# One GPU by index
$ docker run --rm --gpus '"device=0"' <image>
# Specific GPUs
$ docker run --rm --gpus '"device=0,2"' <image>
# By UUID (stable across reboots, unlike indexes)
$ docker run --rm --gpus '"device=GPU-abc123..."' <image>
# A count, letting the runtime choose
$ docker run --rm --gpus 2 <image>
Mind the quoting — the '"device=0"' form needs both quote types. It is the most common syntax error here.
Prefer UUIDs on multi-GPU servers — indexes can be reordered by the driver. Get UUIDs from nvidia-smi -L.
Choosing a CUDA Base Image
| Tag suffix | Contains | Use for |
|---|---|---|
-base | Minimal CUDA runtime | Smallest; when your app bundles its own libraries |
-runtime | CUDA runtime + math libraries | Running most frameworks |
-devel | Adds nvcc and headers | Compiling CUDA code |
Build with -devel and ship with -runtime using a multi-stage Dockerfile — the compiler does not belong in a production image:
FROM nvidia/cuda:12.4.0-devel-ubuntu22.04 AS build
WORKDIR /src
COPY . .
RUN make
FROM nvidia/cuda:12.4.0-runtime-ubuntu22.04
COPY --from=build /src/out/app /usr/local/bin/app
ENTRYPOINT ["app"]
PyTorch images already include CUDA — if you start FROM pytorch/pytorch:... you do not need a CUDA base image — the runtime is already inside. Stacking both wastes gigabytes.
Docker Compose
Compose uses the device reservation syntax rather than a --gpus flag:
services:
inference:
image: my-inference:latest
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all # or: count: 1
capabilities: [gpu]
volumes:
- ./models:/models
ports:
- "127.0.0.1:8000:8000"
# Pin specific GPUs instead of a count
- driver: nvidia
device_ids: ["0", "2"]
capabilities: [gpu]
Note the port binding to 127.0.0.1 — inference servers generally have no authentication, so keep them off the network and proxy in front, as covered in our LLM server comparison.
Running Ollama in a Container
A practical example that keeps models on a host volume so they survive container replacement:
$ docker run -d --name ollama --gpus all \
-v ollama-models:/root/.ollama \
-p 127.0.0.1:11434:11434 \
--restart unless-stopped \
ollama/ollama
$ docker exec -it ollama ollama run llama3.2
$ docker exec -it ollama ollama ps # confirm GPU is being used
Compare with the native install in our Ollama guide — the container route is better when you want the host clean, the native route when you want systemd integration.
Podman and Rootless Docker
Podman uses CDI (Container Device Interface) rather than a custom runtime:
# Generate the CDI spec (regenerate after driver updates)
$ sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml
$ nvidia-ctk cdi list
$ podman run --rm --device nvidia.com/gpu=all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi
Rootless needs a config change — set no-cgroups = true in /etc/nvidia-container-runtime/config.toml for rootless operation, since an unprivileged user cannot manage device cgroups.
Regenerate CDI after driver upgrades — the spec references specific driver library paths. A stale spec is a common post-upgrade failure.
Troubleshooting
"could not select device driver \"\" with capabilities: [[gpu]]" — the container toolkit is not installed or Docker was not reconfigured. Run nvidia-ctk runtime configure --runtime=docker and restart Docker.
nvidia-smi works on host but not in container — same cause as above, or you forgot --gpus. The flag is not implied.
"CUDA driver version is insufficient for CUDA runtime version" — the image needs a newer driver than the host has. Use an older CUDA image tag or upgrade the host driver.
Works as root, fails rootless — set no-cgroups = true in the container runtime config.
GPU disappears after a driver upgrade — restart Docker, and for Podman regenerate the CDI spec. The injected library paths changed.
torch.cuda.is_available() is False inside the container — check nvidia-smi inside the container first. If that works, you have a CPU-only PyTorch build in the image.
Out of shared memory during training — PyTorch data loaders need more than Docker's default 64 MB — pass --shm-size 1g or larger.
Best Practices
1. Keep the host minimal — driver plus container toolkit only. No system CUDA toolkit to drift out of sync.
2. Pin image tags — never deploy :latest for CUDA images — a tag move can change the required driver version.
3. Mount models as volumes — model weights are huge and should never be baked into images.
4. Multi-stage for compiled code — build with -devel, ship with -runtime.
5. Bind inference ports to localhost — then reverse-proxy with authentication.
6. Verify after every driver upgrade — a one-line container nvidia-smi check catches breakage immediately.
7. Set --shm-size for training — the default is too small for most data loader configurations.
Frequently Asked Questions
Do I need the CUDA toolkit on the host to run GPU containers?
No. Only the NVIDIA driver and the container toolkit are needed on the host. The CUDA runtime and libraries come from the image, which is the main advantage of this approach.
What causes "could not select device driver with capabilities gpu"?
The NVIDIA Container Toolkit is not installed, or Docker has not been configured to use it. Install it, run sudo nvidia-ctk runtime configure --runtime=docker, then restart the Docker daemon.
How do I give a container access to only one GPU?
Use --gpus '"device=0"' with both sets of quotes, or device_ids in Docker Compose. On multi-GPU servers prefer GPU UUIDs from nvidia-smi -L, since indexes can be reordered by the driver.
Can the container use a newer CUDA version than the host driver supports?
No. The host driver sets the maximum supported CUDA version. A newer driver runs older CUDA images fine, but an older driver cannot run a newer CUDA image — that mismatch produces the "driver version is insufficient" error.
How do GPUs work with Podman instead of Docker?
Podman uses CDI. Generate a spec with sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml, then run with --device nvidia.com/gpu=all. Regenerate the spec after driver upgrades, and set no-cgroups = true for rootless use.