Most broken NVIDIA installations on Linux come from one mistake: mixing installation methods. Someone installs the distro driver, then runs NVIDIA's .run installer, then adds the CUDA repo — and now three package sources each believe they own libcuda.so.
This guide covers what you actually need, how to install it cleanly per distro, how Secure Boot fits in, and how to fix the errors you will hit.
Driver, Toolkit, or Container Runtime?
These are three separate things, and most people need fewer of them than they install:
| Component | What it gives you | Do you need it? |
|---|---|---|
| NVIDIA driver | Kernel module + nvidia-smi + the CUDA driver API | Always |
| CUDA toolkit | Compiler (nvcc), headers, libraries, profilers | Only to compile CUDA code |
| Container toolkit | Exposes the GPU to Docker/Podman containers | Only if you run GPU containers |
The big shortcut — if you only want to run PyTorch, TensorFlow, Ollama, or a prebuilt inference server, you need the driver only. Those pip wheels and binaries ship their own CUDA runtime. Skip the toolkit entirely — it is several gigabytes and a common source of version confusion.
Installing the Driver
Use your distribution's packaging. It integrates with your kernel updates, Secure Boot handling, and package manager — the .run installer does none of that.
# --- Ubuntu / Pop!_OS ---
$ ubuntu-drivers devices # shows your GPU and recommended driver
$ sudo ubuntu-drivers install # installs the recommended version
$ sudo reboot
# --- Debian (enable non-free-firmware + contrib first) ---
$ sudo apt install nvidia-driver firmware-misc-nonfree
$ sudo reboot
# --- Fedora (requires RPM Fusion nonfree) ---
$ sudo dnf install akmod-nvidia
$ sudo reboot
# --- Arch ---
$ sudo pacman -S nvidia-dkms nvidia-utils
$ sudo reboot
A reboot is genuinely required — the old kernel module has to be unloaded and the new one loaded, and the display stack usually holds it open. Verify afterwards:
$ nvidia-smi
# Should print a table with driver version, CUDA version, and your GPU
Reading nvidia-smi Correctly
The CUDA Version in the top-right of nvidia-smi output confuses almost everyone. It is the newest CUDA version this driver can support — not what you have installed, and not what your application uses.
So nvidia-smi can say CUDA 12.x while nvcc --version says something older, and both are correct. PyTorch reports its own bundled version separately:
$ nvidia-smi # max CUDA the DRIVER supports
$ nvcc --version # the TOOLKIT you installed (if any)
$ python -c "import torch; print(torch.version.cuda, torch.cuda.is_available())"
Secure Boot and DKMS
This is the number one cause of "it worked yesterday". With Secure Boot enabled, the kernel refuses to load unsigned out-of-tree modules — and the NVIDIA module is out-of-tree.
# Is Secure Boot on?
$ mokutil --sb-state
# Is the module built for the running kernel?
$ dkms status
# Did the kernel refuse to load it?
$ sudo dmesg | grep -i -E "nvidia|taint|signature"
Ubuntu/Debian — the driver package prompts you to set a one-time password and enroll a Machine Owner Key (MOK) on the next boot. If you skipped that blue screen, the module will never load. Re-run enrollment with mokutil --import.
Fedora — akmods can sign modules with a local key, but it must be enrolled in MOK as well.
Simplest option — on a dedicated workstation, disabling Secure Boot in firmware is a legitimate choice. On a managed or compliance-bound machine, enroll the key instead.
Installing the CUDA Toolkit (Only If You Need It)
If you are compiling CUDA code — building llama.cpp with the CUDA backend, custom kernels, or older projects that need nvcc — install the toolkit from NVIDIA's repository.
# Add NVIDIA's repo keyring package for your distro/version
# (download the cuda-keyring .deb or .rpm from NVIDIA's site)
$ sudo dpkg -i cuda-keyring_<version>_all.deb
$ sudo apt update
# Install the TOOLKIT ONLY — this does not touch your driver
$ sudo apt install cuda-toolkit
# NOTE: the 'cuda' metapackage ALSO installs a driver.
# That is how people end up with two competing drivers.
That distinction is the single most valuable thing in this section: install cuda-toolkit, not cuda, unless you deliberately want NVIDIA's driver to replace your distro's.
Then put the toolkit on your PATH:
# ~/.bashrc
export PATH=/usr/local/cuda/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH
$ source ~/.bashrc && nvcc --version
GPU Access Inside Containers
Containers do not need their own driver — they use the host's. The NVIDIA Container Toolkit wires the device nodes and libraries into the container at runtime:
# Install nvidia-container-toolkit from NVIDIA's repo, then:
$ sudo nvidia-ctk runtime configure --runtime=docker
$ sudo systemctl restart docker
# Verify — this should print the same table as the host
$ docker run --rm --gpus all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi
This is a genuinely good pattern for AI work: the host carries only the driver, and every CUDA version lives inside an image. See our Docker guide for the surrounding workflow.
Troubleshooting
"NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver" — the kernel module is not loaded. Almost always: you have not rebooted, the DKMS build failed for the running kernel, or Secure Boot blocked an unsigned module. Check dkms status and dmesg.
Works, then breaks after an update — a new kernel was installed and the module did not rebuild. Boot the previous kernel from the GRUB menu, fix DKMS, then update again.
nouveau is still loaded — the proprietary driver cannot bind while nouveau holds the card. Distro packages blacklist it automatically; confirm with lsmod | grep nouveau and rebuild the initramfs if it persists.
torch.cuda.is_available() returns False — check nvidia-smi first. If that works, you likely installed a CPU-only PyTorch build — reinstall from the CUDA wheel index.
"no space left on device" during install — CUDA toolkits are multi-gigabyte. Check /usr and /var free space before installing.
Two drivers installed — if you ran the .run installer over distro packages, use its --uninstall option, then reinstall cleanly from packages. Do not try to make them coexist.
Laptop: external monitor or GPU not used — hybrid graphics needs PRIME configuration. Confirm which GPU is active with nvidia-smi while a load is running.
Best Practices
1. One installation method — distro packages OR NVIDIA's repo. Never the .run installer alongside either.
2. Install the driver, skip the toolkit — unless you compile CUDA code. It removes an entire class of version mismatch.
3. Keep a fallback kernel — never remove the previous kernel until the new one has booted with a working GPU.
4. Handle Secure Boot deliberately — enroll a MOK key or disable Secure Boot — decide once, and document which you chose.
5. Pin driver versions on servers — hold the package so unattended upgrades cannot swap your driver at 3 AM.
6. Verify after every kernel update — a one-line nvidia-smi check in your post-update routine catches breakage before your workload does.
Frequently Asked Questions
Do I need the CUDA toolkit to run PyTorch or Ollama?
No. PyTorch pip wheels and tools like Ollama bundle their own CUDA runtime, so a working NVIDIA driver is sufficient. You need the toolkit only to compile CUDA code yourself, such as building llama.cpp with the CUDA backend.
Why does nvidia-smi show a different CUDA version than nvcc?
They report different things. nvidia-smi shows the highest CUDA version the installed driver supports, while nvcc reports the toolkit version you installed. It is normal and expected for them to differ.
Why did nvidia-smi stop working after a kernel update?
The kernel module did not rebuild for the new kernel, or Secure Boot blocked the unsigned module. Check dkms status and dmesg for signature errors, boot the previous kernel from GRUB if needed, and re-enroll your MOK key if Secure Boot is enabled.
Should I use the .run installer from NVIDIA's website?
Generally no. It does not integrate with your package manager or kernel updates and is the most common cause of unrepairable driver setups. Use your distro's packages or NVIDIA's apt/dnf repository instead.
How do I give Docker containers GPU access?
Install the NVIDIA Container Toolkit on the host, run nvidia-ctk runtime configure --runtime=docker, restart Docker, then pass --gpus all when running a container. The container uses the host driver; only the CUDA runtime lives in the image.