Kubernetes does not understand GPUs natively. It learns about them through a device plugin that advertises them as an extended resource — and extended resources behave differently from CPU and memory in ways that surprise people: they cannot be overcommitted, they cannot be fractional, and requests must equal limits.
Get that model right and GPU scheduling is straightforward. Get it wrong and you spend a day staring at a Pending pod.
Node Prerequisites
Every GPU node needs three things before Kubernetes can schedule to it:
| Layer | Component | Verify with |
|---|---|---|
| Driver | NVIDIA kernel driver | nvidia-smi |
| Container runtime | NVIDIA Container Toolkit | A GPU docker run/nerdctl test |
| Kubernetes | Device plugin or GPU Operator | kubectl describe node |
The runtime layer is the one most often missed — see our GPU containers guide. If a plain container cannot see the GPU on that node, Kubernetes never will.
Device Plugin or GPU Operator?
NVIDIA device plugin — a DaemonSet that discovers GPUs and advertises them. You install and maintain drivers and the container toolkit on the nodes yourself. Right when nodes are already provisioned by your own tooling.
GPU Operator — manages the whole stack — drivers, container toolkit, device plugin, and monitoring — as Kubernetes resources. Right for clusters that autoscale or where you do not want to bake drivers into node images.
# Confirm GPUs are advertised on a node
$ kubectl describe node <node> | grep -A6 "Allocatable"
# Look for: nvidia.com/gpu: 4
# Cluster-wide view
$ kubectl get nodes -o custom-columns=\
NAME:.metadata.name,GPU:.status.allocatable.'nvidia\.com/gpu'
If nvidia.com/gpu does not appear under Allocatable, no manifest will help — the problem is on the node.
Requesting a GPU
apiVersion: v1
kind: Pod
metadata:
name: inference
spec:
restartPolicy: Never
containers:
- name: server
image: my-registry/inference:1.4.2
resources:
limits:
nvidia.com/gpu: 1 # requests are set to match automatically
memory: 16Gi
cpu: "4"
requests:
memory: 16Gi
cpu: "4"
Only specify the limit — for extended resources, Kubernetes sets requests equal to limits. Specifying a different request is rejected.
You cannot ask for 0.5 GPUs — the value is an integer. Sharing requires time-slicing or MIG, covered below.
Set CPU and memory too — a GPU pod with no memory limit can still be evicted or starve its neighbours. The GPU is not the only resource it uses.
Keeping Non-GPU Work Off GPU Nodes
GPU nodes are expensive, and by default any pod can land on them. Taint the nodes so only workloads that explicitly tolerate the taint are scheduled there:
$ kubectl taint nodes gpu-node-1 nvidia.com/gpu=present:NoSchedule
$ kubectl label nodes gpu-node-1 accelerator=nvidia-a100
spec:
nodeSelector:
accelerator: nvidia-a100
tolerations:
- key: nvidia.com/gpu
operator: Exists
effect: NoSchedule
Label by GPU model, not just "has a GPU". In a mixed cluster a job that needs 80 GB of VRAM must not land on a 24 GB card, and only a label makes that expressible.
Sharing One GPU Between Pods
Inference workloads often use a fraction of a card, and one-pod-per-GPU wastes money. Three mechanisms, with real tradeoffs:
| Mechanism | Isolation | Use when |
|---|---|---|
| Time-slicing | None — pods share memory and compute | Dev/test, bursty low-load inference |
| MPS | Limited | Cooperative workloads from one tenant |
| MIG | Hardware-partitioned | Production multi-tenant on supported datacenter GPUs |
# Time-slicing via the device plugin config: advertise 4 "GPUs" per physical card
version: v1
sharing:
timeSlicing:
resources:
- name: nvidia.com/gpu
replicas: 4
Time-slicing does not partition memory — four pods on one card share its VRAM. One pod loading a large model will OOM the others. It multiplies scheduling slots, not capacity.
MIG is the real answer for isolation — it partitions the GPU in hardware, so each slice has its own memory and compute. Only available on supported datacenter GPUs.
MIG changes the resource name — profiles appear as resources like nvidia.com/mig-1g.10gb, so manifests must request the profile rather than nvidia.com/gpu.
Batch Jobs vs Long-Lived Servers
Training and inference want different Kubernetes objects, and using the wrong one is a common source of confusion:
# Training — a Job that runs to completion and frees the GPU
apiVersion: batch/v1
kind: Job
metadata:
name: finetune
spec:
backoffLimit: 2
template:
spec:
restartPolicy: Never
containers:
- name: train
image: my-registry/trainer:2.1.0
resources:
limits:
nvidia.com/gpu: 2
A Deployment for training means a completed run gets restarted forever. A Job for an inference server means it is never replaced when it dies. Use a Job for finite work — like the LoRA fine-tuning in our fine-tuning guide — and a Deployment for servers.
Monitoring
Kubernetes reports GPU allocation, never GPU usage. A node can show all GPUs allocated while every card sits idle. DCGM exporter fills that gap:
# Allocation view — what Kubernetes knows
$ kubectl describe node <node> | grep -A5 "Allocated resources"
# Actual utilization — from DCGM metrics
# DCGM_FI_DEV_GPU_UTIL, DCGM_FI_DEV_FB_USED, DCGM_FI_DEV_GPU_TEMP
Track allocation against utilization together: a persistent gap between them is the clearest signal that you are paying for GPUs nobody is using. Our GPU monitoring guide covers the metrics.
Troubleshooting
Pod stuck in Pending — run kubectl describe pod and read the events. "Insufficient nvidia.com/gpu" means no node has a free GPU — either all are allocated, or no node advertises any.
nvidia.com/gpu missing from Allocatable — the device plugin is not running on that node, or the node lacks the driver/container toolkit. Check the plugin DaemonSet pod logs on that node.
Pod runs but sees no GPU — the container runtime is not configured for NVIDIA on that node. Test with a plain container run outside Kubernetes.
"CUDA driver version is insufficient" — the image needs a newer driver than the node has. Pin image tags per node pool or upgrade drivers.
Requests must equal limits error — expected for extended resources. Specify only the limit and let Kubernetes mirror it.
Pods OOM after enabling time-slicing — time-slicing shares VRAM without partitioning it. Reduce replicas, or move to MIG for real isolation.
GPU works on some nodes only — a heterogeneous cluster with inconsistent driver versions. Label nodes by GPU model and driver, and select explicitly.
Frequently Asked Questions
Why is my GPU pod stuck in Pending?
Run kubectl describe pod and read the events. "Insufficient nvidia.com/gpu" means either every GPU is already allocated — extended resources cannot be overcommitted — or no node advertises GPUs, which points to a missing device plugin, driver, or container toolkit on the node.
Can two pods share one GPU in Kubernetes?
Yes, through time-slicing, MPS, or MIG. Time-slicing simply advertises more scheduling slots and provides no memory isolation, so one pod can OOM the others. MIG partitions the GPU in hardware and is the right choice for production multi-tenancy on supported datacenter cards.
Should I use the NVIDIA device plugin or the GPU Operator?
Use the device plugin when you already manage drivers and the container toolkit on nodes yourself. Use the GPU Operator when you want Kubernetes to manage the whole stack, which is particularly valuable for autoscaling clusters where nodes appear without pre-baked drivers.
Why can't I request half a GPU?
GPUs are exposed as an extended resource, which must be a whole number and cannot be overcommitted. Fractional sharing requires time-slicing or MIG rather than a fractional resource request.
Does Kubernetes show me GPU utilization?
No. Kubernetes reports allocation only, so all GPUs can appear allocated while sitting idle. Deploy DCGM exporter to get real utilization, memory, and temperature metrics, and compare them against allocation to find waste.