GPUs in Kubernetes: How It Actually Works Under the Hood

GPUs in Kubernetes are the most expensive line item in almost every AI infrastructure budget, and the platform manages them worse than almost anything else it schedules. A CPU request can be fractional, throttled, and reclaimed the moment a pod goes idle. A GPU request is a binary claim on an entire physical device, and Kubernetes has no built-in way to tell whether that device is doing any work at all.

This gap exists because GPUs sit outside the kernel primitives Kubernetes relies on for CPU and memory. Cgroups meter and cap what a container can use, and namespaces isolate what it can see, but a GPU is opaque to both, so Kubernetes falls back to counting whole devices and handing the real allocation work to the NVIDIA driver stack. Kubernetes can confirm a GPU has been assigned to a pod. It has nothing to say about whether that pod is using 3 percent of the device or 99 percent.

That distinction between allocated and used is not academic. GPU nodes routinely cost several times what an equivalent CPU node does, and a fleet where every device shows as claimed can still be running at a fraction of its real capacity, with no native signal telling anyone that a problem exists.

This guide is the first in a five-part series on running GPU workloads on Kubernetes, and it covers the mechanics everything else in the series builds on: the four-layer stack that turns a bare GPU into something nvidia.com/gpu: 1 can schedule against, what that resource request actually promises and doesn’t, a working pod manifest, and the observability gap that leaves utilization invisible until DCGM closes it. The next section picks up from there, showing how DCGM telemetry becomes something a cluster can act on automatically instead of a number an engineer checks by hand.

Why GPUs Aren’t Like CPUs in Kubernetes

When you run a workload on Kubernetes, you are really asking the Linux kernel to share a machine’s CPU and memory across many processes. Kubernetes leans on two kernel primitives to do this. The first is cgroups, which meter and cap how much CPU time and memory each container receives. The second is namespaces, which isolate what each container is allowed to see. Because these controls live in the kernel, Kubernetes can do things with CPU and memory that it cannot do with a GPU at all: it can hand a pod 500 millicores, throttle that pod the moment it exceeds its share, reclaim memory when the node comes under pressure, and pack dozens of containers onto a single physical core. To the kernel, a CPU is effectively divisible and fully reclaimable. A GPU is none of those things. The kernel does not schedule work onto an NVIDIA GPU; the NVIDIA driver does, and that driver is a closed-source blob with its own programming layer, CUDA, owned end to end by the vendor. cgroups cannot meter GPU compute, and namespaces cannot partition GPU memory, because the kernel that Kubernetes depends on has no authority over the device in the first place.

As far as Kubernetes is concerned, the file at /dev/nvidia0 is just an opaque character device it knows nothing about. This is the reason GPUs are so often called second-class citizens in Kubernetes. The platform was built around Linux primitives that GPUs sit entirely outside of, so it cannot subdivide, throttle, or reclaim a GPU the way it does a CPU. It falls back to the only thing it can do, which is to treat the GPU as an opaque, countable resource and hand the real work of allocating it to an external component. That component is the NVIDIA device plugin, and the next section follows exactly what it does. The difference becomes concrete the first time you put a CPU request and a GPU request side by side in the same pod spec:

resources:
  requests:
    cpu: "500m"       # half a core: fractional, throttleable, reclaimable
    memory: "1Gi"     # metered continuously by the kernel via cgroups
  limits:
    nvidia.com/gpu: 1 # one whole GPU: integer-only, all-or-nothing

cpu: 500m is a fluid request that the kernel enforces moment to moment. nvidia.com/gpu: 1 is a binary claim: the pod is given one entire physical GPU, or it does not schedule at all. There is no syntax for half a GPU in this field, two pods cannot share the device through it, and Kubernetes records only that the GPU has been assigned, never whether a single CUDA kernel is actually running on it. That distance between “allocated” and “actually used” is the thread this series pulls on, and it runs straight back to the hardware. A CPU is designed to context-switch, pausing one process mid-instruction, running another, and resuming the first a few microseconds later. A GPU is designed for raw throughput rather than interleaving. A computation launched onto the device generally runs to completion, and memory is committed to the GPU’s VRAM in large contiguous blocks rather than paged in and out on demand. The chip is engineered to finish enormous parallel jobs quickly, not to be sliced thinly across many tenants. Kubernetes inherited that constraint the moment it added GPU support through the device plugin framework in version 1.8, and almost every pattern in the rest of this guide exists to work around it.

The Full GPU Stack, Layer by Layer

A GPU node is not usable the moment it joins the cluster. Between the silicon and a container that can actually call torch.cuda.is_available() sit four distinct layers, each installed and operated separately, and a request for nvidia.com/gpu: 1 only succeeds when all four are in place. Walking them in order is the clearest way to understand what “GPU support” in Kubernetes really consists of.

Layer 1: the kernel driver. The NVIDIA driver is installed on the host as a Linux kernel module. Loading it creates the character devices that everything above depends on, including /dev/nvidia0 for the first GPU, /dev/nvidiactl for control operations, and /dev/nvidia-uvm for unified memory. Nothing in user space can reach the GPU until these device files exist. This is also the layer that pins your hardware to a specific driver version, which matters for the compatibility rules in the next layer.

Layer 2: CUDA in user space. CUDA is the programming layer that frameworks such as PyTorch and TensorFlow actually call; when a model runs on a GPU, it is issuing CUDA calls, not talking to the kernel directly. For operators, the driver version and the CUDA toolkit version do not have to match, but they do have to be compatible. NVIDIA publishes a forward-compatibility matrix that defines which CUDA releases a given driver supports, and a container built against a newer CUDA toolkit can run on an older host driver only when that matrix allows it. A large share of first-time GPU failures trace back to a version pairing that falls outside it.

Layer 3: the NVIDIA Container Toolkit. A container scheduled onto a GPU node still cannot see the GPU by default, because container isolation hides the host’s devices. The NVIDIA Container Toolkit closes that gap by adding a runtime hook to the container runtime, usually containerd. When a GPU container starts, the hook mounts the driver libraries and the relevant /dev/nvidia* devices into the container’s namespace. Without this layer a pod can request and be granted a GPU and still fail at runtime with a driver-not-found error, which is one of the more common first-time mistakes. The toolkit is wired in through a RuntimeClass that pods reference:

apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: nvidia
handler: nvidia # routes the pod to the nvidia container runtime in containerd,
                # which injects the driver libraries and /dev/nvidia* at startup

Layer 4: the device plugin. Kubernetes still has no native concept of a GPU, so something has to tell it that GPUs exist and how many. That job belongs to the NVIDIA device plugin, a DaemonSet that runs on every GPU node. It queries the driver through NVML, NVIDIA’s management library, to count the GPUs on the node, then registers them with the kubelet over the Device Plugin API. The kubelet in turn advertises them to the control plane as an extended resource named nvidia.com/gpu, which is why a GPU node reports them in its capacity:

$ kubectl get node gpu-node-1 -o jsonpath='{.status.capacity}'
{"cpu":"48","memory":"353814036Ki","nvidia.com/gpu":"4","pods":"110"}

Those four nvidia.com/gpu units are the only thing the scheduler ever sees. It places a pod that requests nvidia.com/gpu: 1 the same way it handles any extended resource: find a node whose advertised count is not yet exhausted, then bind. When the pod lands, the real handoff runs through the device plugin. The kubelet calls Allocate, the plugin selects specific physical GPUs and returns their device IDs, the kubelet passes those to containerd, and the runtime hook from Layer 3 sets NVIDIA_VISIBLE_DEVICES and mounts the matching /dev/nvidiaN into the container. Only at that point does the GPU become visible to CUDA inside the pod.

In practice almost nobody installs these four layers by hand. The NVIDIA GPU Operator packages them together and manages them as a unit on every GPU node: the driver, the container toolkit, the device plugin, node feature discovery for labelling GPU hardware, and the DCGM exporter for metrics, which becomes relevant later in this guide. The Operator is the standard way to turn a fresh GPU node pool into something Kubernetes can schedule against, and it is what most managed offerings install when you add a GPU node pool.

The stack explains how a GPU reaches a container. It does not explain what you are actually getting when you write nvidia.com/gpu: 1, which turns out to be both more and less than most people expect.

What nvidia.com/gpu: 1 Actually Means

The request that gives a pod a GPU is a single line, and its simplicity hides how different it is from every other resource in a pod spec:

resources:
  limits:
    nvidia.com/gpu: 1 # one whole physical GPU. "0.5" is rejected by the API server.

Three properties of that line shape everything downstream, and each is worth being precise about.

First, the number is a count of whole devices. nvidia.com/gpu is an integer-only extended resource, so the only valid values are 0, 1, 2, and upward. A request for 0.5 does not under-allocate a GPU; it fails validation outright. One unit means one entire physical GPU, assigned to exactly one container for the lifetime of the pod.

Second, the request is also the limit. Kubernetes requires request and limit to be equal for extended resources, which removes the burstable middle ground that CPU and memory rely on. With CPU you can request half a core and burst toward a higher limit, and the kernel arbitrates contention in real time and exposes it through throttling metrics. A GPU has none of that elasticity. You ask for a fixed number of whole devices, you receive exactly that number, and there is no overcommit, no bursting, and no soft limit for the platform to enforce.

Third, the number says nothing about use. Kubernetes treats nvidia.com/gpu purely as an allocation count. It does not cap GPU memory, it does not cap GPU compute, it provides no isolation between processes that share the device inside the container, and it records nothing about whether the GPU is doing any work at all. A pod that loads a 200 MB model and serves one request an hour holds its GPU exactly as firmly as a pod saturating all 80 GB of VRAM at full compute. To the scheduler, and to every Kubernetes-native metric, the two are indistinguishable:

$ kubectl describe node gpu-node-1 | grep -A2 "Allocated resources"
Allocated resources:
  Resource         Requests  Limits
  nvidia.com/gpu   4         4

Four GPUs allocated, zero available. Whether those four devices sit at 3 percent utilisation or stay pinned at 99 percent, this is the only number Kubernetes has, and it is the number that drives scheduling decisions, capacity planning, and the GPU line on your cloud bill.

GPUs can be shared. Time-slicing, MPS, MIG, and the newer Dynamic Resource Allocation all let a single physical GPU serve more than one workload, and they are the entire subject of the next post in this series. None of them changes the meaning of the plain nvidia.com/gpu integer. Through that field a GPU is strictly all-or-nothing, and the distance between what Kubernetes has allocated and what the hardware is actually doing is left for something outside Kubernetes to measure. That distance is the subject of the rest of this article.

Your First GPU Pod

The smallest useful experiment is a pod that requests one GPU, prints what it can see, and exits. Here is a complete manifest that does exactly that:

apiVersion: v1
kind: Pod
metadata:
  name: gpu-probe
spec:
  restartPolicy: Never
  runtimeClassName: nvidia # use the NVIDIA container runtime so the driver
                           # libraries and /dev/nvidia* are injected into the
                           # container (installed by the GPU Operator)
  nodeSelector:
    nvidia.com/gpu.present: "true" # only schedule onto a node that has a GPU
  containers:
  - name: cuda
    image: nvidia/cuda:12.4.1-base-ubuntu22.04
    command: ["nvidia-smi"] # report the visible GPU, then the pod completes
    resources:
      limits:
        nvidia.com/gpu: 1 # one whole GPU, request equals limit, enforced

Three fields carry the GPU-specific behavior. resources.limits.nvidia.com/gpu: 1 is the allocation the device plugin satisfies, exactly as covered in the previous section. nodeSelector keeps the pod off CPU-only nodes, which advertise no nvidia.com/gpu capacity and would otherwise leave the pod Pending indefinitely. runtimeClassName: nvidia selects the container runtime that injects the driver, and it is the field most often missing the first time: a pod can be granted a GPU and still fail at startup with a driver-not-found error if the runtime hook never ran.

On a managed platform the node selection and runtime wiring look slightly different. On GKE you choose the hardware with the accelerator label rather than a generic selector, and you usually omit runtimeClassName entirely, because GKE installs the driver through its own managed installer and wires the NVIDIA runtime into the node for you:

nodeSelector:
  cloud.google.com/gke-accelerator: nvidia-l4 # GKE selects an L4 GPU node pool,
                                               # the GKE driver installer handles the rest

With the pod scheduled, the logs show what the container can actually see:

$ kubectl logs gpu-probe
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.183.01          Driver Version: 535.183.01        CUDA Version: 12.4      |
|--------------------------------+-------------------------------+------------------------|
| GPU  Name          Persistence-M | Bus-Id              Disp.A   | Volatile Uncorr. ECC   |
| Fan  Temp   Perf   Pwr:Usage/Cap | Memory-Usage                | GPU-Util  Compute M.   |
|==================================+==============================+========================|
|   0  NVIDIA L4   Off             | 00000000:00:04.0   Off       | 0                      |
| N/A   38C    P8   12W / 72W      | 0MiB / 23034MiB              | 0%        Default      |
+----------------------------------+------------------------------+------------------------+
| No running processes found                                                              |
+-----------------------------------------------------------------------------------------+

The container sees one NVIDIA L4, a 535-series driver, CUDA 12.4, and roughly 23 GB of VRAM, all of it free because this probe loads nothing before exiting. Run the same nvidia-smi inside a real inference pod and the numbers move: a model that occupies two or three gigabytes of VRAM and then waits for requests sits at 0 percent GPU-Util between calls, holding the entire device while using a fraction of it.

That single command is also the edge of what you can easily see. nvidia-smi reports live, accurate figures, but only from inside the pod and only at the moment you run it. None of it reaches Kubernetes. The control plane still records one GPU allocated and zero available, whether the device is idle or saturated. Turning that per-pod snapshot into something the cluster can act on needs a layer Kubernetes does not ship, which is where the next section goes.

The Observability Gap

Everything Kubernetes tells you about a running workload comes from one pipeline. The kubelet’s built-in cAdvisor collects per-container CPU and memory from cgroups, metrics-server aggregates it, kube-state-metrics exposes object state, and kubectl top reads the result. That pipeline was built around the same kernel primitives from the first section, so it reports the two resources the kernel controls, CPU and memory, and nothing else. Ask it about a GPU and there is no field to return:

$ kubectl top pod vllm-server
NAME          CPU(cores)   MEMORY(bytes)
vllm-server   240m         14620Mi   # CPU and memory only. There is no GPU column, and no flag adds one.

The GPU is running the whole time. The nvidia-smi output from the previous section proves the device is busy or idle at any given moment. That information lives inside the pod and never enters the metrics pipeline, so Kubernetes can tell you a GPU is allocated, but it cannot tell you whether that GPU is doing anything. The consequences run straight into the parts of Kubernetes that are supposed to react to load. The default scheduler places GPU pods by allocation count, so once a node’s GPUs are claimed it is full, regardless of whether those GPUs are working. Horizontal pod autoscaling scales on CPU, memory, or custom metrics, none of which include GPU utilisation unless you build that pipeline yourself. Vertical pod autoscaling recommends CPU and memory requests and has nothing to say about a GPU. Every native control loop that could respond to a GPU sitting idle is blind to the one signal that would tell it to act.

At the scale of a real cluster this produces a predictable pattern. Each workload is given a whole GPU because the integer resource offers no other option, each of those GPUs tends to run well below its capacity, and the allocation view reports the fleet as fully committed the entire time. A cluster can show every GPU claimed while the hardware as a whole is doing a fraction of the work it could, and no Kubernetes-native view will surface the difference. That difference is expensive. GPU nodes are the most costly line item in most AI clusters, often by a wide margin, and when the platform cannot see utilisation, paying for allocated-but-idle silicon becomes the default rather than the exception.

The FinOps Foundation’s 2026 State of FinOps survey, representing practitioners managing more than $83 billion in annual cloud spend, found that granular GPU utilisation monitoring is the single most requested capability practitioners say they’re missing today. CNCF’s own reporting on production AI workloads names the same underlying problem from the infrastructure side: maximising GPU utilisation, not CPU or memory, is what teams identify as the dominant bottleneck once AI workloads move onto Kubernetes, because the platform doing the scheduling has no view of utilisation to act on.

Closing the gap means getting real telemetry out of the GPU and into a place the cluster can use. The driver already exposes it. NVIDIA’s Data Center GPU Manager, or DCGM, reads per-GPU compute, memory, temperature, and power directly from the hardware through NVML, and its exporter publishes those figures as metrics a cluster can scrape. Kubernetes does not install this for you, and it is not part of the device plugin. It is the layer you add when you decide that “allocated” is no longer a good enough answer to “is this GPU being used.” What you do with that telemetry once you have it is where the next section goes.

From DCGM Telemetry to Automated Action

DCGM tells you what a GPU is doing. It does not decide what to do about it. That gap, between a metric and an action, is where ScaleOps operates.

ScaleOps is an autonomous resource management platform, not a monitoring tool, and the distinction matters for how it uses DCGM specifically. Where a dashboard built on DCGM data shows an engineer that a fleet of GPUs is running at 12 percent compute utilisation, ScaleOps reads the same NVML-backed signal continuously and feeds it into the same rightsizing decisions it already makes for CPU and memory: whether a pod’s actual resource envelope, this time expressed in GPU terms, matches what it was originally requested with.

Concretely, ScaleOps ingests DCGM’s per-GPU compute, memory, and power series through its existing metrics pipeline, the same one that already reads cAdvisor and kube-state-metrics for CPU and memory decisions. That gives it a workload-level view of GPU behaviour over time rather than a single point-in-time reading, which matters because a model server that idles between bursts of inference traffic and one that is saturated look identical to kubectl describe node, but not to a system watching the DCGM series across the pod’s actual traffic pattern.

Real-Time Pod Rightsizing applies to GPU workloads the same way it does to CPU and memory: recommending, and where configured applying, resource requests that reflect what a workload has actually used rather than the static number an engineer guessed at deploy time. Node Optimization uses the same utilisation signal at the cluster level, consolidating GPU-backed nodes when the workloads scheduled onto them are not using the capacity those nodes provide. Neither of these replaces the Kubernetes scheduler, HPA, or Cluster Autoscaler. ScaleOps runs alongside them and gives them a GPU utilisation signal none of them have natively.

Closing that gap means GPUs which show as fully allocated while running well under capacity become visible, and correctable, without a platform team having to hand-build a DCGM-to-Kubernetes pipeline first.

Where This Goes Next: Sharing a Single GPU

Everything in this guide assumes a pod gets a whole GPU, because that is what the plain nvidia.com/gpu request enforces. It is also the least efficient way to run most inference workloads, since a model that occupies 3 GB of a 24 GB card still holds the entire device for as long as the pod is alive.

The next post in this series covers the four ways Kubernetes lets more than one workload share a physical GPU: time-slicing, NVIDIA’s Multi-Process Service, Multi-Instance GPU on hardware that supports it, and Dynamic Resource Allocation, which changes how far Kubernetes itself can go in expressing partial and structured device claims. None of them changes what this guide covers. The four-layer stack, the driver, CUDA, the container toolkit, and the device plugin, still has to be in place before any sharing model works, and DCGM still has to be reading the device before anyone can tell whether a sharing configuration is actually improving utilisation or just adding contention.

ScaleOps works with your existing HPA, KEDA, and Cluster Autoscaler configuration, whether GPUs are allocated whole or shared, so nothing here requires a migration or a rearchitecture before it applies.

Try ScaleOps free → to see how your current GPU allocation compares to what your workloads are actually using.

Book a demo → to walk through DCGM-based rightsizing on your own cluster with our team.