How to Deploy vLLM on Kubernetes: The Complete Guide to LLM Inference in Production

Running vLLM Kubernetes workloads in production is a different problem from running vllm serve on a workstation. The model is the easy part. The work is everything between a single process holding a GPU and a resilient inference service other teams can call, that scales when traffic arrives, recovers when a pod dies, and does not quietly waste the most expensive hardware in the cluster.

This guide is a single worked example carried end to end. We deploy Llama-3-8B on a GKE cluster with NVIDIA L4 GPUs, expose it as a service, send it a real request, watch its metrics, scale it under load, shard a larger variant across GPUs, and then deal with the failure modes and the cost. Every section advances the same deployment rather than showing a disconnected snippet, so by the end you have a running, observable, scalable vLLM service and the reasoning behind each decision. It is written for platform engineers who are comfortable with Kubernetes but newer to LLM serving.

One decision shapes everything downstream, so it is worth stating as the thesis this guide argues: serve vLLM as a Deployment ****behind a Service, keeping replica count and node count as two independent, composable levers. The alternatives worth knowing each solve a specific problem: a StatefulSet when replicas need stable identity and their own PersistentVolume for the model cache, LeaderWorkerSet when a single replica has to span several pods for multi-node inference (§7), and a serving layer like KServe (§8) when you would rather manage an InferenceService than raw manifests. Everything below assumes that model, and the scaling section is where the choice pays off. Once the service is running, the harder question becomes how much of each GPU you are actually using — a question a layer like ScaleOps AI Infra answers, and one we return to at the end.

This is not a whiteboard exercise. Everything below was validated on a live GKE cluster: Kubernetes 1.35, NVIDIA L4 24 GB nodes for single-GPU serving and a pair of A100-40 GB cards (NVLink) for the multi-GPU sections — running vLLM v0.8.5 on the V1 engine, with KEDA, the kube-prometheus-stack, and GPU metrics fed by ScaleOps, since the bare upstream exporter does not survive GKE (§5). ScaleOps itself ran observe-only throughout, recommendations on and automated optimization off, so every number below is vanilla vLLM behavior. The numbers and the failure modes below are from that run, and where a result depends on the environment it is flagged as ours rather than presented as universal.

What vLLM Is, and Why It Changes GPU Economics

vLLM is an open-source inference and serving engine: it runs a large language model efficiently on datacenter GPUs and exposes an OpenAI-compatible HTTP API through the vllm serve command. It supports 200+ model architectures natively and is the serving layer most model and hardware vendors now target first.

The reason it matters for a platform team comes down to memory. LLM inference produces a KV cache — the keys and values already computed for the tokens in a sequence — and that cache grows as generation proceeds. Earlier serving systems allocated KV-cache memory in large contiguous blocks per request, leaving most of it stranded: only 20–40% was ever used, the rest lost to internal fragmentation and over-reservation. PagedAttention borrows the operating system’s virtual-memory trick. It partitions the KV cache into fixed-size blocks, like pages, and maps logical blocks to physical blocks through a block table, allocating on demand rather than up front. Per the vLLM PagedAttention paper, the result is roughly 4% waste instead of 60–80%, which translates directly into larger batch sizes and higher throughput on the same GPU.

That 60–80%-to-4% jump is worth holding onto, because the same waste reappears one level up: PagedAttention reclaims memory stranded inside a single GPU, while at the fleet level whole GPUs sit idle because Kubernetes treats a card as indivisible and hands each pod the entire thing. Reclaiming that cluster-level waste with fractional GPU allocation is what ScaleOps AI Infra does, and §10 gets concrete about how.

Two related mechanics explain vLLM’s behavior under load. Continuous batching admits and retires requests at the token level rather than waiting for a fixed batch to finish, so one long generation does not block short ones. The scheduling consequence is that a single replica behaves as a high-concurrency server rather than a one-request worker, so one pod absorbs a lot of traffic before you need a second, and the signal that it is finally full is its own queue starting to build (§6), not raw request count. Prefix caching, on by default in the V1 engine, shares the KV blocks of a common prompt prefix across requests, so a shared system prompt is computed once and reused. That cache lives in a single pod’s GPU memory, so the benefit is per-replica: a plain round-robin Service scatters requests that share a prefix across pods and each one recomputes it, which is why prefix-aware routing becomes the real lever once you scale out (§8). Hold onto prefix caching — it reappears in the monitoring and cost sections, because how often it hits is one of the most useful numbers you can watch.

This maps cleanly onto Kubernetes because the unit you schedule — a pod that requests one or more GPUs and exposes an HTTP port — lines up with how vLLM wants to run: one server process per GPU or group of GPUs, load-balanced behind a Service. That is exactly what we build next.

Prerequisites: The Cluster We Are Building On

Our example runs on GKE with a GPU node pool of NVIDIA L4 cards (24 GB each), which comfortably holds Llama-3-8B in FP16 with room for the KV cache. Three things have to be in place before any vLLM manifest will schedule.

First, a GPU node pool. On a managed platform this is a node pool with a GPU machine type, and the nodes are tainted so only workloads that explicitly tolerate the taint land on expensive hardware. Our pool carries the taint nvidia.com/gpu=present:NoSchedule.

Second, the NVIDIA device plugin, which runs as a DaemonSet on every GPU node and advertises the GPUs to the kubelet as an allocatable resource named nvidia.com/gpu. This is the one DaemonSet that belongs in this picture — its job is per-node hardware registration, which is what a DaemonSet is for. On GKE the plugin and driver are installed for you when you add a GPU node pool; on self-managed clusters you install the NVIDIA GPU Operator, which also manages the driver and container toolkit, per NVIDIA’s documentation.

Third, the driver and container toolkit on the node, so containers reach the GPU through the NVIDIA runtime. Confirm GPUs are schedulable before deploying anything:

kubectl get nodes -o custom-columns=NAME:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu
# A GPU node should report 1 (or more) under GPU, not <none>.

If that column is empty, no vLLM pod will ever leave Pending, and no manifest tuning fixes it — the problem is below vLLM, in the device plugin or driver. With the column reporting 1 on our L4 nodes, we are ready to deploy. In our test, a fresh L4 node took about six minutes to provision and report nvidia.com/gpu allocatable — a number worth remembering for §6, where node autoscaling has to add that capacity under load.

Deploy vLLM as a Deployment, Then Call It

Here is the manifest that runs one vLLM replica serving Llama-3-8B on one L4, fronted by a ClusterIP Service. This is the spine of the whole guide; later sections modify this object rather than introduce new ones.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-llama3-8b
  namespace: inference
spec:
  replicas: 1                                   # start at 1; KEDA scales this in §6
  selector:
    matchLabels:
      app: vllm-llama3-8b
  template:
    metadata:
      labels:
        app: vllm-llama3-8b
    spec:
      tolerations:
        - key: nvidia.com/gpu                    # land on the tainted GPU node pool
          operator: Exists
          effect: NoSchedule
      containers:
        - name: vllm
          image: vllm/vllm-openai:v0.8.5         # pin a tested tag — :latest drifts and surprises you on restart
          args:
            - "--model"
            - "NousResearch/Meta-Llama-3-8B-Instruct"   # ungated Llama-3-8B mirror; the official meta-llama repo 403s until Meta approves you (see below)
            - "--gpu-memory-utilization"
            - "0.90"                             # startup sizing knob for weights + KV pool. How high you can go is a startup question (§4)
            - "--max-model-len"
            - "8192"                             # cap context; larger = more KV cache reserved per request
          ports:
            - containerPort: 8000                # vLLM's OpenAI-compatible server listens here
          env:
            - name: HUGGING_FACE_HUB_TOKEN       # Llama is gated; this is required
              valueFrom:
                secretKeyRef:
                  name: hf-token
                  key: token
          resources:
            limits:
              nvidia.com/gpu: "1"                # one physical L4 per replica
          readinessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 60              # weights are still downloading and loading; /health first passed at ~6 min in our test
            periodSeconds: 10
            failureThreshold: 60                 # ~10 min window so a slow cold start is not mistaken for a crash-loop (§9)
          volumeMounts:
            - name: model-cache
              mountPath: /root/.cache/huggingface
      volumes:
        - name: model-cache
          emptyDir: {}                           # swap for a PVC to avoid re-downloading weights on restart (§9)

apiVersion: v1
kind: Service
metadata:
  name: vllm-llama3-8b
  namespace: inference
spec:
  type: ClusterIP                                # internal only; front with an Ingress/Gateway for external traffic
  selector:
    app: vllm-llama3-8b
  ports:
    - port: 80
      targetPort: 8000                           # clients hit :80, vLLM listens on :8000

Apply both, wait for the pod to pass readiness (the first start is slow — that is the cold-start tax we dissect in §9), and send it a real request. vLLM speaks the OpenAI API, so the call is ordinary:

kubectl -n inference port-forward svc/vllm-llama3-8b 8000:80 &

curl -s http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
        "model": "NousResearch/Meta-Llama-3-8B-Instruct",
        "messages": [{"role": "user", "content": "Explain Kubernetes in one sentence."}],
        "max_tokens": 64
      }'

{
  "id": "chatcmpl-3f0c…",
  "object": "chat.completion",
  "choices": [
    {"index": 0, "message": {"role": "assistant",
      "content": "Kubernetes is an open-source platform that automates deploying, scaling, and operating containerized applications across a cluster of machines."},
     "finish_reason": "stop"}
  ],
  "usage": {"prompt_tokens": 17, "completion_tokens": 24, "total_tokens": 41}
}

That usage block is the first place tokens show up, and it is worth noticing now because token throughput becomes the thing we monitor and, ultimately, pay for. Other workloads in the cluster reach the model at http://vllm-llama3-8b.inference.svc.cluster.local/v1/chat/completions, and the Service load-balances across however many replicas exist.

Two practical notes from actually running this. First, the model is gated: Meta’s meta-llama/Meta-Llama-3-8B-Instruct returns 403 ... not in the authorized list even with a valid HUGGING_FACE_HUB_TOKEN until Meta approves your account, so the manifest above uses the ungated NousResearch/Meta-Llama-3-8B-Instruct mirror, an identical drop-in you can run immediately. Second, the first start is slow: in our test the pod sat 0/1 Running for about 6.3 minutes — pulling the ~16 GB of weights, loading them onto the GPU, and capturing CUDA graphs — before /health passed. That is why the readiness probe uses a long failureThreshold rather than a tight delay; §9 covers how to shrink the wait.

Now the thesis in practice. A Deployment fits because our service is stateless from Kubernetes’ point of view: replicas are interchangeable, the weight cache is rebuildable, and KEDA (§6) scales a Deployment’s replicas directly. The alternatives earn their place at specific requirements. A StatefulSet buys stable pod identity and a per-replica PersistentVolume through volumeClaimTemplates, useful when each replica should keep its own warm model cache across restarts. LeaderWorkerSet, the Kubernetes SIG API built for multi-node inference, is what you reach for when one replica must span several pods, which is exactly the shape pipeline parallelism produces in §7. KServe (§8) wraps the same vLLM container in a managed InferenceService.

Production Configuration That Matters

Our pod serves requests, but a few flags separate a server that holds up from one that falls over on the first real burst. Each of these tunes the same vllm-llama3-8b Deployment.

A quick way to reason about GPU choice: FP16 weights take roughly 2 GB per billion parameters, and you need headroom on top for the KV cache, which grows with context length and concurrency. That gives a rough floor for sizing the node pool:

The table is a starting point, not a guarantee: leave room for the KV cache, and confirm the real footprint on your hardware before committing a node pool. This is also why --gpu-memory-utilization and --max-model-len are levers rather than set-and-forget values, since together they decide how much of the remaining memory becomes usable KV cache.

Prefix caching is on by default in V1. You do not enable it, but you should understand it: a shared system prompt across requests is computed once. For chat and agent workloads that prepend the same instructions every call, this is a real throughput win, and §5 shows how to confirm it is actually hitting.

Structured / JSON output (guided decoding) is supported natively and is worth enabling at the request layer when downstream consumers expect strict JSON, rather than parsing-and-retrying in application code.

Finally, the readiness probe. vLLM exposes /health, but a fresh pod is not ready the instant the container starts — it downloads the model, loads weights onto the GPU, and captures CUDA graphs. Probe too early and Kubernetes restarts a pod that was merely starting, producing a crash loop that looks like a vLLM bug and is actually a probe-timing bug. Our initialDelaySeconds: 60 plus a generous failureThreshold reflects that — in our run the pod needed about six minutes before /health passed; §9 explains why startup is slow and how to shrink it.

Monitoring: See What the GPU and the Model Are Doing

You cannot scale on a signal you are not collecting, and Kubernetes exposes neither GPU utilization nor inference health on its own. Two layers fill the gap, and our example needs both.

The first is GPU hardware metrics from NVIDIA’s DCGM exporter, deployed as a DaemonSet on GPU nodes and scraped by Prometheus. Its headline signal, DCGM_FI_DEV_GPU_UTIL, is the percentage of time the GPU’s compute engines were busy, and the exporter labels it with pod and namespace, so you can attribute usage to our vllm-llama3-8b replica. Treat it as an observability signal rather than a scaling trigger; §6 shows, with measurements, why a busy percentage cannot carry the scaling decision. A minimal ServiceMonitor:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: dcgm-exporter
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: dcgm-exporter
  endpoints:
    - port: metrics
      interval: 15s
      honorLabels: true        # critical: without this Prometheus overwrites the metric's pod/namespace and per-pod GPU attribution silently breaks

Two findings from standing this up on GKE, because the one-line version hides real friction. First, the upstream DCGM exporter is not a drop-in on managed Kubernetes. On GKE it failed four separate and mostly silent ways: its DaemonSet does not tolerate the nvidia.com/gpu taint, so it scheduled on zero nodes; its priorityClassName: system-node-critical was rejected by GKE’s critical-pod quota; its distroless image swallowed the error; and even once running it logged NVML doesn't exist because GKE injects the GPU driver only into containers that request a GPU, and the exporter requests none. The fix is to use GKE Managed DCGM, the NVIDIA GPU Operator, or a GPU-aware DCGM build (ScaleOps ships one) instead of the bare upstream DaemonSet.

Second, and easy to miss: the ServiceMonitor needs honorLabels: true (above). Without it, Prometheus relabels the scraped metric with the exporter pod’s own identity, overwriting the pod and namespace of the vLLM pod the metric actually describes. Every per-pod GPU query then attributes usage to the exporter instead of the model, so dashboards and anything else reading those labels are silently wrong, with no error to tell you. We watched the query come back empty before the flag and return the right pod after.

The second layer is vLLM’s own telemetry. vLLM serves a Prometheus endpoint at /metrics, and the signals there are the ones that actually tell you whether inference is healthy — far more than raw GPU busy-ness:

histogram_quantile(0.95,
  sum(rate(vllm:time_to_first_token_seconds_bucket{namespace="inference"}[5m])) by (le))

Alert when that crosses your latency budget. It tends to fire before a raw utilization threshold would, because TTFT degrades as the queue builds, while a GPU can read busy long before requests start backing up.

This is the first natural place ScaleOps fits, and the hook is a genuinely hard problem you just set up. DCGM sees the device; vLLM’s /metrics sees the model; and the moment several models share a GPU, neither can tell you which pod is burning what. Closing that gap for LLM inference is where ScaleOps is heading: per-pod GPU utilization and memory attribution that holds up even when models share a card, vLLM’s own signals (TTFT, prefix cache hit rate, running and waiting requests, queue size) folded into the same per-pod view, and GPU spend mapped back to the workload driving it. The building block exists today, already providing device utilization and memory to the consuming pod.

Scaling: KEDA for Replicas, Cluster Autoscaler for Nodes

Scaling our service well means keeping two layers distinct: how many vLLM replicas run, and how many GPU nodes exist to run them on.

Replica scaling is KEDA’s job. A ScaledObject with a Prometheus trigger drives the replica count of the vllm-llama3-8b Deployment, and the signal it reads is vLLM’s own queue depth rather than GPU utilization:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: vllm-llama3-8b
  namespace: inference
spec:
  scaleTargetRef:
    name: vllm-llama3-8b              # the Deployment from §3; KEDA scales its replicas
  minReplicaCount: 1                  # keep one warm; scale-to-zero pays the full cold-start tax (§9)
  maxReplicaCount: 8
  cooldownPeriod: 300                 # GPU pods are costly to churn; scale down deliberately
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus.monitoring.svc:9090
        query: |
          avg(vllm:num_requests_waiting{namespace="inference", pod=~"vllm-llama3-8b.*"})
        threshold: "5"               # queued requests per replica; a starting point, tune to your latency budget

Send sustained load at the service and the queue on the busy replica builds; once the average crosses the threshold, KEDA adds replicas and the Service spreads requests across them. The threshold is doing real work, so it is worth seeing why the queue is the right signal. We characterized the candidates on our L4 with a concurrency sweep before wiring the trigger. vllm:num_requests_waiting held at exactly zero through the healthy range and went positive at the same moment p95 time-to-first-token jumped from under two seconds to 29. That is the moment a second replica is needed, and it is a moment GPU utilization cannot see: the card read 92% busy serving a single request, sat in a flat 92 to 98% band across the healthy range, and dropped to 73% at the overload point. A utilization threshold either fires for one user or stays silent during a pileup. For a tighter tail-latency target, scale on vllm:num_requests_running or a p95 TTFT SLO instead; in our sweep the running batch and KV-cache usage rose smoothly with load, so they move earlier than the queue does. The cooldownPeriod matters more here than for a typical web service: a GPU pod is expensive and slow to start, so you want it to scale up promptly but down slowly rather than thrash a node in and out of the pool.

To see this work, put sustained load on the service and watch the controllers respond. A simple load generator pointed at the OpenAI endpoint is enough,but pick the concurrency deliberately. At 16 concurrent requests our L4 held every one of them in vllm:num_requests_running with the queue at zero and the GPU pinned at 98%, and KEDA, correctly, did nothing: the server was busy, not overloaded. It took 80 concurrent requests to build a real queue:

# 2,000 requests, 80 concurrent — at 16, vLLM absorbs everything into the running batch and no queue forms
hey -n 2000 -c 80 -m POST \
  -H "Content-Type: application/json" \
  -d '{"model":"NousResearch/Meta-Llama-3-8B-Instruct","messages":[{"role":"user","content":"Summarize the CAP theorem."}],"max_tokens":1024}' \
  http://localhost:8000/v1/chat/completions

In a second terminal, watch the scaling objects move:

watch -n 2 kubectl get scaledobject,hpa,pods -n inference

KEDA manages a standard HorizontalPodAutoscaler under the hood, so what you see is the HPA’s current metric climbing past the target and the replica count following it up:

NAME                                  MIN   MAX   REPLICAS
scaledobject.keda.sh/vllm-llama3-8b   1     8     4

NAME                                       TARGETS       REPLICAS
horizontalpodautoscaler.../keda-hpa-...    37/5 (avg)    4

NAME                              READY   STATUS
vllm-llama3-8b-7c9d...-2xr4q      1/1     Running
vllm-llama3-8b-7c9d...-mq7wd      0/1     ContainerCreating   # new replica, paying the cold-start tax
vllm-llama3-8b-7c9d...-x8kfn      0/1     Pending             # no free GPU yet; node autoscaling below

Two things are worth watching on the metrics side while this happens. vllm:num_requests_waiting climbs past the target on the saturated replica, which is what triggers the scale-up. In our run it was peaking at 37 and taking the Deployment from one replica to four, then falls back toward zero once the second replica absorbs the queue. GPU utilization, if you keep that dashboard open too, barely moves through the whole event, which is exactly why it is not the trigger. When the load generator stops, the queue clears, and after the 300-second cooldownPeriod KEDA scales back down, slowly and on purpose, four replicas back to one in about six minutes in our run, because spinning a GPU pod back up is not free. That replica stuck in ContainerCreating is the cold-start cost made visible, and it is the concrete reason minReplicaCount stays at 1 rather than 0.

Node autoscaling is a separate concern you do not implement yourself — you let Cluster Autoscaler or Karpenter handle it. When KEDA adds a replica and no L4 node has a free nvidia.com/gpu, the new pod sits Pending, and the cluster autoscaler provisions another GPU node. The layers compose cleanly: KEDA decides how many replicas, the autoscaler decides how many nodes, and neither needs to understand the other’s internals.

There is a caveat this series covers in depth. When node autoscaling provisions a fresh GPU node, the new vLLM pod pays a cold-start cost — image pull, model download, weight load, CUDA graph capture — before it serves, which is why minReplicaCount stays above zero for latency-sensitive services. That latency is the subject of GPU Cold Starts in Kubernetes. In our test it took about 6.3 minutes for a fresh L4 pod to go from scheduled to serving, on top of the roughly six minutes to provision the node itself.

A second node-layer reality the happy path hides: GPU capacity is not guaranteed. When KEDA scaled up and the autoscaler asked for a node, GCE returned out of resources for both L4 and A100 in-zone, and the new pod simply stayed Pending — the configuration was correct, the capacity was not there. Treat FailedScaleUp and out-of-resources as an expected failure mode: alert on it, and use multi-zone GPU node pools or capacity reservations in production. Capacity did recover for us on retry, with an A100 node provisioning in about 66 seconds once the stockout cleared.

But there is one scaling subtlety that bites teams running GPU sharing. A device-level metric like DCGM_FI_DEV_GPU_UTIL reports the whole card, not the pod, so it cannot drive per-workload decisions on a shared GPU at all. The queue-depth trigger above survives sharing, because each vLLM server exposes its own vllm:num_requests_waiting. It is still a late signal, though: the queue only moves once requests are already waiting, and in our sweep vllm:num_requests_running and KV-cache usage climbed smoothly long before it did. A per-workload signal that correlates with load earlier, and holds up when models share a card, is what ScaleOps is building for LLM inference: an inference-aware custom metric that feeds the same HPA and KEDA controllers you already run rather than replacing them.

Multi-GPU: Tensor and Pipeline Parallelism for Bigger Models

Suppose the example grows: the team wants Llama-3-70B, whose weights plus KV cache do not fit on one L4. You shard it across several GPUs on one node with tensor parallelism. vLLM splits each layer’s tensors across the GPUs and runs them in parallel, exchanging activations between them. The flag is --tensor-parallel-size, and its value must equal the number of GPUs the pod requests:

containers:
  - name: vllm
    image: vllm/vllm-openai:v0.8.5
    args:
      - "--model"
      - "meta-llama/Meta-Llama-3-70B-Instruct"
      - "--tensor-parallel-size"
      - "4"                               # shard across 4 GPUs
    resources:
      limits:
        nvidia.com/gpu: "4"               # must equal --tensor-parallel-size
    volumeMounts:
      - name: dshm
        mountPath: /dev/shm               # NCCL talks over shared memory; the default 64 MiB hangs TP
volumes:
  - name: dshm
    emptyDir:
      medium: Memory

Three constraints decide whether this performs. First, tensor parallelism is chatty — the GPUs exchange data on every forward pass — so they must be connected by a high-bandwidth interconnect (NVLink) on the same node. Spreading a tensor-parallel group across nodes over ordinary networking bottlenecks on communication; the cross-node job belongs to pipeline parallelism (described in the next paragraph). Second, the GPU count and --tensor-parallel-size must agree, or the pod fails to start in a confusing way. Third, a non-obvious one we hit: tensor parallelism communicates through shared memory, and a container’s default /dev/shm of 64 MiB is too small, so NCCL hangs at startup — mount an in-memory emptyDir at /dev/shm, as the manifest above does. We validated this path end to end with Llama-3-8B at --tensor-parallel-size 2 on two NVLink-connected A100-40 GB cards: NCCL 2.21.5, two workers at world size 2, serving completions. Note what tensor parallelism is for: fitting a model too large for one GPU. For more throughput on a model that already fits, add replicas (§6), not GPUs per replica.

Tensor parallelism is one of two ways vLLM splits a model, and the second one matters the moment a model outgrows a single node. Tensor parallelism cuts within each layer: every GPU holds a slice of every weight matrix, all GPUs work on every token together, and they exchange partial results at each layer, following the Megatron-LM scheme vLLM implements. That constant exchange is why it wants NVLink on one node, and it is also why it helps per-token latency, since four GPUs push each token through the model at once. Pipeline parallelism cuts between layers instead: each GPU or node holds a contiguous stage of the model, and a request hands its activations from stage to stage. A stage boundary moves one activation tensor rather than per-layer partial sums, so pipeline parallelism tolerates ordinary datacenter networking, and it is how vLLM spans nodes. The tradeoff runs the opposite way: memory scales across nodes cheaply, but each token walks every stage in sequence, so per-token latency rises, and the pipeline only reaches full throughput when enough requests are in flight to keep every stage busy, which the continuous batching from §1 exists to do. The flag is –pipeline-parallel-size, it multiplies with –tensor-parallel-size (a replica uses TP × PP GPUs), and vLLM’s sizing rule for models beyond one node is worth memorizing: set tensor parallel size to the GPUs per node and pipeline parallel size to the number of nodes. Stages also allow uneven layer splits, which helps when one stage carries extra memory such as the embedding layer. On Kubernetes, a replica that spans nodes no longer fits in one pod, which is the multi-pod replica shape LeaderWorkerSet (§3) was built for. The decision rule: prefer tensor parallelism alone while the model fits on one node’s GPUs, add pipeline parallelism when it genuinely does not, and treat it as a capacity move rather than a speed move. We validated the single-node case live (the TP=2 run above); the multi-node pipeline path is documented guidance we describe rather than a configuration we ran.

Scaling Out: Prefill–Decode Disaggregation

There is a ceiling to tensor and pipeline parallelism and replica scaling, and beyond it the frontier is splitting the phases of inference across pods. Inference has two phases with very different hardware profiles. Prefill processes the entire prompt in one forward pass and is compute-bound: it wants raw FLOPs. Decode then generates tokens one at a time, each step reading the whole KV cache, so it is memory-bandwidth-bound and latency-sensitive. Run both on the same GPU and they interfere: a long prompt’s prefill monopolizes compute and stalls the token-by-token decode of every other in-flight request.

Prefill–decode disaggregation runs the two phases on separate pools sized for their respective bottlenecks, then transfers the KV cache from the prefill workers to the decode workers. A prefill pool can run on compute-dense GPUs and scale with prompt volume, while a decode pool optimizes for memory bandwidth and many concurrent token streams. The reference architecture in the vLLM ecosystem is llm-d, which adds LLM-aware, prefix-cache-aware request scheduling (routing each request to the worker most likely to already hold its prefix, which ties straight back to the prefix caching from §4) and uses a KV-connector interface with NVIDIA’s NIXL for the cache transfer. That transfer is not free, so disaggregation pays off only when the interference it removes outweighs the cost of moving the cache, which is the case for workloads with long prompts and high concurrency.

This is the one place in the guide where we could not get a clean end-to-end result, and that is itself the lesson: disaggregation is not a drop-in. Both paths carry hard prerequisites. llm-d’s production disaggregation expects RDMA networking (InfiniBand or RoCE) between pods and roughly eight GPUs, so standard GKE L4 and A100 pools will not do — it needs RDMA-capable nodes such as a3-megagpu H100s with GPUDirect. The lighter vLLM-native NixlConnector (NIXL over UCX, no RDMA) is deployable on a single two-GPU node and validates the mechanic in principle, but it is explicitly experimental: in our attempt on vllm/vllm-openai:latest, the prefill and decode pods both started and loaded NIXL, but the NIXL handshake listener thread crashed with ValueError: not enough values to unpack, the KV leases expired, and the decode side pulled the cache from zero remote workers. If you go here, pin a known-good version and budget for rough edges rather than expecting it to work first try.

What both paths have in common is orchestration: disaggregation means standing up separate prefill and decode pools, a KV connector between them, and prefix-aware routing in front, all assembled by hand on top of raw Deployments. If you would rather operate at a higher level than that, KServe runs vLLM as a declarative InferenceService — the same vLLM container underneath — and is built to manage this kind of multi-component serving:

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: llama3-8b
  namespace: inference
spec:
  predictor:
    minReplicas: 1                      # same warm-floor logic as the KEDA setup in §6
    model:
      modelFormat:
        name: vllm                      # KServe's built-in vLLM serving runtime
      args:
        - "--model"
        - "NousResearch/Meta-Llama-3-8B-Instruct"
      resources:
        limits:
          nvidia.com/gpu: "1"

KServe gives you request-driven scaling, canary rollouts, and a path to disaggregated topologies without hand-rolling the orchestration, at the cost of another layer to operate. The honest guidance holds: reach for any of this only when you have a measured reason, namely that prefill and decode contention is your bottleneck and the replica-scaled Deployment from §6 can no longer hold your latency targets. For most teams deploying vLLM Kubernetes workloads, §3 through §7 is the whole job, and §8 is where you go when the numbers say so.

Operational Pitfalls

The failures that take down a vLLM service are rarely the model. They are operational and predictable, and our example has already brushed against most of them.

Cold-start model downloads. A pod’s first action is pulling the model weights — roughly 16 GB for Llama-3-8B — from the model hub onto the node. With the emptyDir cache from §3, that repeats on every restart and every new node, and in our test it dominated the ~6-minute cold start. Back the cache with a PersistentVolume, or bake the model into the image, so a restart does not re-download. This is the single biggest contributor to the cold-start tax that made minReplicaCount: 1 non-negotiable in §6.

CUDA graph compilation. vLLM captures CUDA graphs at startup to speed up the steady state, and that capture takes time before the pod is ready — which is why §3’s probe pairs a short initialDelaySeconds with a large failureThreshold (we used 60 checks at 10-second intervals, a ten-minute window). If pods restart during startup, suspect the probe before you suspect vLLM.

The --enforce-eager tradeoff. Setting --enforce-eager skips CUDA graph capture, lowering memory use and shortening startup at the cost of somewhat slower per-token inference in the steady state. It is a reasonable lever when you are memory-constrained or churning pods frequently, but make it a deliberate choice, not a default you leave on.

V1 engine defaults. The V1 engine (the 2025 rearchitecture) changed defaults: prefix caching is on, torch.compile integration is enabled, and the engine loop was rebuilt for roughly a 2x throughput improvement. These are good defaults, but if you are following an older tutorial that toggles them by hand you can end up fighting the engine. Read the release notes for the tag you pinned; we ran v0.8.5 on the V1 engine, with prefix caching and torch.compile on by default exactly as described above.

Scale-up latency. The pitfalls above compound into the cold-start chain: GPU node provisioning (about six minutes, §2), image pull, the ~16 GB weight download, and CUDA graph capture. In our test that chain cost 6.3 minutes for the pod alone and about 11 minutes end to end with a first image pull on a fresh node, and every new replica pays some slice of it, on a traffic burst or a fresh node. The traffic that triggered the scale-up gets a multi-minute response instead of a sub-second one, and scaling to zero makes the first request after idle pay it in full. The choice is to keep warm replicas around (the minReplicaCount floor, which costs money) or eat the latency. Shrinking that chain is what ScaleOps targets next: pre-provisioned GPU node capacity so a scale-up does not wait out node provisioning, and a predictive warm replica floor that keeps models serving ahead of demand, complementing the minReplicaCount floor rather than replacing the scaling logic.

The Cost Problem: Right-Sizing GPU Inference

The last problem does not show up as an incident, only as a bill. Our vllm-llama3-8b replica requests nvidia.com/gpu: 1 and gets a whole L4, but inference traffic is bursty, and outside peak hours that GPU sits well under its capacity. This underuse is typical: even well-run clusters rarely push GPU utilization past 20–30%, because Kubernetes sees a GPU as atomic — 1 allocated, 0 available — with no concept of fractional use. A cluster running ten inference services, each using 15% of a GPU, needs ten physical GPUs at full price despite using the equivalent of 1.5.

Closing that gap means matching allocation to real utilization, but a vLLM server complicates the obvious move. As §4 showed, vLLM reserves its VRAM up front — roughly 23 GiB of the L4 held constantly, idle or busy — so external GPU-memory right-sizing has almost nothing to reclaim from a running vLLM server; the memory is committed by design. For a vLLM fleet, the real levers are running the right number of GPUs rather than reclaiming slices of each, and sizing --gpu-memory-utilization deliberately: high on a dedicated card to buy KV concurrency (§4), lower only when several servers intentionally share one. This is where ScaleOps’ GPU Platform fits, and it is the natural home for the per-pod signals from §5:

All of it works with the device plugin, the scheduler, and Cluster Autoscaler or Karpenter rather than replacing them — the existing primitives stay where they are and get better inputs. The honest boundaries: the fractional allocation is advisory at the scheduling layer (it prevents over-packing by declared fractions, not hardware isolation) and applies to single-GPU workloads, not the multi-GPU tensor-parallel pods from §7. For a vLLM fleet specifically, lead with replica and node scaling, and treat fractional memory sharing as the lever for the workloads around vLLM more than for the vLLM servers themselves.

There is also a token-level thread worth pulling. Prefix caching (§4) is vLLM reusing the KV blocks of repeated prompt prefixes, and on hardware you own a cache hit means prefill compute skipped rather than a discount on a per-token bill. Each skipped prefill frees GPU time for other requests, so a high hit rate shows up first as more throughput per replica and lower time-to-first-token, and it turns into money when it lets the replica scaling from §6 hold the same traffic with fewer replicas. That chain is why the prefix-cache hit rate you monitor in §5 belongs on the cost dashboard as much as the latency one. The same token-level lens is where ScaleOps is heading with Agent Cost Intelligence for teams running agentic workloads, where each loop carries the full conversation forward and API token spend compounds silently, extending per-pod cost down to per-agent token spend and budget control. For a self-hosted vLLM service, though, the win is concrete and local: right-size the GPU, keep the model warm, and watch the prefix cache do its job.

The honest summary of all of this: deploying vLLM on Kubernetes is, mechanically, the easy 80% — a Deployment behind a Service, a GPU request, KEDA on vllm:num_requests_waiting, and the cluster autoscaler for nodes, all of which we ran end to end on GKE with L4 and A100 GPUs. The other 20% is the operational detail that decides whether it survives production: the DCGM exporter is not a drop-in on managed Kubernetes, per-pod GPU attribution silently breaks without honorLabels, vLLM holds its VRAM by design so memory savings come from replica scaling rather than fractional memory, GPU stockouts will leave autoscaled pods Pending, and the scale-out paths carry hard prerequisites — a bigger /dev/shm for NCCL, and RDMA plus multiple GPUs for llm-d. You can get the 80% running in an afternoon; budget for the 20%.

ScaleOps AI Infra installs with a single Helm flag and plugs into the KEDA and node-autoscaling setup from §6 without rearchitecture — same Deployment, same Service, same /metrics.

Try ScaleOps free → to see how much of your inference GPU footprint is allocated but idle before you change a single pod spec.

Book a demo → to walk through fractional GPU allocation and GPU rightsizing against your own vLLM workloads with our team.

vLLM Kubernetes Troubleshooting

Most vLLM Kubernetes failures are operational rather than model bugs, and they cluster into a handful of recognizable symptoms. This table consolidates the ones covered above into a quick symptom-to-fix reference.

vLLM on Kubernetes: Quick Reference