Kubernetes 1.36 Resource Management: The Release Where the Defaults Flip

Workload-Aware Preemption arrives, DRA goes default-on, PSI metrics graduate to GA, and pod-level resources finally get a Beta. Here’s what actually changed in Kubernetes 1.36 resource management — with commands and test results from a kubeadm cluster on GCE.

Kubernetes 1.36 resource management is the release where six previously-opt-in features flip to default-on across the device, workload, runtime, and observability layers — closing the integer-GPU plumbing gap, fixing the partial-preemption failure mode for distributed training, and surfacing kernel-level pressure signals as first-class data on the kubelet Summary API.

TL;DR. Kubernetes 1.36 resource management is less about brand-new mechanics and more about the defaults catching up to two years of accumulated AI workload scar tissue. Five resource-management features that previously hid behind feature gates now ship enabled by default, one genuinely new alpha — Workload-Aware Preemption (KEP-5710) — finally treats AI training PodGroups as one preemption unit, and PSI metrics graduate to GA with kubelet-driven node conditions on top. Read the 1.34 release blog and the 1.35 deep dive for the primitives and semantics Kubernetes 1.36 resource management builds on; this post is the operational story of what 1.36 turns on without asking you.

Every cell in the rightmost column lands the moment you switch the API server to 1.36. Most of them used to require an explicit --feature-gates flag.

What Is Kubernetes 1.36 Resource Management?

Kubernetes 1.36 resource management is the cluster-side surface for declaring, scheduling, enforcing, and observing CPU, memory, hugepages, ephemeral storage, and DRA-managed devices on Pods and containers. In 1.36 it spans five primitives that all moved at once: pod-level spec.resources (Beta), DRA partitionable devices and consumable capacity (Beta default-on), DRA device taints and resource health status (Beta default-on), Workload-Aware Preemption over PodGroups (Alpha), and PSI metrics (GA). The traditional building blocks — requests vs limits, QoS classes (Guaranteed, Burstable, BestEffort), Node Allocatable, ResourceQuota, and LimitRange for namespace-level resource governance — still apply unchanged; 1.36 adds new fields on top of them, and on top of the kubelet eviction manager that watches MemoryPressure, DiskPressure, and PIDPressure node conditions to trigger node-pressure eviction. The kubelet’s Topology Manager, CPU Manager, and Memory Manager continue to coordinate NUMA-aware placement and cgroup-level pinning beneath all of this. Failure modes (OOMKilled when a container exceeds its memory limit, CPU throttling under CFS quota) are unchanged. The autoscaling control loop on top — horizontal pod autoscaling (HPA), vertical pod autoscaling (VPA), and node autoscaling via Cluster Autoscaler or Karpenter — consumes the new surfaces but is also unchanged. Operationally, you scrape the new endpoints with Prometheus or metrics-server.

1. The Defaults Flip — What Actually Changed

Kubernetes resource management features have a familiar life cycle: alpha behind a gate for two or three releases, beta but still opt-in for one more cycle, and then default-on. That last step — when a feature stops being something you have to ask for — is when most clusters actually start using it.

Kubernetes 1.36 is that moment for an unusually large chunk of the resource-management surface. Six features hit “default-on Beta” or “GA” simultaneously, and they happen to be the six that mattered most for AI workloads. Most of the 1.36 release notes for resource management are not about new mechanics. They are about defaults.

We’ll walk the four-layer stack from device → workload → runtime → observability, classifying each feature as either brand-new alpha (KEP-5710 + KEP-5304), first-time default-on (Partitionable, Consumable, Taints, Health, Pod-Level Resources), or GA promotion that changes the operational story (PSI). Every demo runs on a real kubeadm 1.36.0 cluster on GCE — outputs are verbatim, gotchas are documented, and where vendor support hasn’t caught up to the K8s spec we say so plainly.

2. Devices — DRA goes on by default

Four DRA features that previously hid behind feature gates ship enabled in 1.36. Together they take the integer-GPU device-plugin model — nvidia.com/gpu: 1 and you own the whole card whether you use 5% of it or 95% — and replace it with primitives that express how modern accelerators are partitioned, shared, tainted, and recovered when they fail.

2.1 DRA Bring-Up Gotchas on kubeadm 1.36

Eleven setup gotchas bit us across the full article. Numbered globally so cross-references stay clear:

Each callout below uses the global Gotcha #N label. Six of the eleven cluster around DRA bring-up:

After bootstrap, the chart registers six DeviceClasses (gpu.nvidia.com, mig.nvidia.com, vfio.gpu.nvidia.com, plus three compute-domain classes) and a single ResourceSlice per node with rich device attributes — productName: NVIDIA L4, architecture: Ada Lovelace, cudaComputeCapability: 8.9.0, capacity.memory: 23034Mi, plus the full PCI bus ID and device UUID. The level of self-description is substantially richer than what the integer nvidia.com/gpu: 1 extended resource exposed under the device plugin model.

2.2 DeviceTaintRule (KEP-5055)

Same operational primitive as node taints, applied one level lower. When a GPU starts throwing ECC errors, taint it; the workloads that don’t tolerate it get evicted; a diagnostic pod can land on it on purpose.

apiVersion: resource.k8s.io/v1alpha3
kind: DeviceTaintRule
metadata: { name: gpu-fault }
spec:
  deviceSelector:
    driver: gpu.nvidia.com         # matches ResourceSlice.spec.driver
                                   # NOT deviceClassName
  taint:
    key: health.scaleops.io/test
    value: fault
    effect: NoSchedule

A pod with a ResourceClaim against gpu.nvidia.com and no toleration stays Pending. The diagnostic-pod escape hatch is a Pod-level toleration (Gotcha #5):

spec:
  tolerations:
  - key: health.scaleops.io/test
    operator: Equal
    value: fault
    effect: NoSchedule
  resourceClaims:
  - { name: gpu, resourceClaimName: diag-claim }

Delete the DeviceTaintRule and normal claims schedule again.

2.3 Resource Health Status (KEP-4680)

When the GPU is unhealthy but the pod is still running, Pod.status historically said nothing. 1.36 surfaces per-device health on each container’s status:

kubectl get pod g -o jsonpath='{.status.containerStatuses[*].allocatedResourcesStatus}' | jq

[{
  "name": "claim:g/gpu",
  "resources": [{
    "health": "Unknown",
    "resourceID": "k8s.gpu.nvidia.com/claim=396d6a00-...-gpu-0"
  }]
}]

The field populates and the JSON structure matches the KEP, but the health value stays at "Unknown" even when nvidia-persistenced is stopped on the host, and the value never transitions to "Unhealthy".

2.4 MPS sharing via GpuConfig

KEP-4815 Partitionable Devices needs MIG hardware to demo (A100/H100). On an L4 the closest equivalent is NVIDIA’s MPS sharing, expressed via GpuConfig in the ResourceClaim:

apiVersion: resource.k8s.io/v1
kind: ResourceClaim
spec:
  devices:
    requests:
    - name: "gpu"          # quoted (Gotcha #6)
      exactly: { deviceClassName: gpu.nvidia.com }
    config:
    - opaque:
        driver: gpu.nvidia.com
        parameters:
          apiVersion: resource.nvidia.com/v1beta1
          kind: GpuConfig
          sharing:
            strategy: MPS
            mpsConfig:
              defaultActiveThreadPercentage: 50
              defaultPinnedDeviceMemoryLimit: 10Gi

With featureGates.MPSSupport=true (Gotcha #4) the claim allocates and the NVIDIA MPS Control Daemon spawns automatically. Two pods sharing one MPS GPU need one shared ResourceClaim referenced by both — two separate exclusive claims compete for the device and the second stays Pending. The two scheduler contracts are distinct: MPS shares a single allocation across multiple consumer pods, whereas partitionable devices expose the underlying hardware as several independently allocatable ResourceSlice entries.

2.5 What 1.36 ships vs what we could validate on L4

Two of five rows are honest “different hardware” or “newer vendor” gaps — not failures on the K8s side. NVIDIA’s roadmap covers both.

2.6 Pick the right DRA model

3. Workloads — Gang Scheduling grows up

If you covered Gang Scheduling Alpha in the 1.35 deep-dive, the short version of what changed in 1.36 is: the Workload API graduates to Beta, PodGroupSpec gains a priorityClassName, and — the actual headline — KEP-5710 Workload-Aware Preemption lands as Alpha. For the first time, the kube-scheduler treats a PodGroup as a single unit when deciding who to evict to make room. No more zombie pods stranded after partial preemption.

3.1 Setup gotchas

--feature-gates=WorkloadAwarePreemption=true,GangScheduling=true,GenericWorkload=true,WorkloadWithJob=true
--runtime-config=scheduling.k8s.io/v1alpha2=true

Error from server (Forbidden): podgroups.scheduling.k8s.io "training-bogus"
  is forbidden: no PriorityClass with name this-does-not-exist was found

3.2 KEP-5710 — preempting a group as a group

Pre-1.36 preemption operates at pod scope. When a higher-priority pod cannot fit, the scheduler picks the smallest set of individual lower-priority pods to evict. That pattern works well for stateless web workloads, but it breaks distributed AI training where the eight ranks of a single job depend on each other and partial scheduling leaves the workload non-functional. The classic failure mode runs as follows: the scheduler preempts one pod of training A to make room for training B, training B still does not fit on the remaining capacity, and the remaining seven pods of training A sit idle holding GPUs because the gang policy refuses to run with seven of eight ranks. Neither workload makes progress.

Workload-Aware Preemption changes the unit of decision. With the gate on, the scheduler treats a PodGroup as one entity — it preempts whole low-priority PodGroups, never partial groups, and only after verifying the high-priority group can actually fit. If the new group cannot be placed even after preemption, no eviction occurs and the existing workloads continue to run undisturbed.

3.3 Demo — group-level preemption

Two PriorityClasses (training-low: 100, training-high: 1000), two PodGroups, identical resource shape. We size each gang at 4 ranks × 600m CPU so it fits on the 4 vCPU node alongside system pods (800m × 4 overcommitted).

apiVersion: scheduling.k8s.io/v1alpha2
kind: PodGroup
metadata: { name: training-a }
spec:
  priorityClassName: training-low
  disruptionMode: PodGroup
  schedulingPolicy:
    gang: { minCount: 4 }
---
apiVersion: batch/v1
kind: Job
metadata: { name: training-a }
spec:
  parallelism: 4
  template:
    spec:
      schedulingGroup:                # the new 1.36 first-class field
        podGroupName: training-a
      priorityClassName: training-low
      restartPolicy: Never
      containers:
      - { name: trainer, image: busybox:1.36, command: ["sh","-c","sleep 3600"],
          resources: { requests: { cpu: 600m, memory: 256Mi },
                       limits:   { cpu: 600m, memory: 256Mi } } }

training-b is the same YAML with name: training-b and priorityClassName: training-high. Apply A first, wait for 4/4 Running, submit B:

kubectl get events --sort-by=.lastTimestamp -n default | tail
45s   Normal   Preempted   pod/training-a-s8h5g
        Preempted by podgroup bda07530-da19-4aa9-957f-ad8ceb146e2c on node cluster
45s   Normal   Preempted   pod/training-a-c8hqm   ...same podgroup UID...
45s   Normal   Preempted   pod/training-a-h7fkt   ...same podgroup UID...
45s   Normal   Preempted   pod/training-a-bz5z9   ...same podgroup UID...

All four Preempted events at the same relative timestamp — batched group decision, not pod-by-pod. The message names the predator by PodGroup UID, not by individual pod. That’s the API change that tells you you’re on the new code path. training-b ends 4/4 Running on the same node within seconds.

3.5 Native Kubernetes Gang Scheduling vs Kueue, Volcano, and YuniKorn

Kubernetes 1.36 makes the in-tree scheduler competent at gang scheduling, which is a meaningful shift from the prior release where the same correctness required a third-party scheduler. It does not make Kubernetes sufficient for clusters where multiple teams contend for a GPU pool or where MPI training depends on NUMA-aware placement, and for those cases Kueue or Volcano continues to be the right layer to add alongside the native primitives.

4. Runtime — Pod-Level Resources Beta

If the 1.35 deep-dive was about container-scope in-place resize finally going GA, 1.36 adds the missing scope: the Pod itself. Pod.spec.resources is now a Beta-default field, alongside container-level spec.containers[*].resources. The two coexist; container-level wins for that container.

4.1 The pod-scope spec.resources field

apiVersion: v1
kind: Pod
metadata: { name: pod-only-resources }
spec:
  resources:
    requests: { cpu: 500m, memory: 512Mi }
    limits:   { cpu: 1000m, memory: 1Gi }
  containers:
  - { name: main,    image: busybox:1.36, command: ["sh","-c","sleep 3600"] }
  - { name: sidecar, image: busybox:1.36, command: ["sh","-c","sleep 3600"] }

Containers inherit; the kubelet writes the limits to the pod-level cgroup slice:

sudo cat .../kubepods-burstable-pod995970c9_..._86a0_..slice/cpu.max
100000 100000        # 100ms quota / 100ms period = 1.0 CPU

sudo cat .../memory.max
1073741824           # 1 GiB exactly

Container-level fields, when set, shadow (don’t replace) the pod-level for the specific container. Pod-scope only accepts cpu, memory, hugepages-*. Try to declare ephemeral-storage at pod scope:

The Pod "pod-bad-resources" is invalid:
  spec.resources.requests[ephemeral-storage]: Unsupported value: "ephemeral-storage":
  supported values: "cpu", "hugepages-", "memory"

Extended resources (GPUs, custom devices, ephemeral storage) stay container-scope.

4.2 In-place resize meets ResizeDeferred

kubectl --subresource=resize works against spec.resources. Shrink works in place (no restart). Grow on a contended 2 vCPU node:

kubectl get events | grep ResizeDeferred
4m30s   Warning   ResizeDeferred   pod/pod-only-resources
  Pod resize OutOfcpu: ...
  "error":"Node didn't have enough resource: cpu, requested: 750, used: 1450, capacity: 2000"

ResizeDeferred is the headline new event type. Pre-1.36, a grow without headroom either restarted the pod or silently went Infeasible. Now the pod stays Running at the old size, kubelet emits a structured event explaining exactly what’s missing, and keeps retrying. The new size eventually applies once Cluster Autoscaler or Karpenter frees capacity. .status.allocatedResources continues to reflect what’s actually reserved, so HPA, VPA, and node autoscaling all see the truth.

5. Observability — PSI Metrics go GA

Pressure Stall Information has been Beta in Kubernetes since 1.34, and the PSI research note we published explains why it’s the right leading metric for horizontal pod autoscaling decisions when your pods don’t have CPU limits. In 1.36, KEP-4205 graduates the kubelet PSI integration to GA. The data surface is wider than it was in Beta — two endpoints expose the same kernel signal.

5.1 The cAdvisor Prometheus endpoint

Same path as Beta, fully populated. Counters in seconds; differentiate with rate(...[1m]) and you get the share of the last minute that the cgroup spent waiting on each resource:

container_pressure_cpu_stalled_seconds_total{...,pod="psi-stress"}    0.159446
container_pressure_cpu_waiting_seconds_total{...,pod="psi-stress"}    0.161832
container_pressure_io_stalled_seconds_total{...,pod="psi-stress"}     0
container_pressure_memory_waiting_seconds_total{...,pod="psi-stress"} 0

5.2 Summary API — the new structured surface

The bigger 1.36 win is on the kubelet Summary API (/stats/summary). PSI is now a first-class object on every container and on the pod aggregate:

{
  "podRef": { "name": "psi-stress", "namespace": "default" },
  "containers": [{
    "name": "stress",
    "cpu": {
      "usageNanoCores": 200300201,
      "psi": {
        "full": { "avg10": 70.87, "avg60": 20.71, "avg300": 4.73, "total": 16114670 },
        "some": { "avg10": 73.9, "avg60": 21.59, "avg300": 4.93, "total": 16713107 }
      }
    },
    "memory": { "psi": { "full": { "avg10": 0, "..." : "..." }, "some": { "avg10": 0, "..." : "..." } } },
    "io":     { "psi": { "full": { "avg10": 0, "..." : "..." }, "some": { "avg10": 1.59, "avg60": 0.46, "..." : "..." } } }
  }]
}

That’s the full PSI tuple — some and full, three rolling averages (10s/60s/5min), cumulative total — as machine-readable JSON. Controllers that consume Summary API (metrics-server, custom HPA adapters, in-cluster autoscalers) get PSI as structured data, not a string-parsed Prometheus dump. cpu.psi.full.avg10 = 70.87 means all tasks in this cgroup were stalled on CPU 70.87% of the last 10 seconds. Unambiguous “this container is starved” signal — and one that distinguishes CFS quota throttling from real CPU starvation.

5.3 The thing that didn’t ship — yet

Honest finding the K8s release notes don’t surface: PSI metrics being GA does not mean PSI-driven node conditions are on. During sustained stress we watched the node:

kubectl describe node | sed -n '/Conditions:/,/Addresses/p'
Conditions:
  Type                 Status  Reason                       Message
  NetworkUnavailable   False   FlannelIsUp                  Flannel is running
  MemoryPressure       False   KubeletHasSufficientMemory   ...
  DiskPressure         False   KubeletHasNoDiskPressure     ...
  PIDPressure          False   KubeletHasSufficientPID      ...
  Ready                True    KubeletReady                 ...

No CPUPressure or IOPressure node conditions appeared, even under sustained load. The kubelet configuration exposed only evictionPressureTransitionPeriod — no psiCpuThreshold, no psiIoEvictionThreshold, and no other field that would wire PSI signals into the existing eviction manager. The auto-tainting and PSI-driven eviction layer on top of these metrics has not shipped in 1.36, which means the kubelet continues to publish the same node-pressure conditions it always did and continues to evict on memory, disk, and PID pressure but not on the CPU or I/O pressure measured by PSI.

Cluster operators that want to act on PSI in production today implement the control loop themselves — scraping the Summary API or the /metrics/cadvisor endpoint, evaluating thresholds against fields such as cpu.psi.full.avg10 or io.psi.some.avg60, and routing the resulting signal into whichever horizontal pod autoscaling, vertical pod autoscaling, or rightsizing pipeline already runs on the cluster. The metrics surface is production-ready in Kubernetes 1.36; the response pipeline that consumes those metrics remains an external concern for the operator or the policy controller layered above the kubelet.

6. KEP-5304: DRA Device Attributes Downward API in Kubernetes 1.36

KEP-5304 introduces a mechanism by which DRA drivers can hand device metadata directly to a workload by mounting a JSON file at /var/run/dra-device-attributes/<claim>/<request>/<driver>-metadata.json inside the container, removing the need for the sidecar pattern that today reads ResourceSlice.spec.attributes from the Kubernetes API. The use cases this targets include KubeVirt-style workloads that need PCI bus identifiers, SR-IOV network interface cards that need link metadata, and any other workload that currently runs an in-pod controller to extract device attributes from the cluster API.

The feature is Alpha in Kubernetes 1.36 and the NVIDIA DRA driver chart 25.12.0 does not yet implement it on the driver side:

kubectl exec g -- ls /var/run/dra-device-attributes/
ls: cannot access '/var/run/dra-device-attributes': No such file or directory

The Kubernetes framework is in place — the kubelet code path that mounts the metadata file is present and the alpha API has been merged — but the vendor driver has not yet implemented its side of the contract. Until a future NVIDIA driver release adds the metadata mount, applications that need device attributes continue to read ResourceSlice.spec.attributes from the Kubernetes API through a sidecar.

7. How ScaleOps Extends the 1.36 Primitives

Every section above ends with K8s shipping a primitive — partition advertising, gang correctness, pod-scope resize, leading-indicator pressure metrics — and acknowledging that the policy layer above them is left for someone else to build. That is the seam this article is really about. Kubernetes 1.36 resource management gives you better inputs to existing tools; it does not change the outputs. Cluster Autoscaler, Karpenter, the default scheduler, horizontal and vertical pod autoscaling, KEDA — all stay in place and operate as before, now with richer signals to act on. ScaleOps slots into that same layer, working with those native primitives rather than replacing any of them, and treating each new 1.36 surface as an input to its policy decisions.

The pattern is the same across every row: Kubernetes 1.36 resource management ships primitives, ScaleOps runs the policy on top. Roku used the combination of Kubernetes-native primitives plus ScaleOps policy across 110+ Kubernetes clusters and reduced vCPU consumption by 32% at 20% rollout coverage (Roku case study); the 1.36 primitives make that pattern more accurate, not redundant. ScaleOps works with your existing Cluster Autoscaler, Karpenter, default scheduler, horizontal pod autoscaling, and KEDA configuration — no migration, no rearchitecture.

Try ScaleOps free → See where the 1.36 primitives land in your own cluster — partition advertising, PSI signals, pod-scope resources — and where the policy layer above them is still doing nothing.

Book a demo → Walk through how the same defaults-flip features map to your AI training and inference workloads with our team, and see what GPU Optimization, Smart Pod Placement, and Automated Pod Rightsizing change once the new K8s primitives are feeding them.

9. KEPs and Sources for Kubernetes 1.36 Resource Management

KEPs covered: