GPU Scheduling on Kubernetes with Kueue

You have a Kubernetes cluster with GPUs. Multiple teams want to use them. What happens next is predictable: someone submits a large training job that grabs all the GPUs, and everyone else waits. Or worse, teams learn to hoard resources by keeping idle jobs running "just in case."

This is a scheduling problem, and Kubernetes alone doesn't solve it well. The default scheduler is first-come, first-served. It doesn't understand quotas, priorities, or fairness. It just assigns Pods to nodes that have the requested resources available.

Kueue fills this gap. It's a Kubernetes-native job queueing system built by the Kubernetes community specifically for batch workloads like ML training.

The problem in concrete terms

Say you have a cluster with 8 GPUs and two teams:

Team A (ML research) runs lots of experimental training jobs. They'd happily use all 8 GPUs if they could.
Team B (production ML) runs weekly retraining jobs. These are critical — if they don't run, production models go stale.

Without Kueue, the scenario plays out like this:

Team A submits 8 single-GPU training jobs on Monday morning.
All 8 GPUs are allocated.
Team B's production retraining job arrives Tuesday. It can't be scheduled — all GPUs are busy.
Team B's job sits in Pending state for hours until one of Team A's experiments finishes.
Team B complains. A policy meeting is scheduled. Nothing gets resolved.

Kueue solves this with three concepts: ResourceFlavors, ClusterQueues, and LocalQueues.

How Kueue works

ResourceFlavors

A ResourceFlavor describes a type of resource in your cluster. For GPUs, this maps to the actual hardware:

apiVersion: kueue.x-k8s.io/v1beta1
kind: ResourceFlavor
metadata:
  name: gpu-a100
spec:
  nodeLabels:
    gpu-type: a100

This tells Kueue: "there's a class of resources called gpu-a100 that lives on nodes labeled gpu-type: a100." If you have mixed GPU types (some A100s, some T4s), you'd create separate flavors for each.

ClusterQueues

A ClusterQueue defines a resource budget. This is where quotas live:

apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
  name: team-a-queue
spec:
  cohort: gpu-cluster
  resourceGroups:
  - coveredResources: ["cpu", "memory", "nvidia.com/gpu"]
    flavors:
    - name: gpu-a100
      resources:
      - name: "nvidia.com/gpu"
        nominalQuota: 4
      - name: "cpu"
        nominalQuota: 32
      - name: "memory"
        nominalQuota: 128Gi

apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
  name: team-b-queue
spec:
  cohort: gpu-cluster
  resourceGroups:
  - coveredResources: ["cpu", "memory", "nvidia.com/gpu"]
    flavors:
    - name: gpu-a100
      resources:
      - name: "nvidia.com/gpu"
        nominalQuota: 4
      - name: "cpu"
        nominalQuota: 32
      - name: "memory"
        nominalQuota: 128Gi

Each team gets a nominal quota of 4 GPUs. The cohort field groups them together — more on that shortly.

LocalQueues

A LocalQueue is a namespaced queue that points to a ClusterQueue. It's how users actually submit jobs:

apiVersion: kueue.x-k8s.io/v1beta1
kind: LocalQueue
metadata:
  name: training-queue
  namespace: team-a
spec:
  clusterQueue: team-a-queue

apiVersion: kueue.x-k8s.io/v1beta1
kind: LocalQueue
metadata:
  name: training-queue
  namespace: team-b
spec:
  clusterQueue: team-b-queue

Team A submits jobs to training-queue in their namespace. Kueue routes them to team-a-queue and enforces the quota.

Submitting jobs

Jobs opt into Kueue by adding a label:

apiVersion: batch/v1
kind: Job
metadata:
  name: experiment-42
  namespace: team-a
  labels:
    kueue.x-k8s.io/queue-name: training-queue
spec:
  template:
    spec:
      containers:
      - name: train
        image: training:latest
        resources:
          requests:
            nvidia.com/gpu: 1
      restartPolicy: Never

When this Job is created, Kueue intercepts it. Instead of being scheduled immediately, it's suspended and placed in the queue. Kueue checks:

Does team-a-queue have available quota for 1 GPU?
Is there an actual GPU available in the cluster?

If both are true, Kueue unsuspends the Job and it runs. If not, it waits.

This is the key difference from vanilla Kubernetes. Without Kueue, the Job would be created and the Pod would sit in Pending state, occupying a "slot" that other Jobs can't use. With Kueue, the Job is explicitly queued and managed.

The cohort field on ClusterQueues enables borrowing. When Team B isn't using their 4 GPUs, Team A can borrow them — and vice versa.

spec:
  cohort: gpu-cluster
  resourceGroups:
  - coveredResources: ["cpu", "memory", "nvidia.com/gpu"]
    flavors:
    - name: gpu-a100
      resources:
      - name: "nvidia.com/gpu"
        nominalQuota: 4
        borrowingLimit: 4

The borrowingLimit: 4 means Team A can borrow up to 4 additional GPUs from the cohort (the full cluster's worth) when others aren't using them.

The behavior:

Both teams idle: 8 GPUs available for whoever submits first.
Team A using 6 GPUs, Team B submits: Team A's jobs beyond their nominal quota (4) get preempted to make room for Team B's quota.
Team A using 4 GPUs, Team B using 2: Team A can borrow Team B's 2 unused GPUs, using 6 total.

This is fair sharing. Resources don't sit idle just because their "owner" isn't using them, but guaranteed quotas are respected when demand appears.

Priorities

Not all jobs are equal. Production retraining should preempt experimental runs. Kueue supports this with WorkloadPriorityClasses:

apiVersion: kueue.x-k8s.io/v1beta1
kind: WorkloadPriorityClass
metadata:
  name: production
value: 1000
description: "Production retraining jobs"
---
apiVersion: kueue.x-k8s.io/v1beta1
kind: WorkloadPriorityClass
metadata:
  name: experiment
value: 100
description: "Experimental training runs"

Jobs reference the priority:

apiVersion: batch/v1
kind: Job
metadata:
  name: weekly-retrain
  namespace: team-b
  labels:
    kueue.x-k8s.io/queue-name: training-queue
    kueue.x-k8s.io/priority-class: production
spec:
  template:
    spec:
      containers:
      - name: train
        image: training:latest
        resources:
          requests:
            nvidia.com/gpu: 2
      restartPolicy: Never

When a high-priority job arrives and there aren't enough resources, Kueue can preempt lower-priority jobs to make room. The preempted jobs go back into the queue and run when resources free up.

Preemption policies

Kueue's preemption is configurable per ClusterQueue:

spec:
  preemption:
    reclaimWithinCohort: Any
    withinClusterQueue: LowerPriority

reclaimWithinCohort: Any — reclaim borrowed resources from other queues in the cohort.
withinClusterQueue: LowerPriority — within the same queue, preempt lower-priority jobs.

This gives you fine-grained control. You might want production jobs to preempt experiments, but not to preempt other production jobs.

Putting it all together

Here's the full setup for our two-team scenario:

# The GPU flavor
apiVersion: kueue.x-k8s.io/v1beta1
kind: ResourceFlavor
metadata:
  name: gpu-a100
spec:
  nodeLabels:
    gpu-type: a100
---
# Team A: 4 GPU quota, can borrow 4 more
apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
  name: team-a-queue
spec:
  cohort: gpu-cluster
  preemption:
    reclaimWithinCohort: Any
    withinClusterQueue: LowerPriority
  resourceGroups:
  - coveredResources: ["cpu", "memory", "nvidia.com/gpu"]
    flavors:
    - name: gpu-a100
      resources:
      - name: "nvidia.com/gpu"
        nominalQuota: 4
        borrowingLimit: 4
      - name: "cpu"
        nominalQuota: 32
      - name: "memory"
        nominalQuota: 128Gi
---
# Team B: 4 GPU quota, can borrow 4 more
apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
  name: team-b-queue
spec:
  cohort: gpu-cluster
  preemption:
    reclaimWithinCohort: Any
    withinClusterQueue: LowerPriority
  resourceGroups:
  - coveredResources: ["cpu", "memory", "nvidia.com/gpu"]
    flavors:
    - name: gpu-a100
      resources:
      - name: "nvidia.com/gpu"
        nominalQuota: 4
        borrowingLimit: 4
      - name: "cpu"
        nominalQuota: 32
      - name: "memory"
        nominalQuota: 128Gi
---
# LocalQueues in each team's namespace
apiVersion: kueue.x-k8s.io/v1beta1
kind: LocalQueue
metadata:
  name: training-queue
  namespace: team-a
spec:
  clusterQueue: team-a-queue
---
apiVersion: kueue.x-k8s.io/v1beta1
kind: LocalQueue
metadata:
  name: training-queue
  namespace: team-b
spec:
  clusterQueue: team-b-queue

Apply with kubectl apply -f kueue-setup.yaml, and both teams can submit jobs to their respective training-queue LocalQueues. Kueue handles the rest.

Monitoring Kueue

Kueue exposes Prometheus metrics out of the box:

kueue_pending_workloads — how many jobs are waiting in each queue
kueue_admitted_active_workloads — how many jobs are currently running
kueue_cluster_queue_resource_usage — current resource consumption per queue

These tell you whether your quotas are right-sized. If kueue_pending_workloads is consistently high for one queue, that team needs more quota (or the cluster needs more GPUs). If kueue_cluster_queue_resource_usage is consistently low, resources are being wasted.

When to use Kueue vs. simpler approaches

Use Kueue when:

Multiple teams or users share GPU resources.
You need quotas to prevent one team from monopolizing the cluster.
Production jobs need to preempt experimental work.
You want fair sharing — idle resources should be usable by anyone, but guaranteed quotas should be respected.

You might not need Kueue when:

One team, one GPU. Just submit Jobs directly.
All jobs have the same priority and there's no contention. First-come, first-served works fine.
You're using a managed service (GKE Autopilot, AWS Batch) that handles scheduling for you.

Kueue adds operational complexity. It's worth it when contention for GPUs is a real problem. If you're not fighting over resources, plain Kubernetes Jobs are simpler.

GPU scheduling is one of the trickier parts of on-prem ML infrastructure. If you're setting up shared GPU clusters and need help with Kueue, quotas, or cluster architecture, deploying.ai does this work — let's talk.