[{"data":1,"prerenderedAt":1315},["ShallowReactive",2],{"article-gpu-scheduling-kubernetes-kueue":3,"content-query-W0HzCZx4B1":741},{"_path":4,"_dir":5,"_draft":6,"_partial":6,"_locale":7,"title":8,"description":9,"date":10,"body":11,"_type":735,"_id":736,"_source":737,"_file":738,"_stem":739,"_extension":740},"\u002Farticles\u002Fgpu-scheduling-kubernetes-kueue","articles",false,"","GPU Scheduling on Kubernetes with Kueue","GPUs are expensive and shared. Kueue is a Kubernetes-native job queueing system that manages quotas, priorities, and fair sharing so teams can share a GPU cluster without stepping on each other.","2025-07-18",{"type":12,"children":13,"toc":717},"root",[14,22,28,33,47,54,59,85,90,119,145,151,157,162,175,196,201,206,215,224,237,242,247,256,265,286,292,297,306,311,331,336,341,347,359,368,380,385,418,423,429,434,443,448,457,462,468,473,482,513,518,524,529,538,558,564,569,605,624,630,638,661,669,687,692,696],{"type":15,"tag":16,"props":17,"children":19},"element","h1",{"id":18},"gpu-scheduling-on-kubernetes-with-kueue",[20],{"type":21,"value":8},"text",{"type":15,"tag":23,"props":24,"children":25},"p",{},[26],{"type":21,"value":27},"You have a Kubernetes cluster with GPUs. Multiple teams want to use them. What happens next is predictable: someone submits a large training job that grabs all the GPUs, and everyone else waits. Or worse, teams learn to hoard resources by keeping idle jobs running \"just in case.\"",{"type":15,"tag":23,"props":29,"children":30},{},[31],{"type":21,"value":32},"This is a scheduling problem, and Kubernetes alone doesn't solve it well. The default scheduler is first-come, first-served. It doesn't understand quotas, priorities, or fairness. It just assigns Pods to nodes that have the requested resources available.",{"type":15,"tag":23,"props":34,"children":35},{},[36,45],{"type":15,"tag":37,"props":38,"children":42},"a",{"href":39,"rel":40},"https:\u002F\u002Fkueue.sigs.k8s.io\u002F",[41],"nofollow",[43],{"type":21,"value":44},"Kueue",{"type":21,"value":46}," fills this gap. It's a Kubernetes-native job queueing system built by the Kubernetes community specifically for batch workloads like ML training.",{"type":15,"tag":48,"props":49,"children":51},"h2",{"id":50},"the-problem-in-concrete-terms",[52],{"type":21,"value":53},"The problem in concrete terms",{"type":15,"tag":23,"props":55,"children":56},{},[57],{"type":21,"value":58},"Say you have a cluster with 8 GPUs and two teams:",{"type":15,"tag":60,"props":61,"children":62},"ul",{},[63,75],{"type":15,"tag":64,"props":65,"children":66},"li",{},[67,73],{"type":15,"tag":68,"props":69,"children":70},"strong",{},[71],{"type":21,"value":72},"Team A",{"type":21,"value":74}," (ML research) runs lots of experimental training jobs. They'd happily use all 8 GPUs if they could.",{"type":15,"tag":64,"props":76,"children":77},{},[78,83],{"type":15,"tag":68,"props":79,"children":80},{},[81],{"type":21,"value":82},"Team B",{"type":21,"value":84}," (production ML) runs weekly retraining jobs. These are critical — if they don't run, production models go stale.",{"type":15,"tag":23,"props":86,"children":87},{},[88],{"type":21,"value":89},"Without Kueue, the scenario plays out like this:",{"type":15,"tag":91,"props":92,"children":93},"ol",{},[94,99,104,109,114],{"type":15,"tag":64,"props":95,"children":96},{},[97],{"type":21,"value":98},"Team A submits 8 single-GPU training jobs on Monday morning.",{"type":15,"tag":64,"props":100,"children":101},{},[102],{"type":21,"value":103},"All 8 GPUs are allocated.",{"type":15,"tag":64,"props":105,"children":106},{},[107],{"type":21,"value":108},"Team B's production retraining job arrives Tuesday. It can't be scheduled — all GPUs are busy.",{"type":15,"tag":64,"props":110,"children":111},{},[112],{"type":21,"value":113},"Team B's job sits in Pending state for hours until one of Team A's experiments finishes.",{"type":15,"tag":64,"props":115,"children":116},{},[117],{"type":21,"value":118},"Team B complains. A policy meeting is scheduled. Nothing gets resolved.",{"type":15,"tag":23,"props":120,"children":121},{},[122,124,129,131,136,138,143],{"type":21,"value":123},"Kueue solves this with three concepts: ",{"type":15,"tag":68,"props":125,"children":126},{},[127],{"type":21,"value":128},"ResourceFlavors",{"type":21,"value":130},", ",{"type":15,"tag":68,"props":132,"children":133},{},[134],{"type":21,"value":135},"ClusterQueues",{"type":21,"value":137},", and ",{"type":15,"tag":68,"props":139,"children":140},{},[141],{"type":21,"value":142},"LocalQueues",{"type":21,"value":144},".",{"type":15,"tag":48,"props":146,"children":148},{"id":147},"how-kueue-works",[149],{"type":21,"value":150},"How Kueue works",{"type":15,"tag":152,"props":153,"children":155},"h3",{"id":154},"resourceflavors",[156],{"type":21,"value":128},{"type":15,"tag":23,"props":158,"children":159},{},[160],{"type":21,"value":161},"A ResourceFlavor describes a type of resource in your cluster. For GPUs, this maps to the actual hardware:",{"type":15,"tag":163,"props":164,"children":169},"pre",{"className":165,"code":167,"language":168,"meta":7},[166],"language-yaml","apiVersion: kueue.x-k8s.io\u002Fv1beta1\nkind: ResourceFlavor\nmetadata:\n  name: gpu-a100\nspec:\n  nodeLabels:\n    gpu-type: a100\n","yaml",[170],{"type":15,"tag":171,"props":172,"children":173},"code",{"__ignoreMap":7},[174],{"type":21,"value":167},{"type":15,"tag":23,"props":176,"children":177},{},[178,180,186,188,194],{"type":21,"value":179},"This tells Kueue: \"there's a class of resources called ",{"type":15,"tag":171,"props":181,"children":183},{"className":182},[],[184],{"type":21,"value":185},"gpu-a100",{"type":21,"value":187}," that lives on nodes labeled ",{"type":15,"tag":171,"props":189,"children":191},{"className":190},[],[192],{"type":21,"value":193},"gpu-type: a100",{"type":21,"value":195},".\" If you have mixed GPU types (some A100s, some T4s), you'd create separate flavors for each.",{"type":15,"tag":152,"props":197,"children":199},{"id":198},"clusterqueues",[200],{"type":21,"value":135},{"type":15,"tag":23,"props":202,"children":203},{},[204],{"type":21,"value":205},"A ClusterQueue defines a resource budget. This is where quotas live:",{"type":15,"tag":163,"props":207,"children":210},{"className":208,"code":209,"language":168,"meta":7},[166],"apiVersion: kueue.x-k8s.io\u002Fv1beta1\nkind: ClusterQueue\nmetadata:\n  name: team-a-queue\nspec:\n  cohort: gpu-cluster\n  resourceGroups:\n  - coveredResources: [\"cpu\", \"memory\", \"nvidia.com\u002Fgpu\"]\n    flavors:\n    - name: gpu-a100\n      resources:\n      - name: \"nvidia.com\u002Fgpu\"\n        nominalQuota: 4\n      - name: \"cpu\"\n        nominalQuota: 32\n      - name: \"memory\"\n        nominalQuota: 128Gi\n",[211],{"type":15,"tag":171,"props":212,"children":213},{"__ignoreMap":7},[214],{"type":21,"value":209},{"type":15,"tag":163,"props":216,"children":219},{"className":217,"code":218,"language":168,"meta":7},[166],"apiVersion: kueue.x-k8s.io\u002Fv1beta1\nkind: ClusterQueue\nmetadata:\n  name: team-b-queue\nspec:\n  cohort: gpu-cluster\n  resourceGroups:\n  - coveredResources: [\"cpu\", \"memory\", \"nvidia.com\u002Fgpu\"]\n    flavors:\n    - name: gpu-a100\n      resources:\n      - name: \"nvidia.com\u002Fgpu\"\n        nominalQuota: 4\n      - name: \"cpu\"\n        nominalQuota: 32\n      - name: \"memory\"\n        nominalQuota: 128Gi\n",[220],{"type":15,"tag":171,"props":221,"children":222},{"__ignoreMap":7},[223],{"type":21,"value":218},{"type":15,"tag":23,"props":225,"children":226},{},[227,229,235],{"type":21,"value":228},"Each team gets a nominal quota of 4 GPUs. The ",{"type":15,"tag":171,"props":230,"children":232},{"className":231},[],[233],{"type":21,"value":234},"cohort",{"type":21,"value":236}," field groups them together — more on that shortly.",{"type":15,"tag":152,"props":238,"children":240},{"id":239},"localqueues",[241],{"type":21,"value":142},{"type":15,"tag":23,"props":243,"children":244},{},[245],{"type":21,"value":246},"A LocalQueue is a namespaced queue that points to a ClusterQueue. It's how users actually submit jobs:",{"type":15,"tag":163,"props":248,"children":251},{"className":249,"code":250,"language":168,"meta":7},[166],"apiVersion: kueue.x-k8s.io\u002Fv1beta1\nkind: LocalQueue\nmetadata:\n  name: training-queue\n  namespace: team-a\nspec:\n  clusterQueue: team-a-queue\n",[252],{"type":15,"tag":171,"props":253,"children":254},{"__ignoreMap":7},[255],{"type":21,"value":250},{"type":15,"tag":163,"props":257,"children":260},{"className":258,"code":259,"language":168,"meta":7},[166],"apiVersion: kueue.x-k8s.io\u002Fv1beta1\nkind: LocalQueue\nmetadata:\n  name: training-queue\n  namespace: team-b\nspec:\n  clusterQueue: team-b-queue\n",[261],{"type":15,"tag":171,"props":262,"children":263},{"__ignoreMap":7},[264],{"type":21,"value":259},{"type":15,"tag":23,"props":266,"children":267},{},[268,270,276,278,284],{"type":21,"value":269},"Team A submits jobs to ",{"type":15,"tag":171,"props":271,"children":273},{"className":272},[],[274],{"type":21,"value":275},"training-queue",{"type":21,"value":277}," in their namespace. Kueue routes them to ",{"type":15,"tag":171,"props":279,"children":281},{"className":280},[],[282],{"type":21,"value":283},"team-a-queue",{"type":21,"value":285}," and enforces the quota.",{"type":15,"tag":48,"props":287,"children":289},{"id":288},"submitting-jobs",[290],{"type":21,"value":291},"Submitting jobs",{"type":15,"tag":23,"props":293,"children":294},{},[295],{"type":21,"value":296},"Jobs opt into Kueue by adding a label:",{"type":15,"tag":163,"props":298,"children":301},{"className":299,"code":300,"language":168,"meta":7},[166],"apiVersion: batch\u002Fv1\nkind: Job\nmetadata:\n  name: experiment-42\n  namespace: team-a\n  labels:\n    kueue.x-k8s.io\u002Fqueue-name: training-queue\nspec:\n  template:\n    spec:\n      containers:\n      - name: train\n        image: training:latest\n        resources:\n          requests:\n            nvidia.com\u002Fgpu: 1\n      restartPolicy: Never\n",[302],{"type":15,"tag":171,"props":303,"children":304},{"__ignoreMap":7},[305],{"type":21,"value":300},{"type":15,"tag":23,"props":307,"children":308},{},[309],{"type":21,"value":310},"When this Job is created, Kueue intercepts it. Instead of being scheduled immediately, it's suspended and placed in the queue. Kueue checks:",{"type":15,"tag":91,"props":312,"children":313},{},[314,326],{"type":15,"tag":64,"props":315,"children":316},{},[317,319,324],{"type":21,"value":318},"Does ",{"type":15,"tag":171,"props":320,"children":322},{"className":321},[],[323],{"type":21,"value":283},{"type":21,"value":325}," have available quota for 1 GPU?",{"type":15,"tag":64,"props":327,"children":328},{},[329],{"type":21,"value":330},"Is there an actual GPU available in the cluster?",{"type":15,"tag":23,"props":332,"children":333},{},[334],{"type":21,"value":335},"If both are true, Kueue unsuspends the Job and it runs. If not, it waits.",{"type":15,"tag":23,"props":337,"children":338},{},[339],{"type":21,"value":340},"This is the key difference from vanilla Kubernetes. Without Kueue, the Job would be created and the Pod would sit in Pending state, occupying a \"slot\" that other Jobs can't use. With Kueue, the Job is explicitly queued and managed.",{"type":15,"tag":48,"props":342,"children":344},{"id":343},"fair-sharing-with-cohorts",[345],{"type":21,"value":346},"Fair sharing with cohorts",{"type":15,"tag":23,"props":348,"children":349},{},[350,352,357],{"type":21,"value":351},"The ",{"type":15,"tag":171,"props":353,"children":355},{"className":354},[],[356],{"type":21,"value":234},{"type":21,"value":358}," field on ClusterQueues enables borrowing. When Team B isn't using their 4 GPUs, Team A can borrow them — and vice versa.",{"type":15,"tag":163,"props":360,"children":363},{"className":361,"code":362,"language":168,"meta":7},[166],"spec:\n  cohort: gpu-cluster\n  resourceGroups:\n  - coveredResources: [\"cpu\", \"memory\", \"nvidia.com\u002Fgpu\"]\n    flavors:\n    - name: gpu-a100\n      resources:\n      - name: \"nvidia.com\u002Fgpu\"\n        nominalQuota: 4\n        borrowingLimit: 4\n",[364],{"type":15,"tag":171,"props":365,"children":366},{"__ignoreMap":7},[367],{"type":21,"value":362},{"type":15,"tag":23,"props":369,"children":370},{},[371,372,378],{"type":21,"value":351},{"type":15,"tag":171,"props":373,"children":375},{"className":374},[],[376],{"type":21,"value":377},"borrowingLimit: 4",{"type":21,"value":379}," means Team A can borrow up to 4 additional GPUs from the cohort (the full cluster's worth) when others aren't using them.",{"type":15,"tag":23,"props":381,"children":382},{},[383],{"type":21,"value":384},"The behavior:",{"type":15,"tag":60,"props":386,"children":387},{},[388,398,408],{"type":15,"tag":64,"props":389,"children":390},{},[391,396],{"type":15,"tag":68,"props":392,"children":393},{},[394],{"type":21,"value":395},"Both teams idle:",{"type":21,"value":397}," 8 GPUs available for whoever submits first.",{"type":15,"tag":64,"props":399,"children":400},{},[401,406],{"type":15,"tag":68,"props":402,"children":403},{},[404],{"type":21,"value":405},"Team A using 6 GPUs, Team B submits:",{"type":21,"value":407}," Team A's jobs beyond their nominal quota (4) get preempted to make room for Team B's quota.",{"type":15,"tag":64,"props":409,"children":410},{},[411,416],{"type":15,"tag":68,"props":412,"children":413},{},[414],{"type":21,"value":415},"Team A using 4 GPUs, Team B using 2:",{"type":21,"value":417}," Team A can borrow Team B's 2 unused GPUs, using 6 total.",{"type":15,"tag":23,"props":419,"children":420},{},[421],{"type":21,"value":422},"This is fair sharing. Resources don't sit idle just because their \"owner\" isn't using them, but guaranteed quotas are respected when demand appears.",{"type":15,"tag":48,"props":424,"children":426},{"id":425},"priorities",[427],{"type":21,"value":428},"Priorities",{"type":15,"tag":23,"props":430,"children":431},{},[432],{"type":21,"value":433},"Not all jobs are equal. Production retraining should preempt experimental runs. Kueue supports this with WorkloadPriorityClasses:",{"type":15,"tag":163,"props":435,"children":438},{"className":436,"code":437,"language":168,"meta":7},[166],"apiVersion: kueue.x-k8s.io\u002Fv1beta1\nkind: WorkloadPriorityClass\nmetadata:\n  name: production\nvalue: 1000\ndescription: \"Production retraining jobs\"\n---\napiVersion: kueue.x-k8s.io\u002Fv1beta1\nkind: WorkloadPriorityClass\nmetadata:\n  name: experiment\nvalue: 100\ndescription: \"Experimental training runs\"\n",[439],{"type":15,"tag":171,"props":440,"children":441},{"__ignoreMap":7},[442],{"type":21,"value":437},{"type":15,"tag":23,"props":444,"children":445},{},[446],{"type":21,"value":447},"Jobs reference the priority:",{"type":15,"tag":163,"props":449,"children":452},{"className":450,"code":451,"language":168,"meta":7},[166],"apiVersion: batch\u002Fv1\nkind: Job\nmetadata:\n  name: weekly-retrain\n  namespace: team-b\n  labels:\n    kueue.x-k8s.io\u002Fqueue-name: training-queue\n    kueue.x-k8s.io\u002Fpriority-class: production\nspec:\n  template:\n    spec:\n      containers:\n      - name: train\n        image: training:latest\n        resources:\n          requests:\n            nvidia.com\u002Fgpu: 2\n      restartPolicy: Never\n",[453],{"type":15,"tag":171,"props":454,"children":455},{"__ignoreMap":7},[456],{"type":21,"value":451},{"type":15,"tag":23,"props":458,"children":459},{},[460],{"type":21,"value":461},"When a high-priority job arrives and there aren't enough resources, Kueue can preempt lower-priority jobs to make room. The preempted jobs go back into the queue and run when resources free up.",{"type":15,"tag":152,"props":463,"children":465},{"id":464},"preemption-policies",[466],{"type":21,"value":467},"Preemption policies",{"type":15,"tag":23,"props":469,"children":470},{},[471],{"type":21,"value":472},"Kueue's preemption is configurable per ClusterQueue:",{"type":15,"tag":163,"props":474,"children":477},{"className":475,"code":476,"language":168,"meta":7},[166],"spec:\n  preemption:\n    reclaimWithinCohort: Any\n    withinClusterQueue: LowerPriority\n",[478],{"type":15,"tag":171,"props":479,"children":480},{"__ignoreMap":7},[481],{"type":21,"value":476},{"type":15,"tag":60,"props":483,"children":484},{},[485,499],{"type":15,"tag":64,"props":486,"children":487},{},[488,497],{"type":15,"tag":68,"props":489,"children":490},{},[491],{"type":15,"tag":171,"props":492,"children":494},{"className":493},[],[495],{"type":21,"value":496},"reclaimWithinCohort: Any",{"type":21,"value":498}," — reclaim borrowed resources from other queues in the cohort.",{"type":15,"tag":64,"props":500,"children":501},{},[502,511],{"type":15,"tag":68,"props":503,"children":504},{},[505],{"type":15,"tag":171,"props":506,"children":508},{"className":507},[],[509],{"type":21,"value":510},"withinClusterQueue: LowerPriority",{"type":21,"value":512}," — within the same queue, preempt lower-priority jobs.",{"type":15,"tag":23,"props":514,"children":515},{},[516],{"type":21,"value":517},"This gives you fine-grained control. You might want production jobs to preempt experiments, but not to preempt other production jobs.",{"type":15,"tag":48,"props":519,"children":521},{"id":520},"putting-it-all-together",[522],{"type":21,"value":523},"Putting it all together",{"type":15,"tag":23,"props":525,"children":526},{},[527],{"type":21,"value":528},"Here's the full setup for our two-team scenario:",{"type":15,"tag":163,"props":530,"children":533},{"className":531,"code":532,"language":168,"meta":7},[166],"# The GPU flavor\napiVersion: kueue.x-k8s.io\u002Fv1beta1\nkind: ResourceFlavor\nmetadata:\n  name: gpu-a100\nspec:\n  nodeLabels:\n    gpu-type: a100\n---\n# Team A: 4 GPU quota, can borrow 4 more\napiVersion: kueue.x-k8s.io\u002Fv1beta1\nkind: ClusterQueue\nmetadata:\n  name: team-a-queue\nspec:\n  cohort: gpu-cluster\n  preemption:\n    reclaimWithinCohort: Any\n    withinClusterQueue: LowerPriority\n  resourceGroups:\n  - coveredResources: [\"cpu\", \"memory\", \"nvidia.com\u002Fgpu\"]\n    flavors:\n    - name: gpu-a100\n      resources:\n      - name: \"nvidia.com\u002Fgpu\"\n        nominalQuota: 4\n        borrowingLimit: 4\n      - name: \"cpu\"\n        nominalQuota: 32\n      - name: \"memory\"\n        nominalQuota: 128Gi\n---\n# Team B: 4 GPU quota, can borrow 4 more\napiVersion: kueue.x-k8s.io\u002Fv1beta1\nkind: ClusterQueue\nmetadata:\n  name: team-b-queue\nspec:\n  cohort: gpu-cluster\n  preemption:\n    reclaimWithinCohort: Any\n    withinClusterQueue: LowerPriority\n  resourceGroups:\n  - coveredResources: [\"cpu\", \"memory\", \"nvidia.com\u002Fgpu\"]\n    flavors:\n    - name: gpu-a100\n      resources:\n      - name: \"nvidia.com\u002Fgpu\"\n        nominalQuota: 4\n        borrowingLimit: 4\n      - name: \"cpu\"\n        nominalQuota: 32\n      - name: \"memory\"\n        nominalQuota: 128Gi\n---\n# LocalQueues in each team's namespace\napiVersion: kueue.x-k8s.io\u002Fv1beta1\nkind: LocalQueue\nmetadata:\n  name: training-queue\n  namespace: team-a\nspec:\n  clusterQueue: team-a-queue\n---\napiVersion: kueue.x-k8s.io\u002Fv1beta1\nkind: LocalQueue\nmetadata:\n  name: training-queue\n  namespace: team-b\nspec:\n  clusterQueue: team-b-queue\n",[534],{"type":15,"tag":171,"props":535,"children":536},{"__ignoreMap":7},[537],{"type":21,"value":532},{"type":15,"tag":23,"props":539,"children":540},{},[541,543,549,551,556],{"type":21,"value":542},"Apply with ",{"type":15,"tag":171,"props":544,"children":546},{"className":545},[],[547],{"type":21,"value":548},"kubectl apply -f kueue-setup.yaml",{"type":21,"value":550},", and both teams can submit jobs to their respective ",{"type":15,"tag":171,"props":552,"children":554},{"className":553},[],[555],{"type":21,"value":275},{"type":21,"value":557}," LocalQueues. Kueue handles the rest.",{"type":15,"tag":48,"props":559,"children":561},{"id":560},"monitoring-kueue",[562],{"type":21,"value":563},"Monitoring Kueue",{"type":15,"tag":23,"props":565,"children":566},{},[567],{"type":21,"value":568},"Kueue exposes Prometheus metrics out of the box:",{"type":15,"tag":60,"props":570,"children":571},{},[572,583,594],{"type":15,"tag":64,"props":573,"children":574},{},[575,581],{"type":15,"tag":171,"props":576,"children":578},{"className":577},[],[579],{"type":21,"value":580},"kueue_pending_workloads",{"type":21,"value":582}," — how many jobs are waiting in each queue",{"type":15,"tag":64,"props":584,"children":585},{},[586,592],{"type":15,"tag":171,"props":587,"children":589},{"className":588},[],[590],{"type":21,"value":591},"kueue_admitted_active_workloads",{"type":21,"value":593}," — how many jobs are currently running",{"type":15,"tag":64,"props":595,"children":596},{},[597,603],{"type":15,"tag":171,"props":598,"children":600},{"className":599},[],[601],{"type":21,"value":602},"kueue_cluster_queue_resource_usage",{"type":21,"value":604}," — current resource consumption per queue",{"type":15,"tag":23,"props":606,"children":607},{},[608,610,615,617,622],{"type":21,"value":609},"These tell you whether your quotas are right-sized. If ",{"type":15,"tag":171,"props":611,"children":613},{"className":612},[],[614],{"type":21,"value":580},{"type":21,"value":616}," is consistently high for one queue, that team needs more quota (or the cluster needs more GPUs). If ",{"type":15,"tag":171,"props":618,"children":620},{"className":619},[],[621],{"type":21,"value":602},{"type":21,"value":623}," is consistently low, resources are being wasted.",{"type":15,"tag":48,"props":625,"children":627},{"id":626},"when-to-use-kueue-vs-simpler-approaches",[628],{"type":21,"value":629},"When to use Kueue vs. simpler approaches",{"type":15,"tag":23,"props":631,"children":632},{},[633],{"type":15,"tag":68,"props":634,"children":635},{},[636],{"type":21,"value":637},"Use Kueue when:",{"type":15,"tag":60,"props":639,"children":640},{},[641,646,651,656],{"type":15,"tag":64,"props":642,"children":643},{},[644],{"type":21,"value":645},"Multiple teams or users share GPU resources.",{"type":15,"tag":64,"props":647,"children":648},{},[649],{"type":21,"value":650},"You need quotas to prevent one team from monopolizing the cluster.",{"type":15,"tag":64,"props":652,"children":653},{},[654],{"type":21,"value":655},"Production jobs need to preempt experimental work.",{"type":15,"tag":64,"props":657,"children":658},{},[659],{"type":21,"value":660},"You want fair sharing — idle resources should be usable by anyone, but guaranteed quotas should be respected.",{"type":15,"tag":23,"props":662,"children":663},{},[664],{"type":15,"tag":68,"props":665,"children":666},{},[667],{"type":21,"value":668},"You might not need Kueue when:",{"type":15,"tag":60,"props":670,"children":671},{},[672,677,682],{"type":15,"tag":64,"props":673,"children":674},{},[675],{"type":21,"value":676},"One team, one GPU. Just submit Jobs directly.",{"type":15,"tag":64,"props":678,"children":679},{},[680],{"type":21,"value":681},"All jobs have the same priority and there's no contention. First-come, first-served works fine.",{"type":15,"tag":64,"props":683,"children":684},{},[685],{"type":21,"value":686},"You're using a managed service (GKE Autopilot, AWS Batch) that handles scheduling for you.",{"type":15,"tag":23,"props":688,"children":689},{},[690],{"type":21,"value":691},"Kueue adds operational complexity. It's worth it when contention for GPUs is a real problem. If you're not fighting over resources, plain Kubernetes Jobs are simpler.",{"type":15,"tag":693,"props":694,"children":695},"hr",{},[],{"type":15,"tag":23,"props":697,"children":698},{},[699,701,708,710,716],{"type":21,"value":700},"GPU scheduling is one of the trickier parts of on-prem ML infrastructure. If you're setting up shared GPU clusters and need help with Kueue, quotas, or cluster architecture, ",{"type":15,"tag":37,"props":702,"children":705},{"href":703,"rel":704},"https:\u002F\u002Fdeploying.ai",[41],[706],{"type":21,"value":707},"deploying.ai",{"type":21,"value":709}," does this work — ",{"type":15,"tag":37,"props":711,"children":713},{"href":712},"mailto:vlad@deploying.ai",[714],{"type":21,"value":715},"let's talk",{"type":21,"value":144},{"title":7,"searchDepth":718,"depth":718,"links":719},2,[720,721,727,728,729,732,733,734],{"id":50,"depth":718,"text":53},{"id":147,"depth":718,"text":150,"children":722},[723,725,726],{"id":154,"depth":724,"text":128},3,{"id":198,"depth":724,"text":135},{"id":239,"depth":724,"text":142},{"id":288,"depth":718,"text":291},{"id":343,"depth":718,"text":346},{"id":425,"depth":718,"text":428,"children":730},[731],{"id":464,"depth":724,"text":467},{"id":520,"depth":718,"text":523},{"id":560,"depth":718,"text":563},{"id":626,"depth":718,"text":629},"markdown","content:articles:gpu-scheduling-kubernetes-kueue.md","content","articles\u002Fgpu-scheduling-kubernetes-kueue.md","articles\u002Fgpu-scheduling-kubernetes-kueue","md",{"_path":4,"_dir":5,"_draft":6,"_partial":6,"_locale":7,"title":8,"description":9,"date":10,"body":742,"_type":735,"_id":736,"_source":737,"_file":738,"_stem":739,"_extension":740},{"type":12,"children":743,"toc":1299},[744,748,752,756,765,769,773,792,796,819,838,842,846,850,858,874,878,882,890,898,908,912,916,924,932,948,952,956,964,968,985,989,993,997,1007,1015,1025,1029,1056,1060,1064,1068,1076,1080,1088,1092,1096,1100,1108,1135,1139,1143,1147,1155,1171,1175,1179,1209,1225,1229,1236,1255,1262,1277,1281,1284],{"type":15,"tag":16,"props":745,"children":746},{"id":18},[747],{"type":21,"value":8},{"type":15,"tag":23,"props":749,"children":750},{},[751],{"type":21,"value":27},{"type":15,"tag":23,"props":753,"children":754},{},[755],{"type":21,"value":32},{"type":15,"tag":23,"props":757,"children":758},{},[759,764],{"type":15,"tag":37,"props":760,"children":762},{"href":39,"rel":761},[41],[763],{"type":21,"value":44},{"type":21,"value":46},{"type":15,"tag":48,"props":766,"children":767},{"id":50},[768],{"type":21,"value":53},{"type":15,"tag":23,"props":770,"children":771},{},[772],{"type":21,"value":58},{"type":15,"tag":60,"props":774,"children":775},{},[776,784],{"type":15,"tag":64,"props":777,"children":778},{},[779,783],{"type":15,"tag":68,"props":780,"children":781},{},[782],{"type":21,"value":72},{"type":21,"value":74},{"type":15,"tag":64,"props":785,"children":786},{},[787,791],{"type":15,"tag":68,"props":788,"children":789},{},[790],{"type":21,"value":82},{"type":21,"value":84},{"type":15,"tag":23,"props":793,"children":794},{},[795],{"type":21,"value":89},{"type":15,"tag":91,"props":797,"children":798},{},[799,803,807,811,815],{"type":15,"tag":64,"props":800,"children":801},{},[802],{"type":21,"value":98},{"type":15,"tag":64,"props":804,"children":805},{},[806],{"type":21,"value":103},{"type":15,"tag":64,"props":808,"children":809},{},[810],{"type":21,"value":108},{"type":15,"tag":64,"props":812,"children":813},{},[814],{"type":21,"value":113},{"type":15,"tag":64,"props":816,"children":817},{},[818],{"type":21,"value":118},{"type":15,"tag":23,"props":820,"children":821},{},[822,823,827,828,832,833,837],{"type":21,"value":123},{"type":15,"tag":68,"props":824,"children":825},{},[826],{"type":21,"value":128},{"type":21,"value":130},{"type":15,"tag":68,"props":829,"children":830},{},[831],{"type":21,"value":135},{"type":21,"value":137},{"type":15,"tag":68,"props":834,"children":835},{},[836],{"type":21,"value":142},{"type":21,"value":144},{"type":15,"tag":48,"props":839,"children":840},{"id":147},[841],{"type":21,"value":150},{"type":15,"tag":152,"props":843,"children":844},{"id":154},[845],{"type":21,"value":128},{"type":15,"tag":23,"props":847,"children":848},{},[849],{"type":21,"value":161},{"type":15,"tag":163,"props":851,"children":853},{"className":852,"code":167,"language":168,"meta":7},[166],[854],{"type":15,"tag":171,"props":855,"children":856},{"__ignoreMap":7},[857],{"type":21,"value":167},{"type":15,"tag":23,"props":859,"children":860},{},[861,862,867,868,873],{"type":21,"value":179},{"type":15,"tag":171,"props":863,"children":865},{"className":864},[],[866],{"type":21,"value":185},{"type":21,"value":187},{"type":15,"tag":171,"props":869,"children":871},{"className":870},[],[872],{"type":21,"value":193},{"type":21,"value":195},{"type":15,"tag":152,"props":875,"children":876},{"id":198},[877],{"type":21,"value":135},{"type":15,"tag":23,"props":879,"children":880},{},[881],{"type":21,"value":205},{"type":15,"tag":163,"props":883,"children":885},{"className":884,"code":209,"language":168,"meta":7},[166],[886],{"type":15,"tag":171,"props":887,"children":888},{"__ignoreMap":7},[889],{"type":21,"value":209},{"type":15,"tag":163,"props":891,"children":893},{"className":892,"code":218,"language":168,"meta":7},[166],[894],{"type":15,"tag":171,"props":895,"children":896},{"__ignoreMap":7},[897],{"type":21,"value":218},{"type":15,"tag":23,"props":899,"children":900},{},[901,902,907],{"type":21,"value":228},{"type":15,"tag":171,"props":903,"children":905},{"className":904},[],[906],{"type":21,"value":234},{"type":21,"value":236},{"type":15,"tag":152,"props":909,"children":910},{"id":239},[911],{"type":21,"value":142},{"type":15,"tag":23,"props":913,"children":914},{},[915],{"type":21,"value":246},{"type":15,"tag":163,"props":917,"children":919},{"className":918,"code":250,"language":168,"meta":7},[166],[920],{"type":15,"tag":171,"props":921,"children":922},{"__ignoreMap":7},[923],{"type":21,"value":250},{"type":15,"tag":163,"props":925,"children":927},{"className":926,"code":259,"language":168,"meta":7},[166],[928],{"type":15,"tag":171,"props":929,"children":930},{"__ignoreMap":7},[931],{"type":21,"value":259},{"type":15,"tag":23,"props":933,"children":934},{},[935,936,941,942,947],{"type":21,"value":269},{"type":15,"tag":171,"props":937,"children":939},{"className":938},[],[940],{"type":21,"value":275},{"type":21,"value":277},{"type":15,"tag":171,"props":943,"children":945},{"className":944},[],[946],{"type":21,"value":283},{"type":21,"value":285},{"type":15,"tag":48,"props":949,"children":950},{"id":288},[951],{"type":21,"value":291},{"type":15,"tag":23,"props":953,"children":954},{},[955],{"type":21,"value":296},{"type":15,"tag":163,"props":957,"children":959},{"className":958,"code":300,"language":168,"meta":7},[166],[960],{"type":15,"tag":171,"props":961,"children":962},{"__ignoreMap":7},[963],{"type":21,"value":300},{"type":15,"tag":23,"props":965,"children":966},{},[967],{"type":21,"value":310},{"type":15,"tag":91,"props":969,"children":970},{},[971,981],{"type":15,"tag":64,"props":972,"children":973},{},[974,975,980],{"type":21,"value":318},{"type":15,"tag":171,"props":976,"children":978},{"className":977},[],[979],{"type":21,"value":283},{"type":21,"value":325},{"type":15,"tag":64,"props":982,"children":983},{},[984],{"type":21,"value":330},{"type":15,"tag":23,"props":986,"children":987},{},[988],{"type":21,"value":335},{"type":15,"tag":23,"props":990,"children":991},{},[992],{"type":21,"value":340},{"type":15,"tag":48,"props":994,"children":995},{"id":343},[996],{"type":21,"value":346},{"type":15,"tag":23,"props":998,"children":999},{},[1000,1001,1006],{"type":21,"value":351},{"type":15,"tag":171,"props":1002,"children":1004},{"className":1003},[],[1005],{"type":21,"value":234},{"type":21,"value":358},{"type":15,"tag":163,"props":1008,"children":1010},{"className":1009,"code":362,"language":168,"meta":7},[166],[1011],{"type":15,"tag":171,"props":1012,"children":1013},{"__ignoreMap":7},[1014],{"type":21,"value":362},{"type":15,"tag":23,"props":1016,"children":1017},{},[1018,1019,1024],{"type":21,"value":351},{"type":15,"tag":171,"props":1020,"children":1022},{"className":1021},[],[1023],{"type":21,"value":377},{"type":21,"value":379},{"type":15,"tag":23,"props":1026,"children":1027},{},[1028],{"type":21,"value":384},{"type":15,"tag":60,"props":1030,"children":1031},{},[1032,1040,1048],{"type":15,"tag":64,"props":1033,"children":1034},{},[1035,1039],{"type":15,"tag":68,"props":1036,"children":1037},{},[1038],{"type":21,"value":395},{"type":21,"value":397},{"type":15,"tag":64,"props":1041,"children":1042},{},[1043,1047],{"type":15,"tag":68,"props":1044,"children":1045},{},[1046],{"type":21,"value":405},{"type":21,"value":407},{"type":15,"tag":64,"props":1049,"children":1050},{},[1051,1055],{"type":15,"tag":68,"props":1052,"children":1053},{},[1054],{"type":21,"value":415},{"type":21,"value":417},{"type":15,"tag":23,"props":1057,"children":1058},{},[1059],{"type":21,"value":422},{"type":15,"tag":48,"props":1061,"children":1062},{"id":425},[1063],{"type":21,"value":428},{"type":15,"tag":23,"props":1065,"children":1066},{},[1067],{"type":21,"value":433},{"type":15,"tag":163,"props":1069,"children":1071},{"className":1070,"code":437,"language":168,"meta":7},[166],[1072],{"type":15,"tag":171,"props":1073,"children":1074},{"__ignoreMap":7},[1075],{"type":21,"value":437},{"type":15,"tag":23,"props":1077,"children":1078},{},[1079],{"type":21,"value":447},{"type":15,"tag":163,"props":1081,"children":1083},{"className":1082,"code":451,"language":168,"meta":7},[166],[1084],{"type":15,"tag":171,"props":1085,"children":1086},{"__ignoreMap":7},[1087],{"type":21,"value":451},{"type":15,"tag":23,"props":1089,"children":1090},{},[1091],{"type":21,"value":461},{"type":15,"tag":152,"props":1093,"children":1094},{"id":464},[1095],{"type":21,"value":467},{"type":15,"tag":23,"props":1097,"children":1098},{},[1099],{"type":21,"value":472},{"type":15,"tag":163,"props":1101,"children":1103},{"className":1102,"code":476,"language":168,"meta":7},[166],[1104],{"type":15,"tag":171,"props":1105,"children":1106},{"__ignoreMap":7},[1107],{"type":21,"value":476},{"type":15,"tag":60,"props":1109,"children":1110},{},[1111,1123],{"type":15,"tag":64,"props":1112,"children":1113},{},[1114,1122],{"type":15,"tag":68,"props":1115,"children":1116},{},[1117],{"type":15,"tag":171,"props":1118,"children":1120},{"className":1119},[],[1121],{"type":21,"value":496},{"type":21,"value":498},{"type":15,"tag":64,"props":1124,"children":1125},{},[1126,1134],{"type":15,"tag":68,"props":1127,"children":1128},{},[1129],{"type":15,"tag":171,"props":1130,"children":1132},{"className":1131},[],[1133],{"type":21,"value":510},{"type":21,"value":512},{"type":15,"tag":23,"props":1136,"children":1137},{},[1138],{"type":21,"value":517},{"type":15,"tag":48,"props":1140,"children":1141},{"id":520},[1142],{"type":21,"value":523},{"type":15,"tag":23,"props":1144,"children":1145},{},[1146],{"type":21,"value":528},{"type":15,"tag":163,"props":1148,"children":1150},{"className":1149,"code":532,"language":168,"meta":7},[166],[1151],{"type":15,"tag":171,"props":1152,"children":1153},{"__ignoreMap":7},[1154],{"type":21,"value":532},{"type":15,"tag":23,"props":1156,"children":1157},{},[1158,1159,1164,1165,1170],{"type":21,"value":542},{"type":15,"tag":171,"props":1160,"children":1162},{"className":1161},[],[1163],{"type":21,"value":548},{"type":21,"value":550},{"type":15,"tag":171,"props":1166,"children":1168},{"className":1167},[],[1169],{"type":21,"value":275},{"type":21,"value":557},{"type":15,"tag":48,"props":1172,"children":1173},{"id":560},[1174],{"type":21,"value":563},{"type":15,"tag":23,"props":1176,"children":1177},{},[1178],{"type":21,"value":568},{"type":15,"tag":60,"props":1180,"children":1181},{},[1182,1191,1200],{"type":15,"tag":64,"props":1183,"children":1184},{},[1185,1190],{"type":15,"tag":171,"props":1186,"children":1188},{"className":1187},[],[1189],{"type":21,"value":580},{"type":21,"value":582},{"type":15,"tag":64,"props":1192,"children":1193},{},[1194,1199],{"type":15,"tag":171,"props":1195,"children":1197},{"className":1196},[],[1198],{"type":21,"value":591},{"type":21,"value":593},{"type":15,"tag":64,"props":1201,"children":1202},{},[1203,1208],{"type":15,"tag":171,"props":1204,"children":1206},{"className":1205},[],[1207],{"type":21,"value":602},{"type":21,"value":604},{"type":15,"tag":23,"props":1210,"children":1211},{},[1212,1213,1218,1219,1224],{"type":21,"value":609},{"type":15,"tag":171,"props":1214,"children":1216},{"className":1215},[],[1217],{"type":21,"value":580},{"type":21,"value":616},{"type":15,"tag":171,"props":1220,"children":1222},{"className":1221},[],[1223],{"type":21,"value":602},{"type":21,"value":623},{"type":15,"tag":48,"props":1226,"children":1227},{"id":626},[1228],{"type":21,"value":629},{"type":15,"tag":23,"props":1230,"children":1231},{},[1232],{"type":15,"tag":68,"props":1233,"children":1234},{},[1235],{"type":21,"value":637},{"type":15,"tag":60,"props":1237,"children":1238},{},[1239,1243,1247,1251],{"type":15,"tag":64,"props":1240,"children":1241},{},[1242],{"type":21,"value":645},{"type":15,"tag":64,"props":1244,"children":1245},{},[1246],{"type":21,"value":650},{"type":15,"tag":64,"props":1248,"children":1249},{},[1250],{"type":21,"value":655},{"type":15,"tag":64,"props":1252,"children":1253},{},[1254],{"type":21,"value":660},{"type":15,"tag":23,"props":1256,"children":1257},{},[1258],{"type":15,"tag":68,"props":1259,"children":1260},{},[1261],{"type":21,"value":668},{"type":15,"tag":60,"props":1263,"children":1264},{},[1265,1269,1273],{"type":15,"tag":64,"props":1266,"children":1267},{},[1268],{"type":21,"value":676},{"type":15,"tag":64,"props":1270,"children":1271},{},[1272],{"type":21,"value":681},{"type":15,"tag":64,"props":1274,"children":1275},{},[1276],{"type":21,"value":686},{"type":15,"tag":23,"props":1278,"children":1279},{},[1280],{"type":21,"value":691},{"type":15,"tag":693,"props":1282,"children":1283},{},[],{"type":15,"tag":23,"props":1285,"children":1286},{},[1287,1288,1293,1294,1298],{"type":21,"value":700},{"type":15,"tag":37,"props":1289,"children":1291},{"href":703,"rel":1290},[41],[1292],{"type":21,"value":707},{"type":21,"value":709},{"type":15,"tag":37,"props":1295,"children":1296},{"href":712},[1297],{"type":21,"value":715},{"type":21,"value":144},{"title":7,"searchDepth":718,"depth":718,"links":1300},[1301,1302,1307,1308,1309,1312,1313,1314],{"id":50,"depth":718,"text":53},{"id":147,"depth":718,"text":150,"children":1303},[1304,1305,1306],{"id":154,"depth":724,"text":128},{"id":198,"depth":724,"text":135},{"id":239,"depth":724,"text":142},{"id":288,"depth":718,"text":291},{"id":343,"depth":718,"text":346},{"id":425,"depth":718,"text":428,"children":1310},[1311],{"id":464,"depth":724,"text":467},{"id":520,"depth":718,"text":523},{"id":560,"depth":718,"text":563},{"id":626,"depth":718,"text":629},1785232227842]