GPU Cluster Scheduling and Chargeback: Why Your Most Expensive Hardware Sits Idle at 3am

Why shared GPU clusters run at half the utilisation their owners assume

At 3am the cluster dashboard shows 92 percent of GPUs allocated. The device telemetry shows 31 percent of them doing any meaningful work. A team reserved eight nodes on Tuesday for a fine tuning run that finished Wednesday morning and nobody released them, because releasing them means queuing again and the queue is uncertain. Two researchers have Jupyter sessions holding a GPU each since last week. An evaluation job that the product team needs before a Friday review is sitting behind a five day hyperparameter sweep that somebody submitted at default priority and forgot about. Everyone in the organisation believes the cluster is full. It is not full, it is held.

This gap between allocation and utilisation is the entire problem, and most organisations do not measure it because the scheduler does not. A scheduler's job is to decide who gets what and it considers itself finished at that point. Whether the process on the other side actually saturated the device is outside its concern. So the reports that reach leadership say the cluster is at capacity, the answer is to buy more, and the newly bought GPUs join the same pattern within a quarter.

The financial shape is unusually stark here compared with most infrastructure. A GPU node depreciates whether or not it computes, sits in a rack you are paying for, and draws serious power at idle rather than nothing. When the platform team cannot attribute that cost to the projects consuming it, two things follow: finance cannot evaluate whether the cluster is worth its budget, and no team has any reason to release capacity they are not using. Both problems are solved by the same work, which is why it is worth doing properly rather than as a monthly spreadsheet.

Problem one: nobody joins device telemetry to jobs, users and projects

The raw signal exists. NVIDIA's DCGM exporter will give you streaming per device metrics including streaming multiprocessor activity and memory occupancy. The scheduler knows which job held which device at which time. Your identity system knows which user belongs to which team. Almost nowhere are those three joined, which means the organisation has utilisation numbers and allocation numbers but no way to say that team A held 4,100 GPU hours last month and computed on 1,400 of them.

Building that join is the foundation and it is more useful than it sounds, because it changes conversations rather than just producing a chart. Once a team can see their own allocated versus consumed ratio, behaviour shifts before any policy is enforced. It also lets you separate the two very different causes of low utilisation: jobs that hold devices while doing nothing, and jobs that hold devices while running badly, for example a data loading bottleneck starving the accelerator. The first is a scheduling problem. The second is an engineering problem that belongs to the team, and you cannot tell them apart without the telemetry.

Problem two: queue policy is an organisational decision, not a scheduler setting

Every organisation with a shared cluster eventually argues about the same questions. Does the production inference team pre empt research? Can a five day sweep be interrupted, and if so who is responsible for making it resumable? Does an unused quota expire or accumulate? Are reservations allowed at all, and if so does an idle reservation still consume the team's budget? These are political questions with technical enforcement, and the failure mode is that they get answered implicitly by whatever defaults the scheduler shipped with.

Slurm is the strongest engine available for the mechanics and has been for years. Gang scheduling for distributed training, backfill, fair share and preemption are all mature, and if you are running one you should keep it. Its accounting database gives you allocated GPU seconds per account, which is genuinely useful and is also where it stops: it has no notion of utilisation, no rate card, no relationship to your finance system, and no interface a team lead will voluntarily open. Kueue solves queueing and quota cohorts on Kubernetes cleanly, and is likewise a building block rather than a product. Run:ai is a real platform with fractional GPU support and a scheduling layer over Kubernetes, but it is opinionated about how workloads reach the cluster and its cost model is its own rather than yours. Determined AI is excellent if your teams adopt its training workflow and less useful as a neutral scheduler across heterogeneous workloads.

Our position: do not write a scheduler. That is a solved and deeply engineered problem and a bespoke one will be worse. Build the policy, quota and accountability layer above whichever engine you run, because policy is the part that is specific to your organisation and no vendor can supply it.

Problem three: interactive sessions are the largest single leak

The biggest source of held but unused capacity in every cluster we have examined is interactive work: notebooks, remote development sessions and debugging shells that acquire a GPU and hold it through lunch, overnight and across weekends. It is also the most politically sensitive thing to reclaim, because the people doing it are your researchers and their objection is legitimate. Losing an hour of loaded model state to an automated reaper is a real cost to them.

The workable design respects that. Detect idleness from device activity rather than session presence, since a connected session doing nothing is exactly the case you are trying to catch. Warn in the channel the person actually reads before doing anything. Give a grace period long enough to save state. Where the workload supports checkpointing, offer suspend and resume rather than termination, so the cost of being reclaimed is minutes rather than a morning. And make the policy visible and negotiated rather than imposed by the platform team, because a reaper that surprises people gets routed around within a week by scripts that print a dot to the GPU every few minutes.

Problem four: chargeback needs a unit finance will defend

Attribution fails when the unit is vague. A defensible model needs a rate per GPU hour that differs by accelerator type, because an H100 hour and an L40S hour are not the same cost, plus a decision about what the rate includes: hardware depreciation over your assumed life, power at your actual facility rate, a share of the interconnect and storage that only exists to serve the cluster, and the platform team's own time if your organisation allocates it. Those inputs are business decisions your finance lead should sign off before anyone writes the calculation.

The sharper decision is whether to charge on allocation or on consumption. Charging on consumption feels fair and is exactly wrong, because it makes holding an idle reservation free and removes the incentive the whole project exists to create. Charging on allocation with utilisation reported alongside is the model that changes behaviour: you pay for what you held, and you can see what you wasted. Add a discount for pre emptible workloads and you have a lever that moves long sweeps off the priority tier without anyone having to enforce it manually.

What this costs and how long it takes

A first release covering the telemetry join across devices, jobs, users and projects, quota and priority policy enforcement on your existing scheduler, and per team allocated versus consumed reporting with a rate card runs $70,000 to $150,000 and ships in 12 to 16 weeks. A full platform adding self service reservations with expiry, idle detection and reclaim with checkpoint support, multi cluster views, cloud burst accounting, and posting into your finance system runs $180,000 to $450,000 over 6 to 12 months.

What moves the number in this category: how many schedulers you actually run, since organisations with both a Slurm cluster and Kubernetes based training need two integrations and a reconciled view. Whether your workloads are checkpointable, because suspend and resume is worth building only if the framework side supports it and often that is an engineering conversation with the ML teams rather than a platform one. Identity integration is usually straightforward but project mapping rarely is, since the project a job belongs to is often not recorded anywhere at submission time and has to be introduced. And cloud burst adds a whole second cost model, because on demand and spot pricing behave nothing like an amortised on premise rate.

Build versus buy, and when buying is right

Buy or adopt open source if you have one team on under about 32 GPUs. Slurm plus its accounting database plus a scheduled report will tell you everything you need and cost you a few days of setup. Kueue is the equivalent answer if you are Kubernetes native. There is no chargeback problem when there is nobody to charge.

Build the layer when three or more teams share the cluster and at least one of them has complained about queue fairness, when finance has asked what the cluster costs per project and the answer took a week to assemble, when your allocated figure and your utilisation figure differ by more than about 30 points, or when a capital request for more GPUs is coming and you cannot evidence that the existing ones are saturated. That last case is the one that usually funds the project, because the build costs a fraction of the hardware it defers.

Buy Run:ai or Determined instead if you want a full training platform and your teams are willing to adopt its workflow end to end. That is a real and reasonable path. It stops being the right path when you have heterogeneous workloads that will not all move, or when the chargeback model your finance team requires does not match the vendor's.

How to choose a developer for GPU platform work

Ask how they would distinguish a job that is idle from a job that is running badly. If they cannot answer that in terms of device metrics, they will build you a dashboard of allocation numbers you already have. Ask what they would do about a distributed training job that needs all its GPUs simultaneously, because gang scheduling is the constraint that breaks naive queue designs.

Ask whether they intend to write a scheduler. If yes, stop. The right answer is to integrate with Slurm, Kueue or Volcano and put the policy and accounting above it. Ask how they will map a job to a project when the submission does not carry one, since that is the practical blocker on every chargeback project and the answer involves process as much as code.

Settle code and infrastructure ownership in writing before kickoff, including the repository and cloud accounts. At Digital Heroes the client owns the code from the first commit. As a first step, run DCGM telemetry alongside your scheduler accounting for two weeks and compute allocated versus consumed GPU hours per team. If that ratio is comfortable, you do not need this project. If it is not, you now have the business case in a single number.