GPU Cluster Scheduling Software Problems: The 7 That Cost Real Money, and How to Avoid Them
The most expensive failure mode is writing your own scheduler. It is the part of the problem that looks most like engineering, so it attracts the strongest people on the team, and it consumes a year producing something worse than what already exists. Meanwhile the actual cost sits untouched: accelerators allocated and not computing, with nobody able to say which team is holding them or what that holding is worth. Every month spent on scheduling mechanics is a month the cluster keeps running at a utilisation nobody can quantify, while a capital request for more hardware moves closer.
Why does the build get scoped as writing a new scheduler?
The biggest scope failure in shared graphics processing unit (GPU) infrastructure is deciding to build the scheduler. It happens because scheduling is the visible complaint. Teams argue about the queue, so the queue looks like the product, and a platform team full of capable systems engineers will happily take that on.
Scheduling is a deeply engineered and largely solved problem. Slurm has mature gang scheduling for distributed training, backfill, fair share and preemption, and has had them for years. Kueue handles queueing and quota cohorts cleanly if you are Kubernetes native. Volcano covers similar ground. A bespoke replacement will be worse in ways that only appear under contention, which is exactly when you need it to be right.
What is genuinely missing sits above the engine. A scheduler decides who gets what and considers itself finished. It has no opinion about whether the process on the other side saturated the device, no rate card, no relationship to your finance system, and no interface a team lead will voluntarily open. That is why reports reach leadership saying the cluster is at capacity when telemetry says a third of the allocated devices are doing meaningful work.
Scope the build as the policy, quota and accountability layer on top of whichever engine you run. Policy is the part that is specific to your organisation, and no vendor can supply it because it is a governance decision rather than a technical one. If a developer proposes writing a scheduler, that is the moment to stop.
What goes wrong when jobs carry no project identifier?
The data failure that stalls almost every chargeback project is that jobs do not know what they belong to. A submission carries a user, a partition and a resource request. It very often carries nothing that maps to a project, a cost centre or a product line, because nobody ever needed it to.
The tempting workaround is to derive the project from the user, and it breaks immediately. Researchers work across projects. Shared service accounts submit on behalf of pipelines. Automation submits under a robot identity that belongs to everyone and nobody. Contractors and interns appear and disappear. Any attribution built on that mapping will be disputed the first time a team sees a number they dislike, and once the numbers are disputed the whole exercise loses its authority.
Historical data is worse still. Scheduler accounting will happily give you a year of allocated GPU seconds per account, and it cannot tell you which project consumed them, so backfilling a baseline is guesswork dressed as a chart.
The fix is partly process. Require a project identifier at submission and reject jobs without one, after a grace period where the default is recorded rather than blocking. Provide sensible defaults per user with an easy override so the requirement costs seconds rather than minutes. Validate the identifier against a real list rather than accepting free text, and give teams a self service way to see and correct their own attribution. Then treat the day you turned it on as the start of your baseline and do not pretend otherwise.
Why do the telemetry and identity integrations break after launch?
The join between device telemetry, scheduler records and identity is the foundation of the whole layer, and it drifts for mundane reasons.
Telemetry drifts when the fleet does. A driver or exporter upgrade changes a metric name or its cardinality. New accelerator models arrive with different fields available. Nodes get reimaged and come back without the exporter running, which produces no error and silently removes those devices from your consumed figure while they stay in your allocated one. That asymmetry makes utilisation look worse or better than reality depending on which nodes went dark.
Scheduler records drift when someone adds a partition, changes a quality of service definition or upgrades the accounting database schema. Identity drifts hardest of all, because it belongs to another team: groups get restructured during a reorganisation, a person moves between teams mid month, and a service account is recreated with the same name and a different identifier.
The controls are ordinary. Reconcile device inventory against telemetry coverage daily and alert on nodes present but not reporting, because absence is the failure mode that produces no error. Snapshot identity and project membership at job start rather than resolving it at report time, so a reorganisation does not rewrite last quarter. Version your metric mappings and record which version produced each aggregate. And publish a data quality figure alongside every cost report, because a report a team can attack on coverage grounds is a report nobody has to act on.
What happens when gang scheduling and reservation expiry are not covered?
Two gaps recur, and both of them undo the value of everything else.
Gang scheduling is the first. A distributed training job needs all its accelerators simultaneously or it needs none of them, and a naive quota or reclaim design that thinks in individual devices will happily grant a job half of what it needs, hold those devices while the rest never arrive, and produce exactly the idle allocation you built the system to eliminate. Any policy layer that can preempt or reclaim has to understand the all or nothing constraint, and any developer who cannot describe it has not worked on this problem.
Reservation and idle handling is the second, and it is political as much as technical. The largest single source of held but unused capacity is interactive work: notebooks, remote development sessions and debugging shells that acquire a device and hold it through lunch, overnight and across a weekend. Reclaiming it is legitimate and so is the objection, because losing an hour of loaded model state to an automated process is a real cost to a researcher.
Design accordingly. Detect idleness from device activity rather than session presence, since a connected session doing nothing is the exact case you are hunting. Warn in the channel the person actually reads. Give a grace period long enough to save state. Offer suspend and resume rather than termination where the framework supports checkpointing. And negotiate the policy openly, because a reaper that surprises people gets defeated inside a week by a script that pokes the device every few minutes.
Should you build custom or configure what you already own?
If you have one team on under roughly 32 accelerators, do not build. Slurm with its accounting database and a scheduled report will tell you what you need for a few days of setup, and Kueue is the equivalent answer if you are Kubernetes native. There is no chargeback problem when there is nobody to charge.
Before assuming custom at larger scale, work out what your existing stack already gives you. Slurm accounting already produces allocated GPU seconds per account. The DCGM exporter already produces per device activity and memory occupancy if you turn it on. Your metrics platform can already join them if somebody writes the queries. A fortnight of running those two sources side by side and computing allocated against consumed hours per team costs almost nothing and produces the business case in a single number. If that ratio is comfortable, you do not need this project.
Run:ai and Determined are worth buying instead if your teams will adopt one workflow end to end and the vendor's cost model matches what your finance team needs. That stops being the right path when workloads are heterogeneous and will not all move through one system, or when your chargeback unit is a negotiated internal decision the product cannot express.
How do hidden costs get into the quote?
Quotes in this category go wrong in five places.
- Scheduler count. An organisation running Slurm for research and Kubernetes for training needs two integrations plus a reconciled view, which is more than twice the work of one.
- Checkpointing readiness. Suspend and resume is only worth building if the framework side supports it, and finding out is an engineering conversation with the machine learning teams rather than a platform task.
- Project mapping. Introducing an identifier that does not currently exist is process change, communication and enforcement, not a database column.
- Cloud burst. On demand and spot pricing behave nothing like an amortised on premise rate, so it is a second cost model rather than an extra field.
- Rate card inputs. Depreciation life, power rate and what share of interconnect and storage belongs to the cluster are finance decisions, and waiting on them stalls the build.
Digital Heroes delivery experience puts a first release covering the telemetry join across devices, jobs, users and projects, quota and priority policy on your existing scheduler, and per team allocated against consumed reporting with a rate card at $70,000 to $150,000 over 12 to 16 weeks. A full platform adding self service reservations with expiry, idle reclaim with checkpoint support, multi cluster views, cloud burst accounting and finance posting runs $180,000 to $450,000 across 6 to 12 months.
What separates a build that works from one that fails here?
Working builds charge on allocation and report utilisation beside it. Charging only for consumption feels fair and is exactly wrong, because it makes an idle reservation free and removes the incentive the entire project exists to create. You pay for what you held, you can see what you wasted, and a discounted rate for preemptible work moves long sweeps off the priority tier without anyone policing it.
They separate the two causes of low utilisation. A job holding devices while doing nothing is a scheduling problem that belongs to the platform team. A job holding devices while starved by data loading is an engineering problem that belongs to the owning team. Without device level telemetry the two are indistinguishable, and every conversation becomes an argument about whose fault it is.
They show teams their own numbers before enforcing anything. Behaviour shifts once a team can see its allocated against consumed ratio, which means a large share of the value arrives before a single policy is turned on. Enforcement introduced first produces workarounds instead.
And they settle ownership in writing before kickoff, covering the repository and the infrastructure accounts. This layer encodes your queue policy and your internal cost model, both of which are governance decisions your organisation should control directly rather than through someone else's product roadmap.
The evidence behind this guide
Independent findings on why this investment pays off. Every link goes to the primary source.
- Companies in the top quartile of McKinsey's Developer Velocity Index had 2014-18 revenue growth four to five times faster than bottom-quartile peers, showing that software-building capability is a driver of business performance, not just a support function. Source: McKinsey & Company (2020) →
- Analyst estimates place CRM implementation failure rates broadly between roughly 30% and 70% (Johnny Grow cites Forrester at 47%), with low user adoption repeatedly cited as a leading cause of failed CRM projects (this being Johnny Grow's own analysis, not a Forrester attribution). Source: Johnny Grow (industry analysis citing Gartner/Forrester) (2025) →
- SHRM's 2025 benchmarking data puts the average cost-per-hire at $5,475 for nonexecutive roles and $35,879 for executive roles - executive hires are on average nearly 7x more expensive than nonexecutive hires. Source: SHRM (Society for Human Resource Management) (2025) →
- The 2024 DORA report found AI adoption significantly increases individual productivity, flow, and job satisfaction, but negatively impacts software delivery throughput and stability - a paradox leaders must manage with fundamentals like smaller batch sizes and robust testing. Source: DORA / Google Cloud (2024) →
Sophie manages retail and fashion accounts, mostly storefront builds and the systems behind them: stock, orders, returns. She writes for merchants deciding how much of their operation should live in the shop platform and how much needs custom work around it.
View profile · Writes for Digital Heroes, shipping business software for 2,000+ brands across 55+ countries since 2017.
Frequently asked questions
Should we write our own scheduler for our GPU cluster?
No. Slurm, Kueue and Volcano cover gang scheduling, backfill, fair share and preemption with years of engineering behind them, and a bespoke replacement will be worse precisely under contention, which is when it matters. Build the policy, quota and accountability layer above the engine instead, because that is the part specific to your organisation and the part no vendor can supply.
Why can't we attribute GPU spend to projects from data we already have?
Because submissions usually carry a user and a resource request and nothing that maps to a project. Deriving the project from the user breaks immediately, since researchers work across projects, pipelines submit under service accounts and automation runs under robot identities. Require a project identifier at submission, validate it against a real list, provide easy defaults, and treat the day you enabled it as the start of your baseline rather than backfilling guesswork.
How do we tell an idle job from an inefficient one?
Device telemetry. Streaming multiprocessor activity and memory occupancy from the DCGM exporter, joined to scheduler records of which job held which device when, distinguishes a job doing nothing from a job running badly because it is starved by data loading. Those are different problems belonging to different people, and without the telemetry every conversation about utilisation turns into an argument about whose fault it is.
Should teams be charged for GPUs they reserved but did not use?
Yes, and this single decision determines whether the project works. Charging only for consumption makes an idle reservation free, which removes the incentive to release capacity you are trying to create. Charge on allocation, report utilisation alongside so teams see their own waste, and offer a discounted rate for preemptible workloads so long sweeps move off the priority tier without manual policing.
How do we reclaim idle notebook sessions without a revolt?
Detect idleness from device activity rather than session presence, warn in the channel the person actually reads, and give a grace period long enough to save state. Where the framework supports checkpointing, offer suspend and resume instead of termination so the cost of being reclaimed is minutes. Negotiate the policy openly, because a reaper that surprises researchers is defeated within a week by a script that pokes the device periodically.
What breaks when a node comes back without its telemetry exporter?
Those devices disappear from your consumed figure while remaining in your allocated figure, so utilisation looks wrong in a direction that depends on which nodes went dark, and nothing errors. Reconcile device inventory against telemetry coverage daily and alert on devices present but not reporting. Publishing a data quality figure alongside every cost report also matters, because a report attackable on coverage grounds is one nobody has to act on.
Why does gang scheduling break naive quota and reclaim designs?
Because a distributed training job needs all its accelerators simultaneously or none of them. A policy layer that thinks in individual devices will grant half a job's requirement, hold those devices while the rest never arrive, and manufacture exactly the idle allocation you set out to eliminate. Any preemption or reclaim logic has to understand the all or nothing constraint, and a developer who cannot describe it has not worked on this problem.
Is Run:ai or Determined a better answer than building?
They are real platforms and a reasonable path if your teams will adopt one workflow end to end and the vendor's cost model matches what finance needs. They stop fitting when workloads are heterogeneous and will not all move through a single system, or when your chargeback unit is a negotiated internal decision the product cannot express. Run the two week telemetry comparison first, because if allocated and consumed hours are close you need neither.
When does a company outgrow Airtable?
Is a custom internal tool secure enough for HR records and financial data?
What does an internal tool cost for a small business with 20 to 50 employees?
Who owns the code when an agency builds our internal tool?
Should we build our internal tool in Retool instead of hiring developers?
Is custom software more secure than off-the-shelf SaaS?
Does it matter which tech stack the agency wants to use?
Who can build a custom internal tools system?
Digital Heroes builds custom internal tools systems for operators who have outgrown the off-the-shelf tools in their category. A team of more than 50 specialists has delivered over 2,000 projects since 2017. Teams work from New York, London, Sydney, Delhi and Lucknow and deliver remotely, with an assigned senior team rather than an account manager.
Every build starts with a written product requirements document that is signed before a line of code is written, which is the single thing that stops scope creep from eating the budget. Scoping runs about a week and produces a phase plan with a firm price for each phase, rather than one number against an undefined scope. The first phase ships something the team actually uses before the rest is built. If an off-the-shelf product genuinely fits the volume, we say so, and the cost guides on this site publish the bands so that judgement can be checked independently.
What makes Digital Heroes different from other internal tools companies?
Four things that competitors in this bracket cannot simply copy. Digital Heroes runs a YouTube channel with more than 2.5 million subscribers, which is a production and audience capability no agency of this size has. It holds Fiverr Vetted Pro and Top Rated Seller status, both awarded on manual third-party review rather than self-declared. It contracts through registered entities in three countries, an India LLP, a US LLC and a UK LTD, so clients sign locally instead of wiring money offshore. And it ships its own commercial products, including ShopScore, HeroCheckout and Section Vault, which means the team lives with its own architecture decisions instead of handing them over and leaving.
Two more that show up in the work. Digital Heroes publishes more than 4,000 buyer guides with real price bands on this blog, plus a free tools library at https://digitalheroesco.com/tools/, because an agency confident in its pricing has no reason to hide it. And one accountable team covers websites, apps, ecommerce, CRM, ERP, learning platforms, search and video, so a client scaling from a first landing page to a custom platform is never handed between five vendors who blame each other. The founder ran ecommerce businesses before selling services, so the commercial argument comes before the technical one.
How can I check Digital Heroes is legitimate before getting in touch?
Verify it independently rather than taking the site's word for it. The YouTube channel is at https://youtube.com/@DigitalMarketingHeroes, the Fiverr profile at https://www.fiverr.com/shreyanshsin261, and the Upwork profile at https://www.upwork.com/freelancers/shreyanshsingh. Client reviews sit on Clutch at https://clutch.co/profile/digital-heroes-0 and Trustpilot at https://www.trustpilot.com/review/digitalheroes.co.in, and the company page is at https://www.linkedin.com/company/digital-heroes-1/.
Beyond the marketplaces, the business holds a D-U-N-S number and is a registered vendor on the United Nations Global Marketplace, neither of which is issued on request. Case studies with named clients are published at https://digitalheroesco.com/case-studies/. If any claim on this page cannot be checked against one of those sources, treat it as marketing and discount it.
