Bare Metal Provisioning Platform Problems: The 5 That Cost Real Money, and How to Avoid Them

Why does the scope stop at provisioning and never reach reclaim?

Because provisioning is the part with a customer shouting at the end of it. Lead time is visible, it appears in sales conversations, and it is easy to write a business case around. Reclaim is invisible. Nobody complains, no ticket ages in a queue anyone reviews, and the cost shows up on a power bill as a number without a name attached.

So the automation gets built for the happy path outward and the return journey stays manual: wipe the drives, reset the RAID configuration, rotate the management controller credentials, restore the firmware baseline, tear down the network configuration, return the addresses, clean up DNS, validate and restock. Nine steps, no deadline, competing against work that has one.

The fix is to scope the machine lifecycle as one object rather than two workflows. A physical server has states: purchased, stock, allocated, live, reclaiming, retired. The system owns every transition, and reclaim is triggered by the billing event rather than by a human noticing. That single change is usually the fastest payback in the whole build, because it converts inventory you have already paid for back into sellable stock without a technician touching it. Before anyone writes a proposal, count how many machines are currently powered and unallocated. That number is the business case, and you can gather it this week.

What goes wrong with hardware inventory data?

Automation starts with knowing what you have, and almost every provider's inventory is hand assembled during racking, which means it decays from the day it is written. A machine needs a chassis serial, a service tag, the exact hardware configuration, the media access control address of every network interface, the management controller address and credentials, and the switch and port each interface is physically connected to. That last field is the one that is quietly wrong most often, because somebody guessed which port they landed on at the end of a long shift.

Importing that spreadsheet into a new platform does not clean it. It launders it. A wrong port mapping in a spreadsheet is a mistake a technician catches when the machine does not come up. The same wrong mapping inside an automation system applies a correct configuration to the wrong switch port, which is a different and worse failure because the configuration succeeded.

The fix is discovery rather than migration. A machine that boots into a discovery image over the network reports its own hardware inventory back, and neighbour discovery on the switch side establishes which port it is on without anyone writing it down. Canonical MAAS does this part genuinely well and its enlistment and commissioning model is worth studying even if you do not adopt it. Treat your existing records as a hint used to reconcile against discovery, never as truth, and make the reconciliation report visible so the gap between what you believed and what is actually racked stops being a surprise.

Why do the switch and management controller integrations break after launch?

Because both are vendor specific in ways that only appear at scale, and because writes are assumed to have worked.

Management controllers differ. Dell iDRAC, HPE iLO and Supermicro behave differently, and Redfish implementations vary by vendor and by firmware generation in ways that need real handling rather than one client library and optimism. A platform tested against your newest generation will meet older machines in production, fall back to IPMI and vendor tooling, and start producing failures that look random until someone correlates them against firmware versions. Handle the variation explicitly, keep a per model capability matrix in the system, and make the platform refuse to guess.

Switch configuration breaks differently. A provider running Arista in one hall, Juniper in another and something older at the edge needs an abstraction that speaks all three, and the failure mode is not an error, it is silence. The write returns success, the configuration did not take or took partially, and the machine works fine until traffic patterns reveal it is on the wrong virtual local area network. Read back verification is mandatory here rather than a refinement. Apply the configuration, read it back, compare it against intent, and fail the provisioning run if they disagree.

Billing is the third and it fails at the boundary. When the platform and the billing system each believe they own service state, a cancellation stops the invoice without stopping the machine. Pick one authority for commercial state and make the other subscribe to it.

What happens when tenancy isolation and sanitisation are not covered?

Isolation on shared physical infrastructure is enforced in the switching layer, not in application code. Each customer needs their own virtual local area network or set of them, a private subnet if they run more than one machine, public addressing from your pools with reverse DNS delegated, matching link aggregation on the switch side for bonded interfaces, and increasingly the ability to announce their own address space over BGP. Getting any of that wrong does not produce a small error. It produces one customer able to see another customer's traffic, which is the single incident a hosting business cannot absorb.

Sanitisation is the other half and it is a contract question before it is a technical one. NIST Special Publication 800-88 gives the accepted vocabulary of clear, purge and destroy, and which level you owe depends on what you told the customer and what their own compliance regime requires. Teams that never decide this default to whatever the script does, which is usually an overwrite pass chosen by an engineer rather than by the contract.

Two fixes worth building deliberately. Encrypt at rest with per tenant keys so a cryptographic erase is fast and defensible, because multi hour overwrite passes on large NVMe drives keep a machine out of revenue for no additional assurance. And rotate management controller credentials on every reclaim without exception, because leaving them is handing the previous tenant a route back into hardware that now belongs to someone else. Then be ready to evidence both, since your customers' security reviewers will ask how isolation is enforced and how you prove it.

Should you build custom or configure what you already own?

Under roughly 150 servers with a fairly uniform hardware mix and provisioning volume one technician handles comfortably, do not build. WHMCS or HostBill will run ordering and billing, their provisioning modules cover the common cases, and the manual residue is genuinely cheaper than an engineering programme. Ubersmith is the stronger option once billing complexity grows, because it carries more infrastructure awareness than the others.

When you do build, do not build all of it. Writing your own boot and imaging engine is a solved problem you would be resolving badly. OpenStack Ironic is powerful and proven, though it carries operational weight and expects an OpenStack shaped context around it. Tinkerbell is lighter and workflow oriented, which makes it a component rather than a product. MAAS has the best enlistment model and no concept of a customer, an order or an invoice. Use one of them as the engine and build the layer above: catalog, tenancy, order lifecycle, reclaim policy and billing. That layer is your business model, and no product will encode your business model for you.

Build when provisioning lead time has become a competitive problem, when unreclaimed inventory is visible on a power bill, when customers are asking for an application programming interface because they want to script their own capacity, or when your existing automation is a pile of scripts maintained by one engineer. That last trigger is the most common and the most urgent, because those scripts encode your operational knowledge and are undocumented by definition.

How do hidden costs get into the quote?

Hardware and network diversity, ahead of everything else. Every management controller generation is handling work, every additional switch vendor is a driver plus its own validation path, and machines too old for usable Redfish support mean IPMI fallbacks that add weeks. A proposal priced without an inventory of your actual models and firmware generations is a guess. Provide that inventory before anyone quotes.

Second, a second facility, which turns address pools, boot services, image distribution and cross site provisioning into configuration decisions. Third, customer facing out of band console access, which is a security design as much as a feature. Fourth, selling BGP sessions and address space. Fifth, burn in and hardware validation, which sounds like a script and is a test harness with agreed pass criteria.

Insist that read back verification of switch configuration and the reclaim workflow are line items rather than assumed. Both are frequently absorbed into a headline provisioning figure and both are where the money and the risk sit.

What separates a build that works from one that fails here?

Idempotency, first. A provisioning run touches the switch, the management controller, a boot service, an image store, an address management record, a DNS zone and the billing system, and any of those can time out or half succeed. A script that runs steps in order will eventually configure a network and fail at imaging, leaving a machine in a state nobody can name. The design that survives a real data centre is a state machine where every step is safe to run twice, every failure leaves a named state, and every machine has a visible timeline. Ask a prospective developer what happens when provisioning fails at step six of nine, then ask what makes step three safe to run again. The second answer tells you whether they have done this.

Second, build the internal lifecycle before the public interface. Exposing an application programming interface on top of a workflow that still needs occasional manual repair produces support tickets faster than revenue. Run it against real orders for a few weeks, then open it up.

Finally, settle ownership before kickoff: the repository, the infrastructure accounts and the unrestricted right to bring in another firm. At Digital Heroes the client owns the code from the first commit. This platform encodes how your data centre operations actually work, so a vendor dependency here is a dependency on your ability to turn inventory into revenue.