Bare Metal Hosting Provisioning Platform: Why a Dedicated Server Still Takes Days and a Technician

Why manual provisioning caps the size of a hosting business

An order comes in at 4pm on Friday for a dual socket server with 512 GB of RAM and four NVMe drives. A technician pulls one from stock on Monday, racks it, cables it, guesses which switch port he landed on, logs into the switch to set the VLAN, boots the machine, installs the operating system from a USB stick because the network boot was not set up for that rack, picks the next free IP from a spreadsheet, sets a root password, and emails the customer on Tuesday afternoon. The customer, who has used a public cloud before, is confused about why it took four days.

The other half of the same problem is quieter and costs more. A customer cancels on the last day of the month. The billing system stops charging. The server keeps running, drawing power, occupying a rack unit and holding an IP allocation, because reclaiming it requires a technician to wipe it, reset the RAID controller, rotate the management credentials and return it to stock, and nobody is chasing that ticket. In most providers we have looked at, the number of machines sitting in this state is larger than anyone in the building expects, and it is pure cost with no revenue against it.

These two failures are the same failure. There is no system that owns a physical server through its whole life, from purchase order to stock to allocated to live to reclaimed to retired. Instead there is a billing system that knows about a service, a spreadsheet that knows about IPs, a switch that knows about a VLAN, and a technician who knows about the machine. The knowledge is complete only when all four are in the same room, which is once a week at best.

Problem one: the machine has no identity until a person gives it one

Automation starts with knowing what you have. A server needs an identity that exists before it is allocated to anyone: chassis serial, service tag, exact hardware configuration, the MAC address of every network interface, the management controller address and credentials, and the switch and port each interface is physically connected to. Almost no provider has this reliably, because it is assembled by hand during racking and hand assembled data decays.

The fix is discovery rather than data entry. A machine that boots into a discovery image over the network reports its own hardware inventory back, and neighbour discovery on the switch side tells you which port it landed on without anyone writing it down. That single loop removes the most common cause of provisioning failure, which is a correct configuration applied to the wrong switch port. Canonical MAAS does this part genuinely well and is worth studying even if you do not adopt it, because its enlistment and commissioning model is the right shape.

What MAAS does not do is run a multi tenant hosting business. It assumes an internal fleet on a network model it controls, with no concept of a customer, an order, an entitlement, a contract or an invoice. That gap is what most providers end up filling with scripts, and scripts written around MAAS become the least documented and most load bearing code in the company.

Problem two: provisioning is a distributed workflow that fails halfway

A provisioning run touches the switch, the management controller, a DHCP and boot service, an image store, an IPAM record, a DNS zone and the billing system. Any of those can fail, time out or half succeed. A shell script that runs those steps in order will, sooner or later, configure a VLAN and then fail at imaging, leaving a machine in a state nobody can describe and a technician resolving it by hand.

The architecture that survives contact with a real data centre is a state machine with idempotent steps, explicit retries and a visible per machine timeline. Every step must be safe to run twice. Every failure must leave the machine in a state the system can name and resume from. Tinkerbell and OpenStack Ironic both provide credible engines for this layer and are reasonable things to build on rather than reinvent. Ironic is powerful and well proven but carries real operational weight and expects an OpenStack shaped context around it. Tinkerbell is lighter and workflow oriented, which makes it a component rather than a product. Either way, the engine is maybe a third of the work. The product around it, meaning catalog, order, tenancy isolation, reclaim, billing and the customer API, is the rest, and it is the part nobody sells you.

Problem three: the network configuration is where multi tenancy actually lives

Isolation between customers on shared physical infrastructure is a switching problem, not a software one. Each customer needs their own VLAN or set of VLANs, their own private subnet if they have more than one machine, public addressing allocated from your pools with reverse DNS delegated, and increasingly the ability to announce their own address space over BGP. Bonded interfaces need the switch side link aggregation configured to match. Getting any of this wrong does not produce a small error, it produces one customer able to see another customer's traffic.

This is also where a generic product stops being useful, because the configuration is specific to your switch vendors and your topology. A provider running Arista in one hall, Juniper in another and something older at the edge needs an abstraction layer that speaks all three and a validation step that reads the configuration back after applying it. We treat read back verification as mandatory in these builds. Writing a config and assuming it took is how you get a machine that works fine until the customer's traffic patterns reveal it is on the wrong VLAN.

Problem four: reclaim is where the money is and nobody automates it

Reclaim has more steps than provisioning and none of them have a customer waiting. The machine has to be powered down, drives sanitised, RAID configuration reset, management controller credentials rotated, firmware brought back to your baseline, network configuration removed, IP addresses returned to the pool, DNS cleaned up, and the machine validated before it can be offered to the next customer. Skip the credential rotation and you have handed the previous tenant a way back in. Skip the validation and you have handed the next customer a machine with a failing drive.

Data sanitisation deserves a specific decision rather than a default. NIST Special Publication 800-88 gives the accepted vocabulary of clear, purge and destroy, and which one you owe a customer depends on what you told them in your contract and what their own compliance regime requires. Encrypt at rest with per tenant keys and a cryptographic erase becomes fast and defensible, which is why we usually push providers towards that pattern rather than multi hour overwrite passes on large NVMe drives that block the machine from earning revenue.

Automating reclaim converts idle inventory back into sellable stock without a technician touching it. For most providers this is the single fastest payback in the whole build, because the machines are already bought, already racked and already costing power.

What this costs and how long it takes

A first release covering discovery based hardware inventory, the provisioning state machine with switch and image automation, IPAM, and the automated reclaim loop runs $90,000 to $180,000 and ships in 14 to 20 weeks. A full platform adding a public customer API, self service ordering, private networking and BGP sessions, hardware validation and burn in, out of band console access for customers, image and template management, and billing integration runs $200,000 to $500,000 over 6 to 12 months.

The cost drivers in this category are hardware and network diversity above everything else. Management controller behaviour differs between Dell iDRAC, HPE iLO and Supermicro, and the Redfish implementations differ by vendor and by firmware generation in ways that require real handling rather than a single client library. Every additional switch vendor is another driver plus another validation path. Older hardware without usable Redfish support means falling back to IPMI and vendor tooling. Beyond that: whether you need customer facing console access, whether you sell BGP and address space, and whether you have more than one facility, since a second site turns several assumptions into configuration.

Build versus buy, and when buying is right

Buy if you are under roughly 150 servers with a fairly uniform hardware mix and provisioning volume that one technician handles comfortably. WHMCS or HostBill will run your ordering and billing, their provisioning modules will cover the common cases, and the manual residue is genuinely cheaper than an engineering programme. Ubersmith is the stronger option once billing complexity grows and it has more infrastructure awareness than the others.

Build when provisioning volume has made lead time a competitive problem, when your idle unreclaimed inventory is large enough to notice on a power bill, when customers are asking for an API because they want to script their own capacity, or when the automation you already have is a pile of scripts maintained by one engineer. That last one is the most common trigger and the most urgent, because those scripts encode your operational knowledge and they are undocumented by definition.

Our position on what to build versus assemble: do not write your own boot and imaging engine. Ironic or Tinkerbell already solved that and solved it better than a bespoke effort will. Do build the layer above it, because catalog, tenancy, order lifecycle, reclaim policy and billing are your business model, and no product is going to encode your business model for you.

How to choose a developer for a bare metal platform

Ask what happens when provisioning fails at step six of nine. If the answer is a retry, ask what makes step three safe to run a second time. Idempotency is the whole design and a developer who has not thought about it will produce something that works in a demo and needs a technician every week in production.

Ask which management controllers and switch platforms they have actually driven, by vendor and by generation, not in the abstract. Ask how they verify a switch configuration took effect rather than assuming the write succeeded. Ask how tenancy isolation is enforced and how they would prove it to a customer's security reviewer, because that question is coming from your customers whether or not you are ready for it.

Settle ownership before kickoff: repository, cloud and infrastructure accounts, and the right to hire anyone else to continue. At Digital Heroes the client owns the code from the first commit. A sensible first step is to instrument what you have now: count how many machines are currently powered but unallocated, and measure the median hours between order and credentials over the last quarter. Those two numbers are the business case, and both can be gathered this week.