Telecom Site Power and Generator Monitoring: Why Battery Strings Fail Silently and Generators Run Dry

Site power is the outage cause you can actually prevent

It is 2:14am and a storm line has crossed a region where you run 900 macro sites plus a few hundred edge and small cell locations. The NOC wall shows 63 power alarms. Forty of them are mains fail on sites with healthy strings and eight hours of autonomy, which is noise. Six are on sites whose battery string was last capacity tested in 2021. Two of those six carry a hospital's dedicated fibre and a public safety tenant. Nobody on shift can say which two, because site criticality lives in a spreadsheet the RF planning team maintains and the power team has never been given. The on-call tech is dispatched to whichever site is closest to his house. By 5am the site that mattered has been down for two hours and you are writing an outage narrative for a tenant whose contract has service credits in it.

The tooling around this is normally a mix: the rectifier vendor's own controller web pages, a legacy SNMP trap collector that emails, generator telemetry from whichever aftermarket box the last contractor fitted, a fuel supplier who sends delivery dockets as PDFs, and a WhatsApp group per region for the field crew. None of that assembles into an answer to the only question that matters at 2:14am, which is: of the sites currently on battery, which will die first, which one hurts most, and can my nearest available tech get there before it does.

The cost of not answering it is concrete. Across power and field infrastructure work we have delivered, the repeating pattern is fuel spend that runs 10 to 20 percent above actual burn because deliveries are scheduled rather than measured, battery replacements that happen either years early on a blanket age rule or one outage too late, and a steady trickle of truck rolls to sites that did not need a visit. Every one of those is a line item you can measure. The site that dropped because a string had lost half its capacity is the one you cannot, because it shows up as a tenant escalation instead.

Battery strings fail quietly and your alarm system is the last to know

A valve regulated lead acid string does not announce its decline. Float voltage looks perfect right up until the moment it has to deliver current, and then a string rated for eight hours gives you forty minutes. The traditional answer is a scheduled discharge test, which nobody performs at fleet scale because it means taking a live site to battery on purpose, sending a crew, and risking exactly the outage you are trying to avoid.

What Vertiv Environet, Schneider EcoStruxure and Eaton Brightlayer do well is present the state their own hardware reports, and inside a data centre hall with a coherent single-vendor estate they do it properly. The gap opens on an outdoor site fleet accumulated over fifteen years of acquisitions and build programmes, where you have four rectifier families, two generator controller brands, some sites with battery monitoring at string level and many with nothing but a shunt. Those platforms model assets and show telemetry. They do not build a health opinion from your fleet's own history, because they have not seen your fleet's history.

What a custom build does: treat every unplanned mains failure as a free capacity test. When a site goes to battery, the system records the discharge curve, ambient temperature, load in amps, and time to recovery. Over a year, every site with real outage exposure generates its own capacity trend without a single deliberate test. Sites in benign grid areas get flagged for a scheduled test precisely because they never discharge, which is the small minority worth sending a crew to. The scoring is not exotic maths: temperature-compensated coulomb counting against the string's rated capacity, trended per string, ranked fleet-wide. What it gives you is a replacement list ordered by risk rather than by install date, which is what turns a blanket five year swap programme into a targeted one.

Fuel is the second largest field cost and it runs on a calendar

Diesel delivery to a generator site is usually scheduled: every six weeks, or after a set number of run hours reported by whoever visited last. That produces two failures at once. Sites in stable grid areas get topped up when they are still nearly full, and sites in load-shedding or storm-heavy areas run dry during exactly the week they were needed. Meanwhile the gap between fuel purchased and fuel that could have been burned given actual run hours is where theft lives, and on a fleet with remote sites and contracted delivery, that gap is rarely reconciled by anyone.

Generic building management platforms will chart a tank level if you give them a sensor. What they will not do is model consumption per site as a function of load and run hours, then compare modelled burn against delivered litres against tank telemetry, and raise an exception when the three disagree. That reconciliation is the whole point and it is specific to your fleet, your generator models, your delivery contractor and your invoicing.

What a custom build does: each site gets a consumption model derived from its own history, litres per run hour at its typical load. Predicted days of autonomy is computed continuously and drives the delivery run, so the truck goes to the sites that will actually run out. A delivery docket is reconciled against the tank level step it should have produced, and a docket claiming 400 litres into a tank that rose by 220 becomes a ticket, not a paid invoice. Run hours reported by the generator controller get compared against fuel consumed, and a site burning fuel without run hours is either a leak or a siphon. None of that requires new hardware beyond a tank sender on the sites where it pays back.

Alarms without criticality are worse than no alarms

Every operator eventually reaches the point where the NOC filters power alarms out because there are too many. That is the real failure state: not a missing alarm, an ignored one. The reason is that a mains fail on a site with a working generator and full tank and a mains fail on a battery-only site with a degraded string arrive looking identical.

What a custom build does: fuse the telemetry with the things only you know. Site criticality tiering that reflects your tenants, your backhaul topology and any public safety or emergency services obligations, because a hub site carrying twelve downstream sites is not the same as a leaf. Predicted time to site down, from current load, battery state of health and generator status, rather than a binary alarm. Suppression of the noise: mains fail on a site with autonomy above a threshold is logged, not raised, until the threshold is crossed. Then dispatch is ranked by predicted time to down against travel time from wherever your techs actually are, which is where an integration into your field service scheduling earns its money. This is also the part no packaged platform can supply, because criticality and crew structure are your business, not the vendor's.

The protocol layer is where projects go wrong

The honest engineering risk in this category is not the analytics, it is talking to the estate. You will have Modbus RTU over serial into a rectifier controller with a vendor-specific register map, SNMP from a newer controller, DNP3 at any site inherited from a utility partnership, a generator controller speaking its own protocol over a cellular modem, and a handful of sites where the only reliable data path is the tech's phone. Backhaul at these sites drops precisely when the interesting events happen, so telemetry must buffer at the edge and replay on reconnect, otherwise you lose the discharge curve for every outage you actually care about.

Budget this properly. Each new controller family is real integration work, typically one to three weeks including a lab unit and a site validation, and you cannot skip the lab unit because register maps documented by the vendor and register maps implemented in firmware version 4.2 are not always the same document. Plan the first release around your two or three highest-count families, which usually covers 70 to 80 percent of sites, and add the long tail as a funded backlog rather than a launch dependency.

What it costs and how long it takes

Across the 2,000-plus projects Digital Heroes has delivered, this category shapes up as follows. A first release covering edge collection with store-and-forward, two or three rectifier and generator families, battery health scoring from opportunistic discharge, fuel burn modelling with delivery reconciliation, and criticality-ranked alarm triage runs $70,000 to $150,000 over 12 to 18 weeks. A full platform adding dispatch integration, generator preventive maintenance, spares and battery capex planning, tenant SLA and outage evidence reporting, and a contractor portal runs $180,000 to $450,000 phased over 8 to 14 months.

What pushes cost up in this category specifically: the number of distinct controller families and firmware vintages, whether any sites need new hardware fitted before they can report anything, whether you carry tenants with contractual service credits that require defensible evidence, and multi-country deployments where fuel procurement and grid behaviour differ enough to need separate models. What holds cost down: starting with one region, your top two hardware families, and the single question of predicted time to site down. That one output changes dispatch behaviour on day one.

Build versus buy

Buy if you run under roughly 150 sites on a substantially single-vendor rectifier estate. Vertiv, Schneider or Eaton monitoring against their own hardware will serve you, the integration is done, and a custom build would be an expensive way to reach a similar screen. Buy also if your sites are grid-stable, mostly battery-only with no generators, and your fuel spend is negligible, because two of the three problems above do not apply to you.

Build when three things are true together: you have a genuinely multi-vendor fleet, your fuel and battery spend is large enough that a 10 percent improvement is a real number, and you carry tenants or obligations where a site down event has a contractual cost. Tower companies and neutral host operators hit all three early, because their entire commercial model is uptime sold to multiple tenants with different criticality. Mobile operators hit it once their site count passes the point where the NOC has started filtering power alarms, which is a symptom you can look for.

How to choose a developer for site power monitoring

Ask how they will handle a site that is offline during the event you care about. If the answer does not include buffering at the edge and replaying on reconnect, they have not built for outdoor telecom sites and your discharge data will have holes exactly where the outages are.

Ask them to describe the battery state of health method they would use before they have seen your data. A credible answer talks about temperature compensation, load at time of discharge, and trending per string. An answer that is just a voltage threshold is a repackaged alarm.

Ask which controller protocols they have actually driven on a live site, and get specifics: the vendor, the model, the transport. Modbus over TCP to a modern controller and Modbus RTU over a serial-to-cellular converter at a site with a marginal signal are different projects, and only one of them teaches you anything.

Ask who owns the code, the cloud accounts and the device credentials, and get it in the contract before kickoff. Site telemetry systems accumulate years of operational history that becomes your capex evidence, and you should never be in a position where that history sits in a vendor's tenancy. At Digital Heroes the client owns the repository from the first commit, and we would tell you to walk from anyone who hedges on it.