Outage Management System Development: How Do You Predict the Failed Device From Scattered Calls and AMI Pings?

Hour three of the storm

The wind hit at 19:20. By 22:10 the dispatcher has 480 calls in the queue, sixty five percent of meters on two substations reporting last gasp, three breakers open in supervisory control, and eleven crews out, four of them borrowed from a neighbouring cooperative who do not know the system. The board on the wall shows outages by call location, which means it shows where the phones are, not where the faults are. Two crews are driving toward the same lateral because two dispatchers logged the same trouble twice under different descriptions.

Then the calls change character. Members stop asking whether the utility knows and start asking when the power comes back. The dispatcher has no basis for an answer, so she says morning. At 06:30 several thousand members are still out, and now the utility has an accuracy problem on top of a restoration problem, plus a board meeting where somebody will ask why the estimate was wrong.

Everything about that night is a data problem wearing an operations costume. The information needed to predict the failed device exists: the calls, the meter last gasp messages, the breaker and recloser status, the faulted circuit indicators. It arrives in four systems with four identity schemes and no common time base, and the person expected to join them is doing it at 22:10 in her head.

Why the prediction engine is the whole system

Everything downstream of prediction is logistics. If you know that the open device is a specific fuse on a specific lateral, crew assignment, estimated restoration and member communication all follow. If you do not, you are dispatching to call locations and discovering the topology by driving it.

Prediction is an inference over your connectivity model. Calls and last gasp messages identify affected meters, meters roll up to transformers, transformers to sections, sections to protective devices. When enough of the meters downstream of a fuse are dark and the meters upstream are not, the fuse is the candidate. Add supervisory control status to confirm or exclude upstream devices, add faulted circuit indicator reports to narrow the section, and weight the result by confidence so the dispatcher sees a ranked prediction rather than an assertion.

Which means the model is everything, and this is exactly where packaged systems assume more than most cooperatives have. Products expect a clean, connected, phase accurate model where every meter maps to a transformer, every transformer maps to a section, and every protective device is present with correct coordination. Real distribution models have unmapped meters after a rebuild, phase data that is confidently wrong on some laterals, temporary switching from a project two years ago that was never reflected, and services added by a crew who told nobody in engineering. A packaged engine given that model produces predictions the dispatcher stops trusting within one storm, and after that she uses the call board, and you have bought an expensive call board.

A build that respects reality does two things instead. It carries data quality explicitly, so a prediction based on a section with known bad phase data comes with lower confidence and says why. And it treats model correction as an operational workflow: when a crew reports that the fuse was actually on the other side of the tap, that becomes a queued correction to engineering rather than tribal knowledge. Systems that improve their own model each storm end up ahead of systems that assumed a perfect one on day one.

Estimated restoration times, and being honest about them

ETR accuracy is the single largest driver of member satisfaction during an event, and it is where most systems perform worst. The reason is that a static rule, four hours from outage start, is wrong the moment crews are reassigned, and nothing recalculates it.

A useful ETR is computed from the actual restoration plan: which crew is assigned, where they are now, travel time on roads that may be blocked, expected work duration for that damage type given your own history, and the position of that job in the queue behind other jobs assigned to the same crew. When a crew gets pulled to a hospital feeder, every ETR behind them moves automatically. Publish confidence with the estimate: assessing damage, crew en route, estimate 03:00, is far better received than a wrong precise number, and it is honest.

The utilities that get this right also accept that early in an event they do not know. Saying assessment underway rather than inventing a time protects credibility for the point later in the night when you do have a real estimate and need members to believe it.

The integrations are the project

Be clear eyed about where the effort goes, because the outage engine is perhaps a third of it.

• Customer information system, for member and service point data, contact details and account status. For cooperatives this often means MultiSpeak, the integration standard developed within the cooperative community, and every implementation of it has local variation.
• Meter head end, for last gasp and power restored notifications plus on demand pings. The behaviour under mass outage matters more than the interface: head ends throttle, messages arrive late and out of order, and a system that treats a last gasp as instant truth will chase ghosts.
• Supervisory control, for breaker, recloser and switch status. This is the highest quality signal you have and it needs a read path that operations trusts, with a hard boundary preventing anything flowing the wrong way.
• Interactive voice response, so a member call becomes a structured trouble report attached to a service point rather than a queue entry, and so the outbound message reflects the current ETR.
• Geographic information system, for the connectivity model itself, with a defined synchronisation process rather than a one time import.
• Mobile for crews, which must work with no signal, because the places you send crews during a storm are exactly the places with no coverage.

Each of these has a failure mode during an event, which is when it matters. Design every integration to degrade rather than stop: if the meter head end goes quiet, the system keeps working on calls and supervisory data and says clearly that meter data is stale.

Reliability reporting is not an afterthought

Your reliability indices go to a state commission, to a board, and in the cooperative world to members who understand them better than utilities expect. SAIDI, SAIFI and CAIDI are computed from the same outage records the dispatcher creates at 22:10, which means data quality during the worst night of the year determines the number you report for the year.

Build the reporting requirements into the operational data model rather than reconstructing them in a spreadsheet each January. That means capturing cause codes and equipment failure detail at the crew level while the crew is standing at the pole, recording customer minutes properly through partial restorations and switching, and implementing the IEEE 1366 major event day classification so that storm performance is separated from blue sky performance in the way your commission expects. Utilities that treat this as a reporting problem discover every year that the operational records will not support the report. Utilities that treat it as a data capture requirement produce the report in an afternoon.

What it costs and how long it takes

From the projects Digital Heroes has delivered, a first release covering the connectivity model with data quality handling, the prediction engine, dispatcher interface, crew mobile with offline capability, and integration to your customer information system and supervisory control runs $150,000 to $300,000 and ships in 16 to 24 weeks. The full system adding meter head end integration, dynamic ETR calculation, member facing outage map and notifications, interactive voice response integration, damage assessment, mutual aid crew handling and reliability index reporting runs $400,000 to $900,000 phased over 12 to 18 months.

What drives cost up here: the state of the connectivity model, which is the dominant variable and cannot be assessed from outside, so budget a model quality review before committing to a number. The number and age of systems to integrate. Supervisory control integration in particular, because the security boundary is non negotiable and the engineering is exacting. Mutual aid handling, since crews from other utilities need to be productive on your system within an hour of arriving. And storm scale performance testing, which is real work: a system that handles 500 events comfortably and falls over at 5,000 has failed on the only night that counts.

What keeps it down: ship the prediction engine and dispatcher tools first and run them alongside your current process through one storm season before adding the member facing layer.

When to buy instead

If you are a smaller utility with a clean, well maintained model and a conventional vendor stack, buy. Milsoft DisSPatch is widely deployed in cooperatives for good reason and Survalent covers control room needs well. An implementation is faster and cheaper than a build and we would tell you so.

If you are pursuing full advanced distribution management with volt and reactive power optimisation and fault location isolation and service restoration automation, buy. Oracle Utilities Network Management System, Schneider Electric EcoStruxure ADMS and GE Vernova PowerOn each carry engineering in those areas that is genuinely hard to rebuild, and for an investor owned utility with the model quality and the integration budget to match they are the right answer. What they ask in return is a connectivity model that meets their assumptions and an implementation measured in years, which is the trade a cooperative or municipal utility often cannot make.

Build when your model does not meet packaged assumptions and improving it is a multi year effort you cannot wait for, when your integration estate is unusual enough that the connectors are the bulk of any implementation anyway, when you need your own data out in real time for analytics rather than through a vendor report, or when quoted licensing and implementation for your member count is close to what a purpose built system would cost. That last comparison is worth actually running, because for mid sized cooperatives it is closer than most boards assume.

How to choose a developer

Ask them to explain how the prediction engine handles a model with known bad data, and expect confidence scoring and an explicit correction workflow rather than a claim that the model will be cleaned first. It will not be cleaned first.

Ask what happens when the meter head end delivers ten thousand last gasp messages in ninety seconds and then goes quiet. Storm scale behaviour is the requirement, not an optimisation.

Ask whether they have integrated MultiSpeak, and to which specific systems. Ask how they would approach supervisory control integration and listen for the security boundary before the data model. Anyone casual about that boundary should not be near your control room.

Ask how the crew application behaves with no signal for four hours, then reconnects with sixty queued updates. Conflict handling on reconnect is where these applications live or die.

Ask how they will test at storm scale before a storm, and expect a real answer involving replayed historical events at multiples of their original volume.

Get code and infrastructure ownership written into the contract before kickoff. At Digital Heroes the client owns the code from the first commit. A system your dispatchers depend on at 22:10 during the worst night of the year should belong to the utility, not to a supplier you cannot replace.