Storm Restoration and Mutual Aid Software Problems: The 7 That Cost Real Money, and How to Avoid Them

Why does the roster intake scope failure happen so often?

Because the requirement gets written as "import crews", and importing is not the problem. What arrives when your regional mutual assistance group activates is six or eight spreadsheets in six or eight column orders, a PDF or two, and occasionally a photograph of a printed sheet. Names are spelled inconsistently. One utility sends crews with a lead and members, another sends individuals you have to assemble into crews yourself. Equipment is free text. Nothing shares a key with anything.

A developer who has built workforce systems reads that as a data cleanup task for an administrator. In an event it is a 2am transcription job standing between an arriving bus and a 6am assignment, and it is the reason the whole thing ends up back in Excel.

The fix is to scope intake as an ingestion problem with a forgiving front door rather than an import screen. Accept whatever arrives, run a structured extraction pass over it, and produce proposed crew and personnel records that a human confirms in bulk rather than retypes. This is the one place in storm work where a language model genuinely earns its keep, and it is not chat. Critically, identity must be issued at commitment, not at check in, because every record for the next two weeks, meaning assignment, lodging, hours, briefing attendance and eventually invoice reconciliation, hangs off that identifier. Create it late and you spend the entire event reconciling.

What goes wrong when identity is created at the check in desk?

Everything downstream forks. A crew that was committed on Tuesday, checked in on Wednesday under a slightly different spelling, and invoiced in March under the contractor's own employee numbering becomes three separate populations that a person has to marry up afterwards with two stacks of paper and no ability to ask questions, because the crew went home to another state weeks ago.

The visible symptom during the event is duplication. The same lineman appears twice because he arrived on a different bus than his roster said. A crew is assigned to two circuits because two coordinators created records for it. A meal headcount is submitted to the caterer for 340 when 296 people are actually on site, and nobody can prove which number was right.

The less visible symptom arrives at cost recovery. Hours cannot be tied to a named, qualified individual on a specific assignment, so the evidence package is reconstructed rather than captured. For public power utilities and eligible cooperatives that package supports a Federal Emergency Management Agency Public Assistance submission with equipment reimbursed against published rate schedules. For investor owned utilities the same evidence defends deferred storm costs in a regulatory filing where intervenors will test it line by line. Neither audience accepts a spreadsheet with a formula error.

The fix is dull and decisive: one durable identifier per person and per crew, issued at commitment, carried through check in, assignment, hours and invoice reconciliation, with every record written to an append only log so an auditor asking whether hours were adjusted after the fact receives an answer rather than a promise.

Why do the integrations that matter here break after launch?

Because they are tested on a normal Tuesday and used on the worst night of the decade. The outage management system connection is the obvious one. A storm tool should read assignment context from the outage system rather than try to replace it, and that read path is fine at low volume and unpleasant when the storm system is generating restoration progress updates for four hundred crews against an outage system already under load. Ask what the behaviour is when the outage system is slow, not when it is down. Slow is the case nobody designs for.

Contractor invoicing is the second. Each contractor arrangement carries its own rate structure and its own invoicing pattern, and the reconciliation only earns its keep if the variance queue is trustworthy. If the mapping between their line items and your logged hours was never agreed in writing, the queue fills with noise and your accounting team goes back to paper on the first event.

The third is the one that actually decides whether the system is used: connectivity. Communications infrastructure is often part of what the storm broke, and check in and hour capture are precisely the functions you cannot postpone. Every device must capture locally and reconcile on reconnect. A cloud only design does not degrade during an event, it disappears.

The fix is to test integrations under the conditions that matter. Disable the network deliberately during a tabletop exercise. Throttle the outage system connection. If the vendor has not proposed either test, they have not run a system through an event.

What happens when qualifications, escorts and briefings are not covered?

You take on a safety and liability exposure that nobody has priced. A foreign crew arrives and says they are qualified. Qualified for what, specifically. Energised work or de energised only. Which voltage class. Bucket or climbing. Do they need an escort familiar with your system, and what is your ratio. Are their tree crews certified for line clearance near energised conductors, or are they general tree care who must stay outside minimum approach distance.

On a normal week nobody thinks about this because your own crews' qualifications sit in your training records. During mutual aid it is a stack of paper handed to whoever is checking people in as fast as they physically can. If something goes wrong on a 34.5kV circuit and the investigation asks how you verified that crew, the honest answer at most utilities is that somebody said so at 2am.

The fix has to be fast or it will be bypassed. Capture the qualification claim at check in against your own defined categories, photograph the supporting document straight onto the person record on a tablet, and make assignment rules enforce it, so a circuit flagged as requiring energised work will not accept a crew without it. Escort assignment becomes a required field rather than an assumption. The arrival safety briefing gets signed on the device so attendance is a record and not a memory. The whole flow has to run in about two minutes per crew. Designed as a two minute flow it works. Designed as a form it gets skipped by the second bus.

Should you build custom or configure what you already own?

A lot of utilities should not build this, and saying so plainly matters. If you are a cooperative or municipal utility receiving fewer than roughly 30 foreign crews in a bad year, the honest advice is to standardise the roster template with your mutual assistance group, tighten your crew sheet so it captures identity and equipment consistently, and spend the capital on vegetation management or hardening where it prevents the outage instead of documenting it.

If you already run ARCOS for callout under your collective bargaining agreement and it works, keep it and do not touch it. Callout under a labour agreement is genuinely difficult and ARCOS does it well. ARCOS Crew Manager extends into storm logistics and is worth configuring properly before you consider anything custom. Clevest and Oracle Utilities Mobile Workforce Management are likewise built around your own configured crews doing dispatched work, and if that is most of your exposure, configuration is the right spend.

Build when two or more hold. You routinely host or send several hundred foreign personnel. Your cost recovery goes to a federal programme or a commission and has been questioned before. You operate across more than one state or more than one operating company, meaning two sets of cost recovery expectations. Your staging and logistics function lives in one person's spreadsheet, which means it lives in one person. Or your last event produced a lessons learned document whose top three items were coordination failures rather than field failures, which is the usual pattern and the clearest possible signal.

How do hidden costs get into the quote?

Five places. First, offline capability quoted as a checkbox. Local capture with conflict resolution on reconnect is architecture, not a setting, and retrofitting it after the first event costs more than building it.

Second, multi jurisdiction operation. Two states means two sets of documentation expectations and two sets of test cases, and it is routinely priced as one.

Third, the number of contractor arrangements. Each rate structure and invoicing pattern is separate work, and "contractor reconciliation" as a single line usually means one was examined.

Fourth, the outage management system integration, which is frequently quoted from documentation rather than from a connection someone has actually made against your version.

Fifth, the exercise. Running your real storm room against the system before the season is part of delivery, not a follow up favour, and it finds more problems than any test suite. If it is not in the schedule, it will not happen.

There is also a scheduling cost specific to this category. There is a season. A first release landing in October for an Atlantic coast utility means the team's first real use of it is an actual emergency. That is a cost even when the invoice does not show it.

What separates a build that works from one that fails here?

Scope discipline first. In Digital Heroes delivery experience the builds that work cover the first 72 hours and the hour trail, meaning roster intake with extraction, check in with qualification capture and briefing sign off, assignment, and daily hour logging with signature, and ship in 10 to 16 weeks. The builds that fail try to absorb damage assessment and estimated restoration time, which belong in the outage management system, and arrive late with a worse version of software you already own.

The second is who the software is for. Foreign crews do not get accounts. They get identifiers, and a crew lead with a device. If a developer starts by describing user registration and login, they are building an enterprise application for people who will never log in.

The third is a test you can run before signing. Ask the candidate developer to describe the evidence chain for one crew's single shift, from commitment through check in and assignment to hour capture and invoice reconciliation. Someone who has done this walks it without hesitating and mentions append only records. Someone who has not describes a timesheet screen.

Then put your storm room lead and your accounting lead in the same room with the developer for an hour before you sign anything. Those two people disagree about what matters, and how the developer handles that disagreement tells you more than any reference call. Settle code and data ownership in writing at the same time.