SCADA Alarm Management Software Problems: The 5 That Cost Real Money, and How to Avoid Them

Why does the rationalisation get scoped as a spreadsheet so often?

Because the workshop produces a spreadsheet, and the spreadsheet is what gets handed over. It is the natural artefact of three weeks of people sitting round a table with a tag list, and it feels like the deliverable because it took the most effort.

It is not the deliverable. The deliverable is a loop. The master alarm database has to be the authority, every rationalisation decision has to be a versioned record carrying who made it and when, and something has to compare live configuration against that record on a schedule and report the difference. Without the comparison, the record decays from the moment it is signed, because the control system does not know it exists.

The distinction shows up in how drift is reported. A list of alarms that differ is not actionable and nobody opens it twice. A difference against an approved record, naming the tag, the approved value, the live value and the date the divergence appeared, is something a change board can act on in ten minutes.

The tell in a scoping conversation is whether the developer asks where the approved record lives and who is allowed to change it. If they treat the master alarm database as a table you edit, you will end up with a second spreadsheet that happens to be in a browser. If they ask about approval workflow, versioning and how you detect divergence, they understand what they are building.

What goes wrong when you load alarm and event history from several control systems?

Everything that makes the numbers comparable, which is the only reason to consolidate in the first place.

Each vendor names things differently and timestamps differently. One system records the time the alarm condition occurred, another the time the server processed it, another the time the operator workstation displayed it. Mix those without normalising and your response time metrics are measuring three different things. Time zones and daylight saving handling are the second layer of this, and they matter more than they sound, because a flood analysis that shifts by an hour lands on the wrong shift.

Priority is next. Vendors use different scales and different defaults, and most sites have a large population of tags sitting at whichever priority the configuration tool assigned when they were created. Mapping those onto a common scale is a judgement call belonging to your control room engineering team, not to the developer, and it should be explicit rather than absorbed into an import script.

Then there is console mapping, the item that quietly holds the whole project up. Alarm rate per operator requires knowing which points belonged to which console on which shift, and that mapping is rarely documented. Chase it in week one. Without it you can produce alarm counts, but not the per operator rate the published benchmarks are expressed in, which is the number that changes a conversation with a capital committee. Decide retention deliberately too, since a rolling window is cheaper and also means you cannot show a two year trend when somebody asks whether the programme worked.

Why do control system and historian integrations break after launch?

Because each extraction path is specific, and vague talk of connectors during the sales process hides how specific.

Getting alarm and event data out of one system might be an OPC alarms and events or OPC UA subscription. Another might expose a sequence of events file export dropped on a share. A third might require reading directly from a historian. A fourth might have nothing usable at all and need a vendor engagement. These are four separate pieces of work with four separate failure modes, and they should be four separate lines in a quote with the system named in each one.

After launch, the breakage is usually change on the control system side rather than in your platform. A vendor upgrade changes a schema or a service account permission. A server is rebuilt and the subscription is not restored. A remote site loses its network back to the control centre for a week and the collection has no store and forward capability, so that week is simply absent from the metrics and nobody notices because absence looks like quiet.

The design that survives this is one that monitors its own collection. Each source reports when it last delivered data, gaps are visible as gaps rather than as low alarm counts, and late arriving data can be ingested without duplicating what was already recorded. Ask any prospective developer what happens when a source is silent for four days. If the answer is that somebody would notice the chart looked odd, they have not built collection for an estate with remote sites.

What happens when management of change and suppression control are not covered?

You get a measurement tool that cannot change anything, and a suppression register that becomes a hiding place.

On change, the boundary has to be explicit and conservative. Reading alarm history is safe. Writing alarm configuration to a live control system is not, your vendor's change control exists for a reason, warranty terms often depend on it, and a deadband change applied to the wrong tag suppresses something that was doing its job. The design that works is a one way street with a human gate: the platform proposes a change set derived from the approved master alarm database, routes it through your management of change workflow with the approvals your procedure requires, generates the configuration artefact in the vendor's own import format, and then verifies after an engineer applies it that the live value matches what was approved. The platform never writes directly. It prepares, records and verifies, which happens to be exactly the evidence trail an investigator or an auditor asks for.

On suppression, the failure is quieter and more dangerous. Shelving and suppression are legitimate operational tools and they are also how an inconvenient alarm disappears permanently. Without a register carrying an owner, a reason and an expiry that is enforced rather than advisory, shelved alarms accumulate and the number nobody can produce is how many alarms are currently suppressed and why. That is a question you want to be able to answer in an hour, not a question you want asked after an event.

Should you build custom or configure what you already own?

Buy Honeywell DynAMo and stop reading if your estate is Experion end to end. It is genuinely good and genuinely native, and native configuration access will always read your system more faithfully than an outside team can. Its value falls away sharply as soon as half your alarms come from an OSI, Survalent, GE or Schneider system, because you then need a second answer for those and the consolidated view you actually wanted never appears.

Buy Hexagon PAS PlantState Suite if you are a large process plant with dedicated alarm management staff and a budget that matches. It is the most capable multi vendor product in this category and nobody serious should pretend otherwise. The two things that push utilities away from it are commercial structure, since pricing that scales with tag count gets uncomfortable across a distribution SCADA estate, and organisational weight, since it assumes a process plant model with people whose job this is.

TiPS has been in this field a long time and the analytics are sound, though it is narrower than PAS on change management and configuration authority.

Also consider doing nothing yet. If you cannot name the engineer whose job description includes owning the master alarm database, fix that before spending anything, because software does not rationalise alarms, people do, and a platform with nobody accountable becomes another unread dashboard. That is the strongest single predictor of whether this survives its second year.

Build when your estate is genuinely mixed, which describes most electric, water and gas utilities and most pipeline operators. Build when per tag commercial models are absurd at your point count. Build when the rationalisation record has to live inside your own asset hierarchy because your switching orders, operating procedures and maintenance records already do. And build when you need conformance evidence shaped the way your regulator or insurer asks for it rather than the way a vendor reports it.

How do hidden costs get into the quote?

The number of distinct control system vendors is the largest one and it is routinely folded into a single integration line. Each vendor is a separate extraction path with its own schema, its own export quirks and its own upgrade risk. Insist on a line per named system, and insist that somebody looks at the oldest one before quoting, because the system with no documented interface is where the estimate goes wrong.

Historian volume is the second. Years of history rather than a rolling window changes the storage and query design, and it is a business decision about whether you want to prove a multi year trend later.

Third is console and operator modelling. The per operator rate that every published benchmark uses requires a point to console to shift mapping that is usually undocumented, and reconstructing it is real work involving people rather than code.

Fourth is remote sites with unreliable communications, which turn collection into a store and forward problem with its own buffering, deduplication and gap reporting. Fifth is the management of change workflow, which has to match your existing procedure rather than a generic approval chain, so mapping it means sitting with the people who run your change board. Sixth is conformance reporting, where evidence shaped for a specific regulator or insurer is a defined piece of work per audience rather than a report template, and one line item means one audience.

What separates a build that works from one that fails here?

The builds that work start read only and publish numbers before touching anything. Alarm rate per operator per ten minutes with the flood periods identified, chattering alarms meaning tags that activate and clear repeatedly inside a short window, fleeting alarms that cleared before an operator could have acted, stale alarms standing beyond 24 hours, priority distribution against the shape EEMUA 191 suggests, and operator response time by priority. Publish those six per console every month and behaviour changes before a single configuration edit is made. It also gives the control room manager something to take to a capital committee that is not a feeling.

They act on the bad actor list. In most control systems a very small number of tags generate the overwhelming majority of activations, and they are the same tags month after month. Removing the top twenty is usually the largest single reduction in what an operator sees, and it needs no machine learning, only a metrics engine that computes the right things properly.

They respect the benchmarks without pretending they are targets on day one. ISA 18.2 treats more than ten alarms in ten minutes for one operator as a flood condition, and EEMUA 191 puts a manageable steady state at roughly one alarm per operator every ten minutes. Measure your own rate from a month of history first, because operations that have never measured are usually far above the benchmark and a target set before measurement is theatre.

Finally, settle ownership before kickoff and take a cheap first step. You should own the repository and the infrastructure accounts, in writing, and at Digital Heroes the client owns the code from the first commit. Then export one month of alarm and event history from your busiest console and count activations per tag. The top twenty will be roughly the same twenty a year from now unless somebody acts on them, and that list is the business case.