Flood Warning System Software Problems: The 6 That Cost Real Money, and How to Avoid Them

Why does the threshold engine get built without sensor health?

Because thresholds are what the requirement document says. Rainfall depth over a duration, stage level per gauge, send an alert. It is easy to specify, easy to demonstrate and easy to sign off, and it is only half the decision.

The other half is whether the reading can be believed. The failure modes are physical and knowable. Tipping buckets clog with debris, ice up and over-report in high wind. Pressure transducers drift and stick, reporting a value that is entirely plausible and completely wrong. Stilling wells silt in. Radios produce corrupted decodes. And a stage sensor can keep reporting correctly while the channel geometry it was rated against has changed after the last flood, which means the stage to discharge relationship you are alerting on is quietly out of date.

None of that is caught by a rule that compares a number to a limit. It is caught by continuous validation: a gauge against its neighbours, against radar derived rainfall estimates, against its own recent behaviour, and against battery voltage and transmission timing. A value that has not moved in nine hours during a storm is not a reading, it is a symptom.

Retrofitting this is expensive, because confidence has to flow through every alert path and every display, and the operator needs to see why a sensor was distrusted. Specify sensor health as part of the alerting decision before estimation, not as a maintenance report to be added later.

What goes wrong when a mixed telemetry network is normalised?

A typical district network is a museum, and the normalisation work is consistently underestimated because it looks like a parsing exercise.

Some sites transmit on legacy ALERT, a one way radio protocol with no error checking that reports only when a bucket tips or a stage step is crossed. Newer sites run ALERT2, which adds framing and integrity handling. A few remote sites come through satellite data collection because there is no radio path and no cell coverage. Somewhere there is a datalogger polled over cellular that stores tables in its own structure. And there is at least one site that only one retired technician knew how to service.

Normalising those into one time series model means deciding what a reading actually means in each case. An event driven tip is not a sample. A satellite retrieval arrives in a batch on a fixed schedule, hours after the events it describes, which is fine for hydrology and dangerous if you treat the arrival time as the observation time. A polled table can contain a backfill of readings you have already processed. Every one of those needs an explicit rule, and the rules have to preserve source and quality rather than flattening everything into a value and a timestamp.

Then there is site inventory. Districts routinely find that the documented gauge list, the base station configuration and the physical reality disagree on sensor types, offsets and datums. Reconcile that before development rather than during, because a datum error produces confident wrong alerts rather than obvious failures.

Why do telemetry, forecast and notification integrations break after launch?

Three integrations carry the system and each fails in a way that is predictable.

The demodulator and telemetry path is the first, and access to it for testing is usually the schedule risk nobody planned for. You cannot validate a decode path against a production radio during a storm, so the project needs a test rig and a documented way to inject synthetic traffic. Ask for that in the plan.

Forecast and radar feeds are the second. External feeds change format, change endpoints and go down, and a system that treats a missing feed as a missing value will silently downgrade its own validation without telling anyone. Every external feed needs a freshness check and a visible degraded state, because the day the radar product is unavailable is disproportionately likely to be a day with weather.

Notification delivery is the third. Message delivery through a carrier is not confirmation that a human read anything, and this is where warning programmes fool themselves. Delivery receipts, acknowledgment and escalation are three different things. Build for acknowledgment by a named person within a defined window, with escalation on silence, and treat delivery status as a diagnostic rather than as proof.

What happens when acknowledgment and the event archive are not covered?

These two get cut to make a first release affordable, and they are the two that determine whether the system survives its first real event.

Warning programmes run on rotating duty rosters, and the person on call in July is not the one on call in December. An alert that fires into a static distribution list is not an operational system. What is needed is escalation: if the primary duty officer does not acknowledge within a defined window it goes to the backup, then to the manager, and every step is recorded. When a road crew is dispatched to barricade a low water crossing, someone has to confirm the barricade is actually up, because a closure that exists only in a work order is not a closure.

The archive is the other half. Two weeks after a flood, a council wants to know what the district knew and when. Without minute by minute replay showing what data arrived, which thresholds fired, who was notified, who acknowledged and what actions were logged, the answer is a folder of screenshots and some emails. That replay is also how you tune thresholds, and tuning without evidence is guessing with consequences.

One boundary worth holding firmly: do not build public alerting inside a hydrologic system. Hand off to whatever mass notification and federal alerting path the jurisdiction already uses, and record what was handed over and when. Public alerting carries its own approval workflows, language requirements and political risk, and owning it adds a large scope with no operational upside.

Should you build custom or configure what you already own?

Under roughly fifteen gauges on a single vendor's hardware, buy. The packaged base station will do the job and a custom build is a poor use of capital. If your gauges are not maintained, meaning batteries, desiccant, bucket cleaning and annual calibration are behind, fix that first, because better software over unreliable sensors produces confident wrong answers faster. We would say that before quoting.

OneRain Contrail is the most complete purpose built option here and genuinely knows ALERT and ALERT2 decoding, threshold alarms and hydrologic display. If your network sits inside its supported protocol set and your alerting logic fits its configuration model, it is a reasonable buy.

Keep your hydrometric data management where it is. KISTERS WISKI and Aquatic Informatics AQUARIUS are excellent at rating curves, corrections, quality coding and long term archives, which is a different job from decision support at three in the morning. Stretching either into an alerting role is a common and expensive mistake, and a build should feed them rather than replace them. Equally, buy High Sierra Electronics or Campbell Scientific field hardware and do not expect their base station tools to become a county wide warning platform.

The build case starts when you carry mixed telemetry vintages, when your basin logic needs rules a configuration screen cannot express, or when acknowledgment and escalation must match a local emergency operations plan across several agencies.

How do hidden costs get into the quote?

The count of distinct telemetry protocols is the first, because each is a real integration rather than a parser variant. Walk the network and count them, including the site nobody has touched in five years.

Redundancy and the testing regime is the second and it is where inexperienced estimates are always low. A safety critical system needs a second ingestion path, failover for the alerting service, monitoring that alerts you about the system itself through a separate channel, and a monthly test that injects synthetic gauge data through real routing into a test roster. That is engineering with an ongoing operational cost attached, not a hosting option.

A public facing map is the third, bringing load, caching and a communications review with it. Basin logic discovery is the fourth and the most commonly missed: writing down how your hydrologist actually reasons, including which tributary responds in forty minutes and which takes three hours, is weeks of paired work rather than a workshop. Schedule that person's time explicitly.

What separates a build that works from one that fails here?

Three things.

First, local knowledge becomes data rather than staying in one head. Rate of rise rules, antecedent conditions from the storm two days ago, multi gauge combination logic and per crossing thresholds tied to specific road elevations are the highest value content in the system, and they currently live with a hydrologist who will retire. A platform with nowhere to put them leaves the judgement manual, which means you have automated the plumbing and none of the expertise.

Second, the system is tested continuously rather than after the next flood. Monthly injection of synthetic gauge data that fires real thresholds through real routing into a test roster is the only way to know the whole chain still works. Systems that are exercised only by actual events fail during actual events.

Third, the specification comes from your own history. Before scoping anything, pull the last three significant events and list every alert that fired, every one that should have fired and did not, and what the sensor was doing in each case. That list, not a feature comparison, is the requirement. Then settle ownership in writing before kickoff, covering the repository, the infrastructure accounts and the full historical archive, because that archive is the record a council or a court will ask about and access to it should never depend on a supplier relationship being current.