Disease Surveillance Software Problems: The 7 That Cost Health Departments Real Time, and How to Avoid Them

Why does person matching get scoped as an import routine?

Every requirements document in this category contains a line about receiving electronic laboratory results. It is read as a loader. Read the message stream instead and the real problem is obvious: before it is a disease question, it is an identity question.

A positive result leaves a hospital laboratory in the evening, travels through an interface engine and arrives overnight. In the morning the queue holds 340 results. Sixty are repeat tests on people already under investigation. A dozen are the same person tested at three sites with three spellings of a name and two dates of birth. Names get truncated by sending systems. A hospital reports the ordering facility where the performing laboratory belongs. A reference laboratory changes its identifier between months. Test names arrive coded in one feed and as free text in another, and a result is positive in one system, detected in another and reactive in a third for the same organism.

Underneath all of that sits one question: is this a new person, a new episode for a known person, or a duplicate of something you already hold. A loader answers it with exact match on name and date of birth, which fails on exactly the records that matter, because the people hardest to match are frequently the people hardest to reach.

The fix is a real master person index with probabilistic matching, weights tuned to your jurisdiction's demographics rather than a vendor default, an automatic merge band, a manual review band and an unmerge path that restores original records rather than approximating them. Every match decision stores the score and the rule version, so when thresholds change you can see what would have changed historically. Scope this as the core of the build, and refuse a proposal that treats it as a data step.

What goes wrong when you migrate legacy case history?

Departments assume the old case data comes across cleanly because it was structured. It was structured by a schema, not by a discipline.

What is actually in there: the same person as four case records because the legacy system had no person concept and every case created a new demographic row. Conditions recorded against a code list revised twice, so one disease appears under two live codes and a retired third. Investigation answers stored with no version reference, so you cannot tell which form a 2019 answer belongs to. Free text notes carrying the facts that mattered, including exposures never structured. Addresses geocoded once at entry and never revalidated.

The fix is to seed the master person index from legacy data as an explicit deduplication project with human review, before the new system goes live, rather than letting the first live lab feed match against a duplicated population. Migrate current and recent cases at full fidelity with their conditions remapped to a maintained code list, and older cases as structured summary plus preserved free text. Keep the legacy system read only for a defined period. Budget the deduplication review as a named work stream staffed by people who know the jurisdiction, because a developer cannot judge whether two records with the same surname and adjacent birth dates are one person.

Why do laboratory and clinical feeds break after launch?

The standard tells you the structure. It does not tell you the quality, and it does not stop a sending facility changing something on a Tuesday without telling you.

The recurring breakages are specific. A hospital upgrades its laboratory information system and the sending facility identifier changes, so results that were routing correctly now arrive unattributed. A new test goes live at a large sender coded with a local code rather than a standard one, and it silently fails your condition mapping, which means positives for a reportable condition are landing in an unmapped bucket. A message arrives malformed and gets dropped rather than quarantined, so nobody knows it existed. Electronic case reporting from clinical systems arrives with a different message shape than laboratory reporting and with different gaps, and a build that assumed one shape has to be extended.

The fix is operational as much as technical. Quarantine bad messages for human review rather than dropping them, and alarm on the quarantine depth. Monitor per sender volume against its own baseline so a sender that stops or halves is detected within a day rather than at the end of a month. Hold the coding maps as maintained reference data with an owner, and alert on any unmapped code arriving from a sender rather than defaulting it. And onboard each sender as its own small project with its own test messages, because the standard does not make two senders equivalent.

What happens when statutory notification timeframes are not covered?

Reportable conditions carry timeframes set in state rule and the tightest are measured in hours. Most systems treat this as a report you run afterwards to find out you missed, which is the wrong shape entirely.

The failure looks like this. An immediately reportable condition arrives at 18:12 into an overnight queue. Nobody is looking at the queue. It is opened at 08:40 the next morning by an epidemiologist working top to bottom, and the on call investigator who should have been paged the previous evening finds out fourteen hours late. Nothing in the system failed. The system was never asked to know.

The fix makes the clock a first class object. When a case is created the system knows the condition, the applicable timeframe and who is on call, and it pages rather than queues for immediately reportable conditions. It escalates anything sitting unassigned past a threshold. It measures time to first contact attempt rather than time to case closure, because the first attempt is usually what the rule actually governs, and a system that only records closure will report compliance you do not have.

The second half of this is field work. Interviews happen on cell phones from a car, in a shelter, in a hospital corridor, and a system that only works on a desktop inside the department network means notes get written twice and attempt timestamps become approximations. A mobile interface for interview capture, contact elicitation and attempt logging makes the quality measure real.

Should you build custom or configure what you already own?

If you are a small county receiving a modest volume of reports, your state provides a shared instance, and your reportable conditions are the standard list handled without local variation, stay on the NEDSS Base System and put the money into disease intervention staff. It exists precisely so jurisdictions do not each build this, and it carries the national notifiable disease plumbing you would otherwise implement yourself. Conduent Maven is a defensible choice for a mid sized department that wants condition modules maintained by someone else and can live inside a vendor release cycle.

The honest test is authorship. If your epidemiologists need to define a new condition, add three questions to an investigation form and change a routing rule during an event, and the current path is a support ticket in someone else's release calendar, you have outgrown buying. During a slow year that is annoying. During an emerging event it is disqualifying, because the questions you need on Thursday were not knowable on Monday.

The other buy signal to respect: if investigators simply cannot see each other's caseloads, that may be a configuration and process project. Fix the process first, because a build layered on an undisciplined process produces a faster version of the same confusion.

How do hidden costs get into the quote?

• Sender count. A proposal prices laboratory reporting. Your reality is 40 sending facilities, each with its own mapping, its own test menu and its own test cycle. Count them, and start with the three largest, which usually carry most of the traffic.
• Electronic case reporting. Different message shape, different gaps, different onboarding. It is a second integration programme, not a variant of the first.
• Outbound national messaging. Message mapping guides are detailed and validation is strict, and getting a message accepted is a different task from generating one.
• Cross system interfaces. Immunisation registry, vital records and environmental health each require an agreement as much as an interface, and the agreement is usually the longer pole.
• Restricted condition access control. Sexually transmitted infection and HIV related records carry stricter disclosure rules in most states, and field level access with logging is architecture rather than a setting.
• Procurement and authority to operate. In government these are part of the timeline, not overhead around it.

What separates a surveillance build that works from one that fails?

The builds that work put authorship in the department's hands. Conditions, case definitions, questionnaires and routing rules are versioned configuration with an approval step, publishable by a trained analyst in an afternoon, with earlier versions preserved so a case investigated in March still shows the March form. Classification logic for confirmed, probable and suspect is expressed as readable rules against collected data rather than buried where one contractor understands it.

They also treat relationships as a graph rather than as rows with a parent case identifier. Named contact, shared exposure venue, same household, same congregate facility and genomic relatedness returned from the state laboratory are typed edges, which turns cluster detection into a query that can score a new result at intake instead of a link somebody notices in a Tuesday meeting. Exposure venues are real entities with an address, a licensee and a history, which is what lets you see the facility appearing in three unrelated investigations across two years.

And they design for surge from the start, because surge is a requirement rather than a contingency. Narrow role based access provisioned in minutes so a temporary contact tracer can do exactly one job. Intake and matching that scale horizontally, since the lab feed collapses first. Supervised bulk operations that are auditable, because during surge somebody will need to close four thousand cases at once and will do it in the database if you have not given them a supported path.

The builds that fail try to cover every reportable condition in release one. Start with your five highest volume conditions and your three largest senders. That covers most of the message traffic and teaches the build everything it needs before the long tail arrives.