Communicable Disease Surveillance Software: When Lab Results Arrive Faster Than Investigators Can Match Them

Why a health department breaks the surveillance system it was given

A positive result leaves a hospital laboratory at 6:12pm. It travels through an interface engine, arrives in your surveillance system overnight, and lands in a queue. In the morning an epidemiologist opens the queue and finds 340 results. Roughly 60 of them are repeat tests on people already under investigation. Twelve are the same person tested at three different sites with three different spellings of a name and two different dates of birth. Four are conditions that were supposed to trigger a call to the investigator on duty last night, because the state rule says immediately reportable, and nobody called.

That morning queue is the whole job. Everything downstream, the interviews, the contact elicitation, the cluster detection, the state and national reporting, depends on the queue being resolved correctly and fast. And the queue is resolved by a human reading names and birth dates on a screen.

The systems in this space are real and were built by serious people. The NEDSS Base System is provided to jurisdictions and carries the national notifiable disease plumbing. Conduent Maven is a mature commercial surveillance platform with condition modules and a long track record. The trouble is not that they are bad. It is that they encode a generic jurisdiction, and every health department has its own laboratory feeds, its own state reportable condition list, its own legal disclosure limits and its own division of labor between the state, the county and the city. Where your reality differs from the generic model, you fill the gap with people, spreadsheets and email.

Across public health projects we have delivered, the recurring cost is stark. Investigator time spent on deduplication and data cleanup rather than interviews. Conditions that miss their statutory notification window because routing depended on someone reading a queue. And a reporting cycle where the numbers you send up the chain have to be reconciled by hand against your own case counts.

Problem 1: the lab feed is an identity problem before it is a disease problem

Electronic laboratory reporting arrives as HL7 messages, and the standard tells you the structure, not the quality. Names get truncated. A hospital sends the ordering facility as the performing lab. A reference laboratory reports under a different identifier than the one it used last month. Test names are coded with LOINC in one feed and free text in another. Result values are positive in one system, detected in another and reactive in a third, for the same organism.

Under all of that sits the actual question: is this a new person, a new episode for a known person, or a duplicate of something you already have. Get it wrong in one direction and you overcount cases and dispatch investigators twice. Get it wrong in the other and you merge two people, which is a genuine harm and very hard to unwind.

What a custom build must include is a real master person index with probabilistic matching, tunable thresholds, an auto merge band, a manual review band, and an unmerge path that restores the original records rather than approximating them. Every match decision is stored with the score and the rule version that produced it, so when your thresholds change you can see what would have changed. Vendor products ship matching, but the weights are theirs, and the demographics of your jurisdiction, including naming conventions, transliteration patterns and the share of records with no reliable address, should be driving those weights. This is the single most consequential piece of the build.

Problem 2: case definitions and investigation forms change faster than a release cycle

Your reportable condition list is state law, and it changes. Case definitions get revised when national surveillance guidance is updated. During an emerging event, an investigation questionnaire can change three times in a month, and the version you used last week has to remain intact for the cases collected under it.

This is where packaged surveillance products cost you time rather than money. Adding a question to an investigation form or a new condition to the routing rules typically means a change request, a vendor estimate, and a place in someone else's release calendar. In a slow year that is annoying. During an outbreak it is disqualifying, because the questions you need to ask on Thursday were not knowable on Monday.

A build has to put that authoring power in your epidemiologists' hands. Conditions, case definitions, questionnaires and routing rules are configuration with versions and an approval step, not code. A new condition should be publishable by a trained analyst in an afternoon, with the old version preserved so a case investigated in March still shows the March form. Classification logic, meaning confirmed, probable and suspect, should be expressed as readable rules against collected data rather than buried in a stored procedure that one contractor understands.

Problem 3: investigation is field work against a statutory clock

Reportable conditions carry timeframes set in state rule, and the tightest of them are measured in hours, not days. Meeting them is a routing and notification problem, and most systems treat it as a report you run afterwards to find out you missed.

The build should treat the clock as a first class object. When a case is created, the system knows the condition, the applicable timeframe, and who is on call. It pages rather than queues for immediately reportable conditions. It escalates when a case sits unassigned. It tracks time to first contact attempt, not just time to case closure, because the first attempt is what the timeframe actually governs.

Investigators also work outside the office. Interviews happen on cell phones from a car, in a shelter, in a hospital corridor. A surveillance system that only works on a desktop inside the department network means notes get written twice, once on paper and once at 5pm from memory. A mobile interface for interview capture, contact elicitation and attempt logging removes an entire class of transcription error and gives you accurate attempt timestamps, which is exactly what your quality measures depend on.

Problem 4: contacts and clusters are a graph, and lists cannot hold them

A case names three contacts. One of them tests positive and becomes a case, who names four more, two of whom attended the same event as a case from a different investigator's caseload. Represented as rows in a table with a parent case identifier, that structure is invisible. Your investigators discover the link in a Tuesday meeting, or they do not.

Cluster detection needs the relationships stored as a graph with typed edges: named contact, shared exposure venue, same household, same congregate facility, and increasingly a genomic relatedness edge coming back from the state laboratory. Then a cluster is a query rather than a hunch, and a new positive result can be scored against existing clusters at intake instead of at the weekly meeting.

The other half of this is exposure venues as real entities. A restaurant, a daycare, a long term care facility and a correctional facility each need a record with an address, a licensee, a contact, and a history of every case and complaint associated with it. Environmental health has that data in a different system, and joining the two is often the fastest win in the whole build, because the pattern nobody sees is the facility that appears in three unrelated investigations across two years.

Problem 5: surge is a design requirement, not a contingency

Everyone in public health learned this the hard way. A system sized for normal operations, meaning a few hundred results a day and 20 investigators, has to absorb ten or a hundred times that with staff who were hired last week and trained on Zoom.

Designing for surge is mostly about three things. Onboarding: role based access with narrow permissions so a temporary contact tracer can do exactly one job with no access to unrelated records, provisioned in minutes rather than through a ticket. Throughput: intake and matching that scale horizontally, because the lab feed is the first thing to collapse. And guardrails: bulk operations that a supervisor can authorize and audit, because during surge people will need to close 4,000 cases at once and they will do it in the database if you do not give them a supported path.

Legal boundaries get harder under surge, not easier. Sexually transmitted infection and HIV related records carry stricter disclosure rules in most states than general communicable disease records, and a surge workforce is exactly when someone sees a record they should not. Access control at the condition level with logging is not optional, and it is worth designing before you need it.

What this costs and how long it takes

Across the 2,000-plus projects Digital Heroes has delivered, this category prices as follows. A focused first release covering laboratory result intake, person matching and deduplication, automated case creation, and condition specific investigation workflow with the statutory clock runs $90,000 to $200,000 and ships in 16 to 24 weeks. A full platform adding contact tracing with a relationship graph, outbreak and cluster detection, provider and laboratory reporting portals, national notifiable disease messaging, mobile investigator interfaces and surge scaling runs $250,000 to $700,000 phased over 9 to 18 months.

What drives cost up in public health specifically: the number of distinct laboratory feeds, because each sending facility is its own mapping and testing exercise regardless of the standard. Electronic case reporting from clinical systems, which is a different message shape from lab reporting and arrives with different gaps. Outbound messaging to national surveillance, where message mapping guides are detailed and validation is strict. Interfaces to your immunization registry, vital records and environmental health systems, each of which is a separate negotiation as much as a technical task. And your procurement process, which in government is genuinely part of the timeline and should be planned as such.

What keeps cost down: starting with your five highest volume reportable conditions and your three largest laboratory senders. That covers the great majority of message traffic and teaches the build everything it needs to know before you add the long tail.

Build versus buy, and when buying is the right call

Buy, and we would say this to a health officer directly, if you are a small county receiving a modest volume of reports, your state provides a shared instance, and your reportable conditions are the standard list handled without local variation. The NEDSS Base System exists precisely so that jurisdictions do not each build this, and it carries the national reporting plumbing you would otherwise implement yourself. Conduent Maven is a defensible choice for a mid sized department that wants condition modules maintained by someone else and can live inside a vendor release cycle.

Build when two or more of these are true. Your jurisdiction has reportable conditions or investigation requirements that the packaged product cannot express without a change request. You are a large local health department operating under, and reporting into, a state system while needing your own operational workflow. Your epidemiologists cannot change a questionnaire without waiting on a vendor. Your lab feed volume means deduplication is consuming professional staff time daily. Or you are trying to join surveillance with environmental health, clinic services and vital records, and the packaged system treats those as other people's data.

The tipping point is authorship. If your team needs to define a new condition, a new question and a new routing rule during an event, and the current answer is a support ticket, you have outgrown buying. Most departments are not there. The ones that are already know it.

How to choose a developer

Ask them to explain probabilistic person matching and what they do when two records merge that should not have. If they cannot describe an unmerge that restores original state, they have not run a master person index in production and you will find out during an outbreak.

Ask what they have actually built against HL7 laboratory messages. The right answer is specific: segment level handling, LOINC and SNOMED mapping tables, how they cope with a sending facility that changes its identifier, and how bad messages get quarantined for review rather than silently dropped. Anyone who says they support HL7 without a story about a broken feed has not done it.

Ask how epidemiologists will publish a new condition without a developer. Watch for versioned configuration with an approval step. If the answer is that the vendor does it, you have bought the same constraint you are trying to escape.

Ask about access control for restricted conditions, hosting environment and authority to operate, and how they handle records that carry stricter state disclosure rules. A firm that has worked with health departments raises this before you do.

Ask who owns the code and settle it before kickoff. You should hold the repository, the cloud accounts and the right to hire another firm. At Digital Heroes the code belongs to the client from the first commit, and in government work that is not a preference, it is the difference between a system you control and a vendor you cannot leave.