State Longitudinal Data System Problems: The 5 That Cost You Credibility, and How to Avoid Them

Why does reproducibility get scoped out of the build so often?

Because it is invisible in a demonstration. Every stakeholder can see a dashboard, nobody can see a manifest, and so the requirement that decides whether the system survives its first real challenge is the one that never makes it onto a slide. Agencies end up buying reporting and discovering, years later, that they did not buy a system of record.

The specific gap is that three things move independently and most designs track none of them. Submissions get resubmitted and corrected after publication. Business rules get amended by the legislature, by state board action and by federal plan revisions. The identity graph changes as merges and unmerges are applied. Recompute today from the same student records and you will not match the number you published two years ago, and both were correct on the day they were produced.

What has to exist instead is publication as a snapshot. Every published result carries a manifest naming the input dataset versions, the business rule version, the identity graph version and the code release that produced it, and any published figure can be recomputed from that manifest on demand and compared byte for byte with the original. Business rules belong in versioned definitions your own analysts can read and review, not buried in stored procedures that one contractor understands. Test a proposal by asking a single question: how would you reproduce a number published three years ago. If the answer does not name all four of those versions together, they have built reporting.

What goes wrong when you migrate twenty years of legacy identifiers?

Historical migration is where the identity work concentrates and it is the least visible part of the programme. You are not importing records. You are resolving two decades of identifiers assigned by districts that merged, by systems that were replaced twice, and by clerks who typed an autocorrected surname into a field that had no validation.

The dangerous outcome is not a duplicate. A duplicate is annoying and findable. The dangerous outcome is a merge that combines two children into one record, because it looks like success, it silently corrupts every longitudinal analysis that touches either student, and it is often discovered years later when a cohort count does not reconcile. Students change surnames, transfer mid year, enrol in two districts at once during a custody dispute, appear under a nickname in one district and a legal name in another, arrive with no social security number, and are sometimes twins with adjacent birth dates. Every one of those cases sits in the ambiguous band where an automated matcher will guess.

Design the migration around reversal rather than around accuracy. Merges are recorded as events with full history and can be undone cleanly years later. Matching thresholds are tunable and versioned, because they are policy decisions your agency owns rather than settings. A human review queue handles the ambiguous band, with a documented standard for what a reviewer may merge and on what evidence. And every downstream result records the identity graph version it was computed against, so when a 2022 merge turns out to be wrong in 2026 you can name the published numbers it affected instead of guessing.

Why do district submissions and interagency exchanges break after launch?

Two integrations decide whether the platform survives, and neither is the one procurement worries about.

The first is the district feed. Districts are independent local agencies who do not report to you, and their capacity varies enormously: one has a data team and a modern student information system, the next has a part time person and a spreadsheet. Adopting Ed-Fi improves the transport and the model, and you should adopt it, but agencies that treated an Ed-Fi operational data store as the whole answer have generally found they replaced file wrangling with interface wrangling. The failure after launch is not a rejected file, which is loud and gets fixed. It is a district whose submission silently stops carrying one optional element after a vendor upgrade, so a subgroup quietly shrinks and nobody notices until an accountability calculation looks odd. Reconcile counts by district and by element against the prior collection, and treat an unexplained drop as an alert rather than as data.

The second is interagency exchange with postsecondary, workforce and sometimes early childhood or corrections. These break on the legal side more often than the technical one. An agreement lapses on renewal, or a partner agency changes a field definition without telling you, or a match rate falls because their identifier population changed. Every exchange needs an acknowledgement, a match rate you monitor over time, and a named owner on both sides. Ask any developer to name the specific student information systems, the specific partner agency systems and the specific transport they have handled, because a state accounting environment and a modern application programming interface are different projects.

What happens when suppression and data sharing agreements are not covered?

Small numbers identify children. A published table showing a subgroup of four students at one school, cross tabulated against a second public table, can re-identify a specific pupil. Your state has a minimum reporting size and complementary suppression rules, and applying them by hand across dozens of report types is not achievable by any team, so the failure is a matter of when rather than whether.

Suppression has to be a service applied at publication with its own versioned rule set, not a manual step per report. It must handle complementary suppression, meaning it understands that hiding one cell without hiding a second lets a reader subtract to recover the first. Every published artefact records the suppression version that produced it, so a later question about an old table has an answer.

The parallel gap is governance. Where statute forbids your agency holding wage records, the design has to be a token or hash based match executed inside the workforce agency environment returning aggregates only. Where a linked research file is permitted, it belongs in a separately secured environment with a retention clock that actually runs. Build the join first and ask counsel afterwards and you will own a system you are not allowed to operate. The durable pattern is that every dataset carries its governing agreement as metadata, including permitted purposes, permitted recipients, retention and destruction obligation, and access is derived from that metadata rather than maintained in a security team spreadsheet. A query combining two datasets whose agreements do not permit combination is refused by the platform, with a message naming the agreement. That is the feature you will be grateful for during an audit.

Should you build custom or configure what you already own?

If you are a single district, or a small state buying a warehouse rather than operating a system of record, adopt Ed-Fi with a commercial product and stop there. A build at that scale costs more than the programme it serves.

Buy the components that are genuinely generic. Identity resolution is the strongest candidate and eScholar has earned its position there, so evaluate it before you consider building your own matcher. Adopt the Ed-Fi data standard rather than inventing a model, and adopt CEDS element definitions rather than writing your own dictionary, because interoperability with districts and with federal reporting is worth more than a bespoke schema.

Be careful with the pure services route. Public Consulting Group and firms like it deliver real outcomes, and the risk is not competence. It is that the knowledge leaves when the contract ends, so the next legislative change becomes another procurement. If you go that way, make documentation, source code delivery and knowledge transfer contractual deliverables with acceptance criteria rather than good intentions.

Build the parts that encode your law: accountability rules, publication and reproducibility, suppression, the district feedback loop, and the governance metadata that drives access. Those change every time your legislature meets, and you cannot file a change order each time it does.

How do hidden costs get into the quote?

Source system count is the first and largest. A state that can mandate a single submission standard has a materially cheaper programme than one that must accept whatever arrives, and a line item reading district integration is priced for one archetype rather than the eleven you actually have.

Partner agencies are the second, and the cost is calendar rather than code. Interagency agreements are negotiated in quarters, and engineering waits on them. Every additional partner is another legal negotiation, another identifier population and another match rate to defend.

Third, historical migration, which is identity work disguised as a data load. Fourth, public dashboards, which look cheap and are not, because accessibility conformance and defensible suppression are both real engineering and retrofitting either one is expensive. Fifth, the researcher access workflow, which is a request, review, approval, secure environment and retention lifecycle rather than a download button.

Sixth, and never in any vendor number, your own staff. Someone in your agency has to write down the matching policy, the suppression rules and the accountability definitions in a form an engineer can implement. Agencies that already hold a written methodology move quickly. Agencies where the method lives in one analyst's head should budget real weeks to get it out before anyone writes code.

What separates a build that works from one that fails here?

Sequence decides it more than anything else. The programmes that work build the identity spine and the district intake loop first, and refuse to build a dashboard until both exist. The ones that fail invert that order, because the dashboard is what everyone can see, and end up with an elegant front end over numbers the agency cannot defend when they are challenged.

Make the district experience a first class requirement rather than an afterthought. Validation results returned in the district's own language, meaning student names and school names rather than surrogate keys, with a workspace showing every open issue, who owns it, and how this district looked last collection. Certification should be an explicit recorded act by a named district official, so both parties agree what was certified before anything downstream consumes it.

Insist on append only history for identity operations from day one, because retrofitting reversibility onto destructive merges means restating years of results. Then settle ownership in writing before kickoff: source code, infrastructure accounts, documentation and the unrestricted right to hire another firm. At Digital Heroes the agency owns all of it from the first commit. These systems outlive administrations, and the cost of a rebuild driven by lock in always lands on a future commissioner rather than the one who signed.