State Longitudinal Data System Development: Identity, Reproducibility and the P20W Linkage That Actually Holds

Why a state longitudinal data system is not a data warehouse project

A superintendent from a district of 40,000 students has written to the commissioner disputing a four year graduation rate published on your public dashboard. He says eleven students in the denominator transferred out of state and his staff have the documentation. Your team now has to reproduce a cohort computation from three years ago, using submissions that have been resubmitted four times since, under a business rule that was amended by the legislature in the intervening period, with an identity resolution process that merged two records in between. If you cannot reproduce the published number exactly, you will withdraw it, and withdrawing a published accountability number is the kind of event that changes who runs your office.

That is the actual job. The dashboards everybody sees are perhaps a tenth of the work. Underneath them a state longitudinal data system is three hard problems stacked: accepting data of unpredictable quality from independent local agencies who do not report to you, maintaining a single identity for a person across two decades and dozens of institutions, and computing legally consequential results in a way that can be reproduced years later. Data warehousing is the easy part and it is the part every proposal spends its pages on.

The reason this cannot be bought whole is that your state's rules are written in your state's law. Cohort membership rules, what counts as a documented transfer, which assessments substitute for which, the minimum group size for reporting, which agencies may exchange which fields and under what agreement: all of that is statute and regulation specific to you. eScholar and Public Consulting Group are both real players in this space and both have delivered for states. What they carry is a framework. What they cannot carry is your rules.

Problem 1: submissions arrive at wildly different quality and you own the result

A district with a strong data team and a modern student information system submits clean, on time, and responds to validation errors within a day. A district with one part time person submits a file with student names in the wrong column, three weeks late. You publish results for both with equal confidence, and when either one is wrong, it is your number that is wrong.

Most agencies still run this as a collection: open a window, accept files, run validation, email error reports, chase. The feedback loop is measured in days and the district staff member who has to fix it is looking at an error code and a line number without context. Ed-Fi adoption improves the transport and the model but it does not by itself fix the loop, and agencies that treated an Ed-Fi ODS as the answer have generally discovered that they replaced file wrangling with API wrangling.

What a custom build does: validation runs at submission with results returned in the district's own language, meaning student names and school names rather than surrogate keys, with a workspace where a district data coordinator can see every open issue, who it is assigned to, and what the same district looked like last collection. Certification becomes an explicit act by a named district official, recorded, so the agency and the district agree on what was certified before anything downstream consumes it.

Problem 2: the statewide student identifier is an identity problem, not a key

Assigning a number is trivial. Keeping that number attached to the right human being for twenty years is not. Students change surnames, transfer between districts mid year, enrol in two places at once during a custody dispute, appear with a nickname in one district and a legal name in another, have no social security number, are twins with adjacent birth dates, or are entered by a clerk with an autocorrected surname. Every one of those cases either creates a duplicate identity or, much worse, merges two children into one record.

eScholar's identity resolution product is genuinely good at this and several states run on it, so if you are starting from nothing it deserves evaluation. The constraint is that your matching thresholds, your tie breaking rules, and your policy for what a human reviewer may merge are policy decisions your agency owns, and a proprietary model makes those decisions harder to see and slower to change. Merges are the operations that ruin longitudinal analysis, and the ability to reverse one cleanly, years later, with full history, is the capability that decides whether your system ages well.

What a custom build does: probabilistic matching with tunable, versioned thresholds, a human review queue for the ambiguous band, and merge operations recorded as reversible events rather than destructive updates. Every downstream result stores the identity graph version it was computed against. When a merge from 2022 turns out to be wrong, you know exactly which published numbers were affected instead of guessing.

Problem 3: rules change and published numbers must still reproduce

Accountability business rules are amended by legislatures, by state board action and by federal plan revisions. Meanwhile the underlying data keeps being resubmitted and corrected. The combination means a number computed today from the same student records will not match the number you published two years ago, and both were correct at the time.

Agencies handle this today by keeping the final files in a folder and hoping. That works until someone challenges a computation rather than an input, at which point you need the rule as it existed, the code as it existed, and the data as it existed, all three together.

What a custom build does: publication is a snapshot. Every published result carries a manifest recording the input dataset versions, the business rule version, the identity graph version and the code release that produced it, and any published number can be recomputed from that manifest on demand and byte compared to the original. Business rules are expressed as versioned, reviewable definitions your analysts can read, not buried in stored procedures that only one contractor understands. When the legislature changes cohort rules, you run the new rule forward and the old rule stays available for the years it governed.

Problem 4: linking to wage and postsecondary records is a legal design, not an integration

The value of a P20W system is the question nobody can answer without it: what happened to the students who completed this program. Answering it means matching to state unemployment insurance wage records held by your workforce agency, to public postsecondary enrolment and completion, and sometimes to early childhood and corrections data. Each of those matches is governed by a data sharing agreement, by FERPA exceptions such as the audit and evaluation exception, and by state statute that may prohibit the exact join a legislator has just requested in a hearing.

The engineering follows the law, not the other way around. Where statute forbids the education agency from holding wage data, the design is a token or hash based match executed inside the workforce agency's environment returning aggregates only. Where a state permits a linked research file, the design is a separate secured environment with a defined retention clock. Getting this backwards, by building the join first and asking counsel afterwards, is how agencies end up with a system they are not allowed to run.

What a custom build does: every dataset carries its governing agreement as metadata, including permitted purposes, permitted recipients, retention period and destruction obligation, and access control is derived from that metadata rather than maintained separately in a spreadsheet by a security team. A query that would combine two datasets whose agreements do not permit combination is refused by the platform, with a message naming the agreement. That is a feature you will be grateful for during an audit.

Problem 5: publishing without re identifying a child

Small numbers are dangerous. A published table showing a subgroup of four students at one school, cross tabulated with another public table, can identify a specific child. Your state has a minimum reporting size and complementary suppression rules, and applying them consistently across dozens of report types by hand is not achievable.

What a custom build does: suppression is a service applied at publication, not a manual step per report, with the rule set versioned like every other rule. It handles complementary suppression, meaning it knows that suppressing one cell without suppressing a second lets a reader subtract to find the first. Every published artifact records which suppression version was applied.

What this costs and how long it takes

Across the 2,000-plus projects Digital Heroes has delivered, public sector data platforms of this shape follow a consistent pattern. A first production component, which should almost always be submission intake with the district facing validation workspace plus the identity spine, runs $200,000 to $450,000 and takes 5 to 9 months. A full P20W platform adding accountability rule processing with reproducible publication, interagency linkage, public dashboards and a researcher request and access workflow runs $700,000 to $2M phased over 18 to 36 months.

What drives cost up specifically here: the number of district source systems and whether you can mandate a standard, since a state that can require Ed-Fi submission has a materially cheaper project than one that must accept whatever arrives. The number of partner agencies, because every interagency agreement is a legal negotiation measured in quarters, not weeks, and the engineering waits on it. Historical migration, which is where the identity work concentrates, since twenty years of legacy identifiers have to be resolved rather than imported. And public dashboards, which look cheap and are not, because accessibility conformance and defensible suppression are both real work.

What keeps cost down: sequencing the identity spine and the intake loop first and refusing to build dashboards until they exist. Agencies that invert that order build a beautiful front end over numbers they cannot defend.

Build, buy, or contract, and how to tell the difference

Buy a component where the component is genuinely generic. Identity resolution is the strongest candidate, and eScholar has earned its position there. Adopt the Ed-Fi data standard rather than inventing a model, and adopt CEDS element definitions rather than writing your own dictionary, because interoperability with the districts and with federal reporting is worth more than a bespoke schema.

Be careful with the pure services model. Public Consulting Group and firms like it deliver real outcomes, and the risk is not competence, it is that the knowledge leaves when the contract ends and the next change is another procurement. If you contract this way, make knowledge transfer, documentation and source code delivery contractual deliverables with acceptance criteria, not good intentions.

Build the parts that encode your law: accountability rules, publication and reproducibility, suppression, the district feedback loop, and the governance metadata that controls access. Those are the parts that change when your legislature meets, and you cannot afford to file a change order every time it does.

How to choose a developer for an SLDS build

Ask them how they would reproduce a number published three years ago. If the answer does not include snapshotting inputs, rule versions and code releases together, they have built reporting and not a system of record.

Ask what happens when an identity merge performed in 2023 is found to be wrong in 2026. The answer you want is that merges are reversible events with full history and that affected downstream results are identifiable.

Ask them to describe a data sharing agreement they have actually implemented as access control rather than as a document filed by counsel.

Ask about accessibility conformance for public dashboards, because your public reporting will be held to state accessibility requirements and retrofitting is expensive. Then settle ownership in writing before kickoff: source code, infrastructure accounts, documentation and the right to hire any other firm. At Digital Heroes the agency owns all of it from the first commit. In a domain where systems outlive administrations, a vendor who resists that is a procurement risk, not a partner.