Real World Evidence Platform Development: Why Your Cohort Cannot Be Reproduced Six Months Later

Why the second question is the one that breaks you

A HEOR team publishes an analysis showing a treatment effect in a claims-derived cohort. Nine months later a payer's own analysts get a different number and ask how the cohort was defined. The original analyst has moved teams. The code set for the indication lives in a spreadsheet with three tabs, one of which says final_v2_use. The claims extract has been refreshed twice since, so the same query on today's data returns a different denominator. The exclusion for prior therapy was applied in one script and the washout period in another, and nobody can say with confidence which version produced the published figure.

Nothing was done dishonestly. The work was simply never built to be reproduced, and reproducibility is the whole product in real world evidence. An analysis you cannot re-execute to the same result is an opinion with a chart attached, and both the FDA real world evidence framework and payer evidence reviews are explicitly interested in whether your methods were pre-specified and your data provenance is traceable.

Problem 1: the cohort definition is your intellectual property, and it lives in scripts

A cohort is not a query. It is an index date rule, a lookback window, an inclusion phenotype expressed as code sets across several vocabularies, exclusions, a washout, a censoring rule, and a set of covariates each with their own lookback. Change any one and the effect estimate moves. In most organisations all of that exists as analyst code, so the definition used in study A cannot be reused in study B without someone reading a script and hoping.

Make definitions first-class versioned objects. A phenotype has an identifier, a version, an owner, the code sets it uses in ICD-10-CM, CPT, HCPCS, NDC, RxNorm, LOINC or SNOMED CT, a written clinical rationale, and a validation record. A cohort references phenotype versions rather than inlining codes. Then a study references cohort versions, and the whole thing can be re-executed against a pinned data snapshot to produce the exact figure you published. That is what defending a cohort means in practice, and it is achievable engineering rather than an aspiration.

Problem 2: your data assets are not your platform, and vendors blur the difference

Flatiron Health is an oncology data asset built on EHR-derived and abstracted records. Komodo Health is a claims-derived asset with analytics layered on. TriNetX is a federated network that is genuinely excellent for feasibility and cohort discovery across provider data. Datavant is the linkage and tokenisation layer that lets otherwise separate assets be joined without exchanging identifiers. Aetion is the closest thing to a purpose-built RWE analysis platform with reproducibility and regulatory transparency as design goals, and it is a platform you run studies inside, on its method set and its onboarding model.

All five are serious and none of them is the thing a large evidence team eventually needs, which is a governed factory joining several separately licensed assets under their individual contractual restrictions, with methods you specify and provenance you own. If you license three datasets from three vendors and run your rigorous work in a fourth vendor's environment, you have four contracts, four data movements and a reproducibility story that depends on someone else's release notes.

What a custom build does: it puts the common data model, the definition library and the execution record under your control, while continuing to license the data assets from the people who are good at producing them. That division is the right one. Build the factory, buy the raw material.

Problem 3: licensed data comes with restrictions your software must enforce

Data licences are not casual. They typically restrict permitted uses, name which personnel or affiliates may access the data, forbid re-identification attempts, limit or forbid transfer across borders, constrain publication of certain cell sizes, require deletion at contract end and reserve audit rights for the vendor. Most organisations manage this with an email from legal and everyone's good intentions.

Put the restrictions in the system. Every dataset carries its licence terms as structured attributes: permitted purposes, permitted user groups, retention end date, minimum cell size for output, geographic constraints. Then access requests are checked against terms, outputs below the minimum cell size are suppressed by default, and expiry produces a deletion task with evidence. When the vendor exercises audit rights, you produce a report rather than a search. This is unglamorous and it is the feature that keeps a chief legal officer comfortable with the whole programme.

Problem 4: vocabularies and time grains do not line up, and OMOP is a decision not a default

Claims think in adjudicated lines with service dates and place of service. EHR thinks in encounters, orders, results and notes with wildly variable completeness. Registries think in curated study-like records. Lab data arrives with local codes that may or may not map to LOINC. Pharmacy fill data tells you what was dispensed, not what was taken.

A common data model is the way through, and OMOP from the OHDSI community is the most widely used, with Sentinel and PCORnet models serving specific ecosystems. Choosing one is a real decision with consequences: OMOP brings a large vocabulary mapping asset and a community of analytic tools, and it also imposes a shape that loses some source nuance. In our experience the right answer for most sponsors is to standardise to a common model for cross-source analysis while retaining the source-faithful layer underneath, so an analyst can always go back and check what the original record said. Systems that throw away the source layer to save storage end up unable to answer the one question that matters during a methods challenge.

Open claims versus closed claims deserves a specific mention because it catches people repeatedly. Open claims give breadth without complete capture of a patient's care, so denominators and persistence measures behave differently. If your platform does not carry that distinction as metadata on the dataset and surface it in study design, an analyst will eventually compute a rate that cannot be defended.

Problem 5: the unstructured data is where the clinical truth hides

Stage, performance status, biomarker results, reasons for discontinuation and much of oncology's most important detail live in notes and reports, not in structured fields. Manual abstraction is accurate and expensive, which is exactly why Flatiron built a business on it.

This is the strongest legitimate use of language models in the category, with conditions. Extraction from notes must be validated against a manually abstracted gold standard sample, with performance reported per variable rather than in aggregate, and the extracted value must carry its provenance including the source span. Uncertain extractions route to human abstraction rather than being accepted quietly. Model version becomes part of the study record, because a re-run under a different model version is a different study and should be labelled as one. A second reasonable use is proposing candidate code sets from a clinical description for an epidemiologist to review, which speeds phenotype development substantially. The hard prohibition is that no model produces effect estimates. Causal analyses run as pre-specified, versioned, executable specifications with the statistical method chosen by a person who will defend it.

Problem 6: compute cost scales faster than the science

A national claims dataset with several years of history is a large object, and an analyst exploring cohort definitions will run the same expensive scan repeatedly. Without governance the cloud bill becomes a quarterly conversation with finance that nobody enjoys. Practical measures that work: materialised cohort tables so a definition is computed once per data version, per-study cost attribution so leadership can see what an evidence question actually costs, query cost estimates surfaced before execution, and a development sample that analysts iterate on before running against the full asset.

What this costs and how long it takes

A first release with ingestion of two or three licensed datasets, mapping to a common data model with the source layer retained, versioned phenotype and cohort definitions and a reproducible execution record runs $110,000 to $230,000 and ships in 14 to 20 weeks, based on Digital Heroes delivery experience. A full platform adding tokenised linkage, licence term enforcement, note extraction with validation reporting, analysis packaging for regulatory and payer submission, and compute governance runs $300,000 to $750,000 phased over 9 to 16 months.

What drives price up specifically in RWE: the number and heterogeneity of source datasets, since every vendor's delivery format and refresh cadence is its own onboarding project. Linkage, if you need tokenised joins across assets. International data, where each country's privacy regime and hosting requirement is a separate design. Note extraction, because validation against abstracted samples is a real study in its own right. And any requirement for a regulatory-grade audit trail, which raises the bar on everything. What keeps it down: onboarding two datasets properly and building three real studies on them before adding a third source.

Build versus buy, and when licensing wins

Licence, and do not build, if your team runs a handful of feasibility questions a quarter. TriNetX will answer them faster than any build, and when a study needs rigour you can licence Aetion for that study. That is a sensible operating model for a mid-sized organisation and there is no shame in it.

Build when two or more of these hold. You license data from several vendors and the joining is now your problem. Your evidence supports regulatory submissions, label discussions or payer negotiations where methods will be challenged. You have been unable to reproduce a published analysis, which is the single clearest signal. Your phenotype library is a genuine asset that keeps being rebuilt. Or your legal team is uncomfortable that data licence restrictions are enforced by convention rather than by system. The tipping point is when evidence generation stops being a project team and becomes infrastructure the organisation depends on.

How to choose a developer for an RWE platform

Ask them how a study run today reproduces a number published nine months ago. A partner who has done this talks about pinned data versions, versioned definition objects and an execution record that captures the code, the parameters and the data snapshot identifier. A partner who talks about saving the SQL has not solved it.

Ask what they will do with the source-faithful layer once data is mapped to a common data model. If the answer is that it gets discarded, walk away, because you will need it during the first serious methods challenge.

Ask how licence restrictions are enforced in the system rather than in policy, and ask what happens on the day a licence expires. Then confirm in writing before kickoff that you own the repository, the infrastructure accounts, the mapped data and the definition library. The phenotype library in particular is years of epidemiological judgement and it must never be locked inside a vendor environment. At Digital Heroes it is the client's from the first commit.