Real World Evidence Platform Problems: The 7 That Cost Real Money, and How to Avoid Them

Why does a real world evidence platform get scoped as a data warehouse?

The requirement gets described as ingestion. We license claims from one vendor and electronic health record derived data from another, we need it in one place, analysts will query it. A competent data engineering team builds exactly that, and it works, and eighteen months later nobody can defend a published cohort.

What was left out is specific to evidence generation. A warehouse stores data. An evidence platform stores decisions about data, and the decisions are the asset. A cohort is not a query: it is an index date rule, a lookback window, an inclusion phenotype expressed as code sets across several vocabularies, exclusions, a washout, a censoring rule and covariates each with their own lookback. Change any one and the effect estimate moves. When all of that exists only as analyst code, the definition used in study A cannot be reused in study B without someone reading a script and hoping.

The fix is to make definitions first class versioned objects before any analysis runs. A phenotype carries an identifier, a version, an owner, its code sets in ICD-10-CM, CPT, HCPCS, NDC, RxNorm, LOINC or SNOMED CT, a written clinical rationale and a validation record. A cohort references phenotype versions rather than inlining codes, and a study references cohort versions plus a pinned data snapshot. Re-executing that package should return the exact published figure. Ask any prospective partner how a study run today reproduces a number published nine months ago, and if the answer is that the SQL is saved, they have not solved it.

What goes wrong when you map licensed data to a common data model?

Mapping is where most of the schedule goes and where the two decisions that shape everything afterwards get made badly.

The first is discarding the source layer. OMOP from the OHDSI community is the most widely used common data model and brings a large vocabulary mapping asset plus a community of analytic tools, with Sentinel and PCORnet serving their own ecosystems. Choosing one is a real decision with consequences, because standardisation imposes a shape and loses some source nuance. Teams delete the source faithful layer to save storage, then cannot answer the one question that arrives during the first serious methods challenge, which is what the original record actually said. Keep both: standardise for cross source analysis, retain the source layer underneath permanently.

The second is treating all claims alike. Open claims give breadth without complete capture of a patient's care, while closed claims reflect a defined enrolled population, and the difference changes denominators, persistence measures and any rate you compute. If the platform does not carry that distinction as metadata on the dataset and surface it during study design, an analyst will eventually publish a rate that cannot be defended.

The ordinary mapping problems are still real: claims think in adjudicated lines with service dates, electronic health records think in encounters, orders, results and notes with variable completeness, lab data arrives with local codes that may not map to LOINC, and pharmacy fill data tells you what was dispensed rather than what was taken. None of those is fixed by a mapping table. Each needs a documented decision recorded where an analyst will see it.

Why do the vendor data feed integrations break after launch?

Every licensed asset is a moving object. Vendors refresh on their own cadence, restate prior periods when adjudication completes, add and retire fields, and occasionally change a delivery format with limited notice. A pipeline built against one delivery works until the next one differs.

The breakages that matter are the quiet ones. A restated period changes a denominator you have already published against, so two studies run three months apart disagree for reasons unrelated to the science. A vendor drops a field and the mapping produces nulls that look like clinical absence rather than data absence. A refresh arrives partially loaded and an analyst queries it before anyone notices, producing a result wrong in a direction nobody can predict.

The design that survives treats every delivery as a versioned, immutable snapshot with a load identifier, never as an update to a live table. Studies pin a snapshot rather than reading whatever is current. Load validation compares row counts, date ranges, and per field null rates against the previous delivery and blocks promotion when something moves outside expected bounds, because a null rate jumping is usually the first visible symptom of an upstream change. And every snapshot retains its predecessor long enough to re-execute prior studies, which is the whole point of pinning.

What happens when licence terms and privacy controls are not built in?

Data licences are not casual documents. They typically restrict permitted uses, name which personnel or affiliates may access the data, forbid re-identification attempts, limit or forbid cross border transfer, constrain publication below certain cell sizes, require deletion at contract end and reserve audit rights for the vendor. Most organisations manage all of that with an email from legal and everyone's good intentions.

The failure is rarely a deliberate breach. It is a new analyst added to a group holding a dataset their project may not use, an output published below the licence cell minimum, or a dataset still sitting in cloud storage two years after the contract ended because deletion was somebody's task and not the system's. Each is discoverable during a vendor audit, and the consequence is a terminated licence for an asset your evidence programme depends on.

Put the restrictions in the system rather than in policy. Each dataset carries structured licence attributes: permitted purposes, permitted user groups, retention end date, minimum cell size for outputs, geographic constraints. Access requests are checked against terms, outputs below the minimum cell size are suppressed by default rather than by reviewer diligence, and expiry produces a deletion task with evidence of completion. When a vendor exercises audit rights you produce a report instead of running a search. This is unglamorous and it is the feature that keeps a chief legal officer comfortable with the entire programme.

Should you build custom or license what already exists?

License, and do not build, if your team runs a handful of feasibility questions a quarter. TriNetX is a federated network that is genuinely excellent for feasibility and cohort discovery across provider data and will answer those questions faster than any build. When a specific study needs rigour, license Aetion for that study: it is the closest thing to a purpose built real world evidence analysis platform with reproducibility and regulatory transparency as design goals. That is a sensible operating model for a mid sized organisation and there is no shame in it.

Be clear about what these things are, because vendors blur it. Flatiron Health is an oncology data asset built on records derived from electronic health records with abstraction on top. Komodo Health is a claims derived asset. Datavant is the tokenisation layer that lets separate assets be joined without exchanging identifiers. Those are data and plumbing, not platforms, and licensing one does not solve governance.

Build when two or more of these hold: you license from several vendors and the joining is now your problem, your evidence supports regulatory submissions or payer negotiations where methods will be challenged, you have already failed to reproduce a published analysis, your phenotype library keeps being rebuilt from scratch, or your legal team is uncomfortable that licence restrictions are enforced by convention. The correct division is to build the factory and keep buying the raw material from the people who are good at producing it.

How do hidden costs get into the quote?

A first release with ingestion of two or three licensed datasets, mapping to a common data model with the source layer retained, versioned phenotype and cohort definitions and a reproducible execution record runs $110,000 to $230,000 and ships in 14 to 20 weeks in Digital Heroes delivery experience. A full platform adding tokenised linkage, licence term enforcement, clinical note extraction with validation, analysis packaging for regulators and payers, and compute governance runs $300,000 to $750,000 phased over 9 to 16 months. Five things inflate it.

• Source dataset count and heterogeneity. Every vendor's delivery format and refresh cadence is its own onboarding project, which is why the third dataset rarely costs a third of the first three.
• Linkage. Tokenised joins across assets add a design layer and a governance layer, not just a step in the pipeline.
• International data. Each country's privacy regime and hosting requirement is a separate design, not a configuration flag.
• Note extraction. Validating against a manually abstracted gold standard is a study in its own right, with per variable performance reporting, and the abstraction labour is usually missing from the budget.
• Compute. The cost people forget entirely. A national claims dataset with years of history is a large object and an analyst iterating on definitions will scan it repeatedly.

What separates an evidence platform that works from one that fails?

Four things. The first is that reproducibility is proven rather than promised, in the first release. Publish a study, wait a month through a data refresh, re-execute the package and check that the number is identical. If it is not, you have found the gap while the platform is small rather than after a payer finds it.

The second is that compute cost is governed from the start. Materialise cohort tables so a definition is computed once per data version rather than on every exploratory run, attribute cost per study so leadership can see what an evidence question actually costs, surface query cost estimates before execution, and give analysts a development sample to iterate on before hitting the full asset. Without those, the cloud bill becomes a quarterly conversation with finance and the usual response is restricting access, which slows the science and wastes the platform.

The third is discipline about where models are allowed. Extraction from clinical notes is the strongest legitimate use, with validation against a gold standard, per variable performance reporting, provenance including the source span, and uncertain extractions routed to human abstraction. Model version becomes part of the study record, because a re-run under a different version is a different study. Proposing candidate code sets for an epidemiologist to review is a second good use. Effect estimation is not: causal analyses run as pre specified, versioned, executable specifications with a method chosen by a person who will defend it.

The fourth is ownership. Confirm in writing before kickoff that you own the repository, the infrastructure accounts, the mapped data and the definition library. The phenotype library in particular is years of epidemiological judgement and it must never be locked inside a vendor environment. At Digital Heroes it is the client's from the first commit.