Genomics Pipeline Platform Problems: The 5 That Cost Real Money, and How to Avoid Them

Why does provenance scope get cut down to the workflow file?

Teams arrive at a platform project believing reproducibility is solved because the workflow lives in version control. That belief survives the whole scoping conversation, because it is half true and it sounds rigorous. The workflow file is the smallest part of a reproducible run.

Reproducing a result requires the resolved container image digests rather than tags, the exact reference assembly files and index versions with checksums, the complete parameter set including any per sample overrides, the input sample identifiers with their file checksums, and enough detail about the execution environment to know whether a retry landed on different hardware in a way that could change numerical behaviour. When a project prices reproducibility as storing a repository commit, the cost of capturing the rest lands mid build, and it lands as an argument about whether it is really necessary.

The fix is to make the run registry the deliverable and treat the execution engine as replaceable. Every run stores the workflow commit, resolved digests, reference checksums, full parameters, input checksums, environment details and the output manifest with checksums, as an immutable record. Ask a developer what they store to make a run reproducible before you sign. If the answer stops at a repository and a tag, they have not been burned yet, and they will learn on your data. The payoff is that reproduction becomes a command and drift between two analyses becomes a readable difference rather than a debate.

What goes wrong when you assign provenance and consent to historical samples?

Backfilling the samples you processed before the platform existed is usually the largest single work package and it is almost never in the original estimate. It is genuine archaeology. Runs from three years ago may have a workflow name and a date and nothing else. Reference files have been moved. Container tags point at images that have since been rebuilt. Consent scope lives in a research operations registry maintained separately, and the join key between that registry and your sample manifest is a spreadsheet column somebody maintained by hand.

The tempting shortcut is to synthesise the missing provenance, marking a historical run as having used the pipeline version that was current at the time. Do not. A provenance record that is inferred and looks identical to one that is observed destroys the value of the whole registry, because nobody downstream can tell which is which.

The fix is a confidence flag on every historical record and an explicit unknown state. Mark what was observed, what was inferred and from what evidence, and what could not be established. Then decide, deliberately and with the science leads in the room, which historical cohorts are worth reprocessing to bring them onto a known footing and which will simply carry a caveat forever. That is a scientific decision with a cost attached, and it belongs in a meeting rather than in a data migration script.

Why do the billing, engine and sample registry integrations break after launch?

Three integrations carry this platform and each fails in its own way. Cloud billing exports are the first. Cost attribution depends on tags applied at job submission, and tags applied at submission are only as good as the enforcement. One team submits through a path that bypasses the tagging layer, usually for a good reason during an incident, and three weeks of spend lands as untagged. By the time the finance report shows it, the jobs are gone and the attribution cannot be recovered.

The execution engine is second. Nextflow, WDL and CWL runners evolve, and behaviour around retries, resource requests and reporting changes between versions in ways that quietly alter what your registry records. Third is the sample registry, where identifiers get corrected and samples get re-flagged for quality, so a change made for good laboratory reasons silently alters cohort membership downstream.

The fix is enforcement plus reconciliation rather than trust. Make tagging a hard precondition of submission with no bypass path, and run a daily reconciliation that reports untagged spend as an exception with an owner. Pin and test the engine version like any other dependency, with a regression suite that runs a known sample and compares output checksums after every upgrade. And subscribe to changes in the sample registry rather than polling it, so a quality flag applied on Tuesday surfaces against the cohorts that used that sample rather than sitting unnoticed.

What happens when consent scope is not wired into sample selection?

This is the gap that turns a technical project into a governance incident. Execution platforms implement workspace and project permissions, which answer the question of who may enter a room. Consent scope answers a completely different question: which samples may be used for which purpose by which requester, at this moment.

Where the two are not connected, and at most organisations they are not, the bridge is a person. An analyst asks research operations whether a cohort is permissible, gets an answer by email, and builds the cohort. The failure mode is not malice. It is a well meaning analyst assembling a cohort that includes samples consented for cardiovascular research only, or samples from a site whose data use agreement restricts jurisdiction, or a participant who withdrew last year. Nobody notices until an audit or, worse, a publication.

The fix is purpose bound selection at the point of query. Samples carry consent scope with effective dates. A cohort request declares its purpose. The selection layer returns only what is permitted for that purpose at that moment, records the decision, and can replay later exactly which samples were available and why. Withdrawal has to propagate, removing the participant from future cohorts and generating a task list for derived data already produced. No general execution platform is designed to do this, which is usually the argument that gets the build funded.

Should you build custom or configure what you already own?

Keep the execution engine. This is the clearest advice in the category and it runs against our own commercial interest. Nextflow with Seqera Platform is excellent at running and observing workflows at scale, Terra suits academic work in its ecosystem, and DNAnexus brings a stronger compliance posture for regulated work. Rebuilding any of that is a waste of scientific budget, and a proposal that includes an execution engine should be questioned hard.

If you are a single laboratory running a few hundred samples a year on stable pipelines, stop at the platform you already have. Configure its cost reporting, tidy your storage buckets, write down your reference file versions, and spend the money on sequencing instead. A governance layer above three workflows is engineering for its own sake.

Build the layer above the engine when two or more of these apply. You cannot attribute cloud spend to a programme or a grant. You hold identifiable data under consent scopes your execution platform cannot express. You have been asked to reproduce an analysis from more than a year ago and it was difficult. Reprocessing decisions get deferred because nobody can price them. Or an egress or licensing arrangement is starting to make your data expensive to move, which is the moment to notice that a supplier now has real power over your science.

How do hidden costs get into the quote?

Five things move the number in this category. The count of distinct pipelines is first, because each one needs cost profiling and its outputs classified for lifecycle before any of the storage savings become available. Historical backfill is second and usually the largest, for the reasons above. Multi cloud or on premise compute alongside cloud is third, and it roughly doubles the execution abstraction work rather than adding a connector.

Regulatory posture is fourth. A platform supporting clinical rather than research use carries validation documentation that is real effort and has nothing to do with features. Federated access across contributing institutions is fifth, and it is the element most likely to extend a schedule, because each site brings its own data use agreement, identity system and review committee, none of which move at engineering speed.

The fix is to phase against value rather than against completeness. The run registry and cost attribution pay for themselves quickest and make every later decision cheaper to argue, because once you can price a reprocessing run you can have the conversation about whether to do it. Start there, deliberately exclude backfill from the first release, and decide on historical cohorts once the model has proven itself on new work.

What separates a genomics platform build that works from one that fails?

Working platforms answer three questions on demand. What did this result come from, exactly. What will this reprocessing run cost, within a usable range, before I launch it. And which samples am I permitted to use for this purpose today. A platform that cannot answer all three is a job launcher with extra screens.

The cost question deserves a specific test during selection. Ask how a developer would estimate the cost of a 40,000 sample reprocessing run before it launches. The answer you want involves per sample historical cost by pipeline version and by stage, with a stated confidence range. A spreadsheet of instance prices is not an estimate, it is a hope. Ask separately what they have run on spot or preemptible capacity and how they handled interruption, because a large share of genomics compute savings lives there and handling interruption properly is the difference between a platform that is cheap and one that merely looks modern.

Storage discipline is the third marker. Classify every output at generation as raw and irreplaceable, derived and regeneratable, or published and immutable, and attach an expiry to the regeneratable class. This works only because the run registry pins the environment, which is why the provenance decision is also the storage decision. Without it nobody will authorise deletion.

Finally, run in your own cloud accounts and hold the repository from the first commit. Egress charges and platform lock in give whoever holds the billing relationship real influence over which science happens, and that influence should not sit with a supplier. Settle it in the contract before kickoff, not at handover.