Genomics Pipeline and Analysis Platform Development: How Do You Reprocess a Cohort Without the Cloud Bill Deciding the Science?

Why genomics platforms break at population scale

The aligner your pipeline uses has a new major version and the field is moving to a newer reference assembly. Reprocessing the cohort is scientifically obvious. The first question in the room is what it costs, and the honest answer at most organisations is somewhere between a number and three times that number, because nobody can decompose last quarter's cloud bill into compute for which pipeline on whose samples for which programme. The second question is whether the results will be comparable to the last analysis, and that answer depends on whether anyone recorded the container digests and reference files used two years ago, or only the workflow file name.

The stack is usually a workflow language such as Nextflow, WDL or CWL, an execution platform such as Terra, Seqera Platform, DNAnexus, Illumina BaseSpace or Velsera Seven Bridges, object storage growing without a policy, a sample manifest in a spreadsheet or a laboratory system, and consent information sitting in a completely separate registry maintained by a research operations team. Those execution platforms are good at what they claim: running workflows reliably at scale with observability. Seqera in particular is excellent at Nextflow execution and monitoring.

What none of them provides is the layer above execution: a record connecting a sample to a consent scope, to a cohort definition, to a run with everything pinned, to a cost, to a storage lifecycle. That layer is what turns a pile of sequencing output into an asset an organisation can actually govern, and it is the layer people build by hand in tickets and spreadsheets.

Problem 1: reproducibility is the environment, not the workflow file

Teams believe they have reproducibility because the workflow lives in version control. The workflow is the smallest part. Reproducing a result requires the container image digests, not just tags, since a tag can be re-pushed. It requires the exact reference assembly files and index versions. It requires the parameter set and any per-sample overrides. It requires the tool versions inside those containers, and it requires knowing whether a run was retried on a different instance type in a way that could alter numerical behaviour.

Execution platforms record run history to varying degrees, and Seqera is stronger here than most. What they do not typically do is store a complete, immutable provenance record that lets you assert, three years later and possibly on a different platform, that this result came from exactly this environment. When a reviewer or a regulator asks you to reproduce a finding and you have migrated platforms in between, the platform's own run history is not portable evidence.

A custom build makes the run registry the durable artifact and treats the execution engine as replaceable. Every run stores the workflow repository commit, resolved container digests, reference file checksums, full parameter set, input sample identifiers with their file checksums, execution environment details and the output manifest with checksums. Reproduction becomes a command rather than an archaeology project, and drift between two analyses becomes a diff you can read rather than a debate.

Problem 2: unattributed cloud spend means nobody can say no

At scale the compute bill is a scientific constraint, not a finance detail. If a principal investigator cannot see what their cohort reprocessing will cost before launching, and the organisation cannot see afterwards which programme consumed what, then spending decisions get made by whoever launches jobs most confidently. That is not governance.

Commercial platforms complicate this in two ways. Some resell compute with a margin, so the unit economics are not yours to optimise. Others run in your account but attribute costs at the platform level rather than per run, per project or per grant. Data egress is the quiet one: pulling large cohorts out of a platform or between regions produces charges that surprise people annually, and a walled garden such as an instrument vendor's own cloud makes that worse by design.

A custom build runs in your own cloud accounts with tagging enforced at job submission, so every task carries project, grant, pipeline and requester. Then two things become possible that change behaviour. Before a run, an estimator uses historical per-sample cost for that pipeline version to predict the bill within a usable range, so a 40,000 sample reprocessing decision is made with a number attached. After a run, cost lands against the programme that incurred it. Spot and preemptible capacity with automatic fallback typically cuts a large fraction of compute cost for restartable stages, and having per-pipeline cost history is what tells you which stages are worth that engineering and which are not.

Problem 3: consent decides access, and the pipeline platform has never heard of it

Identifiable genomic data carries access constraints that come from the consent a participant signed, from data use agreements with contributing sites, and from controlled access regimes such as dbGaP. Those constraints are specific: this cohort may be used for cardiovascular research but not for commercial purposes, this site's samples cannot leave a jurisdiction, this participant withdrew in 2023.

Execution platforms implement workspace and project permissions. That is a completely different concept. Workspace permissions say who can enter a room. Consent scope says which samples may be used for which purpose by which requester, and it changes over time as participants withdraw or re-consent. Bridging the two is manual at almost every organisation, which means a well meaning analyst can build a cohort that includes samples they should not have.

A custom build puts a consent-aware access layer in front of data selection. Samples carry consent scope with effective dates. A cohort request declares a purpose. The system returns only the samples permitted for that purpose at that moment, records the decision, and can reproduce later exactly which samples were and were not available and why. Withdrawal propagates: a withdrawn participant disappears from future cohorts and generates a task list for derived data already produced. That is a requirement no execution platform is designed to meet, and it is often the compliance argument that funds the build.

Problem 4: cohorts are ad hoc queries, so findings are not defensible

An analyst writes a query, gets 12,400 samples, runs the analysis, publishes. Six months later a colleague runs what they believe is the same query and gets 12,517 samples, because new samples arrived, some were quality flagged and one site's data use agreement changed. Neither number is wrong. There is simply no versioned object representing the cohort.

A custom build makes cohort definition a first-class, versioned, citable entity: the inclusion and exclusion criteria, the resolved sample list, the consent decisions applied, the timestamp, and the person who created it. Analyses reference a cohort version rather than a query. Reproducing a result reproduces the exact sample set. When the cohort is refreshed, the system shows what entered and what left, which is frequently informative in itself.

Problem 5: storage grows forever because nothing knows what is disposable

Raw sequencing output, aligned files, intermediates and per-run outputs accumulate. Intermediates are usually regeneratable from the raw data given the pinned environment, and therefore are candidates for deletion. Nobody deletes them, because nobody can prove they are regeneratable, so an organisation pays to store the same information several times over.

A custom build attaches a lifecycle class to every output at generation: raw and irreplaceable, derived and regeneratable, or published and immutable. Regeneratable outputs get an expiry, and because the run registry pins the environment, regenerating is a defined operation rather than a hope. Archive tiering applies to cold data with retrieval cost surfaced before someone triggers a bulk restore. Storage stops being a monotonic line on the finance report and becomes a policy.

What this costs and how long it takes

Across the projects Digital Heroes has delivered, a focused first release covering the run registry with full provenance pinning, cost attribution and estimation, and consent-aware sample selection runs $90,000 to $200,000 and ships in 12 to 20 weeks. A full platform adding versioned cohort management, storage lifecycle policy, re-analysis planning, controlled access auditing and a self service interface for analysts runs $250,000 to $650,000 phased over 8 to 14 months.

What drives cost up in this category specifically:

• Number of distinct pipelines, since each one needs cost profiling and lifecycle classification of its outputs
• Existing data migration, because assigning provenance and consent scope to samples processed before the system existed is genuine archaeology
• Multi-cloud or on-premise compute alongside cloud, which doubles the execution abstraction work
• Regulatory posture, if the platform supports clinical rather than research use and needs validation documentation
• Federated access across institutions, where each contributing site brings its own data use agreement and its own identity system

What keeps it down: starting with the run registry and cost attribution alone. Those two pay for themselves quickly and make every later decision cheaper to argue.

Build versus buy, and when buying is the right call

Buy if you are a single laboratory processing a few hundred samples a year on stable pipelines. Terra is well suited to academic work in that ecosystem, Seqera Platform is excellent if you are Nextflow-native, and BaseSpace is convenient if you are entirely on Illumina instruments and content to stay there. Building a platform to run a handful of workflows is a waste of scientific budget.

Build when two or more of these apply. You cannot attribute cloud spend to a programme or a grant. You hold identifiable data under consent scopes that your execution platform cannot express. You have been asked to reproduce an analysis from more than a year ago and it was difficult. Reprocessing decisions get deferred because nobody can price them. Or you are approaching an egress or licensing arrangement that makes your data expensive to move, which is the moment to notice that the platform now has leverage over your science.

A note against our own commercial interest: keep using a good execution engine. Nextflow with Seqera, or WDL with a solid runner, is not what you should rebuild. Build the governance, provenance and cost layer above it and treat the engine as swappable.

How to choose a developer for a genomics platform

Ask what they store to make a run reproducible. If the answer stops at the workflow repository and a tag, they have not been burned yet. You want resolved container digests, reference file checksums, full parameters and input checksums.

Ask how consent scope reaches sample selection. A developer who answers with role-based permissions is describing a different problem. You want purpose-bound selection with effective dates and a recorded decision that can be replayed.

Ask how they would estimate the cost of a 40,000 sample reprocessing run before it launches. The good answer involves per-sample historical cost by pipeline version and stage, with a stated confidence range, not a spreadsheet of instance prices.

Ask what they have run on spot or preemptible capacity and how they handled interruption. This is where a large share of genomics compute savings lives, and it is the difference between a platform that is cheap and one that merely looks modern.

Ask who owns the code and the infrastructure accounts, and get it in writing before kickoff. The platform should run in your cloud accounts, and you should hold the repository and the right to hire another firm. At Digital Heroes the code is yours from the first commit, and we would treat any arrangement that puts your genomic data behind someone else's billing relationship as a risk to raise before contract, not after.