Peer Review Software Problems: The 7 That Cost Publishers Time and Credibility, and How to Avoid Them

Why does replicate what we do now become thirty review models?

Scope on a peer review project is usually written as one sentence: replicate our current workflow. Then discovery starts. The clinical title requires a statistical reviewer on any trial. The physics title runs a fast track for manuscripts already posted as preprints. One humanities journal publishes reviewer reports alongside the article. Two titles share a cascade where reviews travel with a rejected manuscript to a sibling journal. An associate editor layer exists on eleven journals and nowhere else. None of this is exotic. It is that every journal negotiated its own process with its own editorial board over a decade, and nobody has ever had to write the whole set down in one place.

This is specific to scholarly publishing because the review model is editorial policy rather than administrative preference. You cannot standardise it away without a conversation with each editor in chief, and those conversations take months. So the scope grows during the build instead of before it, which is the worst possible timing.

The fix is a workflow audit before a line of code. Two weeks producing one table: every journal, its stages, its blinding rule, its required declarations, its decision letter set, its deadlines, its appeal path. Most publishers find that three hundred journals resolve into nine or ten genuine patterns with per title parameters. Build the patterns as configuration and adding a title becomes an afternoon. Skip the audit and journal names end up hard coded into logic, which means every new launch is a release and every editorial change is a ticket.

What goes wrong when twenty years of manuscripts are migrated?

Legacy migration is the single largest line item in almost every editorial build, and it is the one publishers consistently price as a footnote. The reason is that a manuscript is not a row. It is a submission with versions, a set of attachments including supplementary data nobody has opened since 2011, a correspondence thread, a review history where the reviewers were identified by an email address that has since been reused by a different person, and a decision made under a policy that no longer exists.

The scholarly specific problem is reviewer identity reconciliation. Across twenty years the same person appears as three accounts with three institutional emails, and your obligation to keep anonymised reviews anonymous means you cannot simply merge them and hope. Get it wrong and you either fragment a reviewer's history, which destroys the capacity and conflict data the new system depends on, or you join two different people, which is far worse.

The fix that works: migrate the last three to five years fully, with attachments and review history intact, and keep older material available read only in the legacy system for a defined period with a documented plan for the archive. Run identity reconciliation as its own workstream with a confidence threshold and a human queue for anything ambiguous, exactly as you would for any record linkage problem. Publishers who insist on a complete migration before go live add months and usually still end up with a review queue at the end.

Why do Crossref, ORCID and screening integrations break after launch?

The integrations look identical on a slide and behave nothing alike. Crossref deposit is a schema that changes, and a deposit that validated last year can fail this year because your funder metadata is now expected in a different place. ORCID authentication is stable but the records behind it are not, and an author who revokes a permission produces a silent gap. Similarity checking, image analysis and retracted reference detection each come from a separate vendor with a separate contract, a separate rate limit and a separate idea of what an error response means. PubMed Central and funder repositories want different metadata for the same article.

These break after launch rather than during it because build teams test the happy path against a sandbox, and the failure modes are all partial. A deposit that half succeeds, a screening job that returns after the editor has already made a decision, a typesetter returning JATS with a character encoding your parser accepts and your display does not.

The fix is to treat every outbound obligation as a queue with a state, not a call. Each deposit, screening request and repository push carries its own record: attempted, accepted, rejected with reason, retried. Someone owns a dashboard of anything stuck for more than a day. Build a replay path from the first week, because you will need to redeposit a whole year of articles when a schema changes, and a system that cannot replay turns that into a manual project.

What happens when blinding and integrity screening are not covered properly?

Blinding fails through metadata. The interface hides the author name and the Word file carries it in document properties, the PDF carries it in the producer field, the uploaded figure carries a camera owner, and the filename carries the surname. A reviewer who spots that has learned something they were not meant to know and, more importantly, has learned that your process is theatre.

Integrity screening fails differently. Signals arrive from several tools, land as scores in separate fields, and no one view exists that says: this manuscript, these eleven signals, this reviewer's comment about the methods, and this pattern of three similar submissions to three of your titles in a fortnight. Journals treated as islands are exactly what a coordinated submission operation relies on.

The fix on blinding is to enforce it as a data access rule with a stripping step at upload that rewrites document metadata and normalises filenames, plus tests that assert a reviewer role cannot retrieve identifying fields through any route including email templates and export files. The fix on integrity is a case object rather than a score field, carrying evidence, correspondence and a named decision maker, with cross portfolio queries that run in one place. When a story runs about a paper you published, the only question is what you knew and when, and a case record answers it.

Should you build custom or configure what you already own?

Configure, honestly, if you publish under about ten journals with conventional review and no unusual integrity or open access entitlement requirements. Editorial Manager and ScholarOne run very large portfolios reliably, and rebuilding what they already do for a small portfolio is a poor use of money that would be better spent on editorial staff. Open Journal Systems is a legitimate answer for a university press with a named technical owner, and eJournalPress is worth a look where editorial models are unusual, because its configurability is real.

The build case is not about features. It is about who controls the release schedule. Configuration in the large platforms typically routes through the vendor's professional services team, so a new review model or a new submission gate becomes a change request in a queue. If your editorial strategy is a competitive differentiator and it is waiting on someone else's backlog, that is the argument. If your strategy is stable, it is not, and you should stay where you are.

Two other honest triggers: integrity screening involving more than three tools with no case view, and transformative agreement entitlement resolved by a person with a spreadsheet after acceptance rather than at submission. Either alone is a strong reason. Neither is a reason to replace a system that is otherwise working, and a decision layer alongside the incumbent is often the cheaper answer.

How do hidden costs get into a peer review software quote?

The pattern is consistent. Migration is quoted as data transfer and turns out to be identity reconciliation and attachment archaeology. Integrations are quoted per vendor and each one carries a contract negotiation, a sandbox wait and a credential process you do not control. Deposits are quoted as one job and are actually three with different metadata expectations. Training is quoted per person and is actually per editorial office, each of which has its own habits and its own view of what the old system did.

Two more that are specific here. First, the number of distinct review models on day one, because each is genuine configuration work even in a flexible design, and a quote based on your top three journals will miss the long tail. Second, letter templates. A publisher will casually mention decision letters and then produce four hundred variants that editors have personalised over years, every one of which someone believes is load bearing.

The way to avoid it: fix scope with a numbered list of review models, a named list of integrations with the specific document type each carries, a migration cut off date, and a template count. Ask for those five numbers before signing. In Digital Heroes delivery experience a focused first release covering submission, editor assignment, reviewer workflow and decisions runs 80,000 to 180,000 US dollars over 14 to 20 weeks, and a full platform lands at 250,000 to 600,000 over 9 to 18 months. Quotes far below that band are usually missing migration.

What separates a peer review build that works from one that fails?

The builds that work start with the editorial office, not the author. Author facing submission is the visible part and the least broken part. The pain sits with the person chasing seven invitations to secure two acceptances, and a first release that removes that work earns the adoption you need for everything else.

They also treat the model as advisory and never as a decision maker. Reviewer matching against publication embeddings genuinely surfaces qualified people an editor would not have recalled, and conflict detection against co authorship history is a real improvement on memory. Nothing about acceptance, rejection or final reviewer selection should be automated, and a developer proposing it has misread what a publisher sells.

The builds that fail share three traits. They model blinding in the interface. They run one big cutover across every journal at once instead of piloting on ten titles that share a review model. And they leave the retraction path manual, which means the workflow that matters most when the public is watching is the one nobody designed. Ask any prospective developer to describe a retraction on a whiteboard. If the answer involves editing a status field rather than versioned records and DOI relationships, keep looking.