Communications Surveillance and Archiving Problems: The 7 That Fail an Examination, and How to Avoid Them

Why does the whole budget go into capture?

Because capture is the part that made the headlines. Enforcement over off channel communications pushed firms to buy connectors quickly, and buying connectors was genuinely the right first move. Smarsh, Global Relay, Theta Lake, Shield and Behavox capture well, and rebuilding connectors to WhatsApp, Bloomberg chat, Microsoft Teams, Zoom and a voice turret would be an expensive way to reinvent something mature.

The scoping failure is stopping there and treating supervision as a checkbox the platform already covers. What an examiner actually tests is whether a qualified person reviewed the right things, on a defined basis, with documented reasoning, and whether the firm can prove it. Capture is evidence collection. Supervision is the programme.

This is specific to regulated communications because the obligation is personal rather than institutional. A named principal signs off the review. A vendor lexicon that decides what that principal sees has, in practice, been handed a piece of judgement that cannot actually be transferred by contract. When the review is examined, the firm has to explain choices it did not make.

The fix: keep buying capture and scope the build as the supervision, identity and production layer above it. Define review populations as policy decisions with a rationale, a reviewer role, a frequency and an evidentiary record: everything from this desk during a quiet period, all external messages from staff on the restricted list, a stated random sample from every registered person, plus risk triggered items. Then when the question comes, you show the policy and its version history rather than describing a keyword list.

What goes wrong with identity across channels?

This is where production requests fail, and it is almost always the same story. The same person is an email address, a Bloomberg identifier, a Teams object, a mobile number, a chat account and a turret extension. Some of those changed when they married, moved desks, or left and came back as a contractor under a different arrangement. The map linking them is a spreadsheet maintained by whoever remembers.

Two failures follow. Production misses messages that were captured correctly, because nobody knew that identifier belonged to that person. And supervision quietly under covers, because a reviewer assigned to a desk covers the identifiers on the list rather than the people on the desk, so the analyst who joined last Thursday is unreviewed and nobody notices for a quarter.

The fix: make identity a governed record rather than a lookup. Every channel identifier binds to a person with effective dates, sourced from the human resources (HR) system and the registration records so joiners, leavers and role changes propagate automatically instead of waiting for someone to update a file. Reconcile the identity map against active channel accounts on a schedule and raise an exception for any account transmitting messages that is not bound to a person. That single control converts a production request from a six week project into a query, and it is usually the cheapest high value thing in the whole build.

Why do capture feeds break quietly after launch?

Because nothing in the normal operating picture depends on them. A trading system that stops receives immediate attention. An archive that stops receiving messages from one channel looks exactly like a quiet week.

The specific triggers are mundane and recurring. A platform vendor changes an application programming interface version and the connector silently degrades to a subset of message types. A permission grant is revoked during a security tidy up. A user moves to a new device and the mobile capture profile does not follow. A regional deployment of a collaboration platform is stood up by an infrastructure team who did not know the compliance team needed to be told. Attachments capture but link previews or edited message versions do not, so the record is incomplete rather than absent, which is harder to spot.

The fix: build continuous capture assurance and treat it as the first feature rather than a monitoring afterthought. Each channel reports expected against received volumes on a schedule, using a baseline drawn from the platform's own activity data rather than from the archive itself, because an archive comparing against itself cannot see a gap. Deviations raise an alert with a named owner and a due date, and every remediation is recorded with the period affected. Then a production package can carry a completeness statement naming the window, the channels in scope, the identities included and any known gaps with their explanations.

What happens when retention, holds and disposition evidence are not covered?

Retention gets implemented as a policy, singular, and the reality is a matrix. Books and records obligations under SEC Rule 17a-4 sit alongside adviser record keeping requirements, market abuse obligations in the European Union and the United Kingdom, and privacy regimes that push in the opposite direction. A message from a trader in one jurisdiction to a client in another, held on an archive in a third, is subject to several regimes at once, and the answer to how long you keep it and who may read it is not uniform.

Legal hold then collides with all of it. A hold must freeze disposition for a specific matter without breaking the general schedule for everything else, and it must be provable years later that it was in force when it mattered.

The gap that hurts is disposition evidence. Firms can usually say what their retention policy is. Far fewer can show, for a message that was deleted, which rule permitted the deletion and confirm that no hold applied at the time.

The fix: express retention as rules over message attributes with effective dating, where the longest applicable period wins and holds override the schedule entirely. Log every disposition with the rule that authorised it and the hold check that passed. Make hold scope resolve through the same identity model as everything else, so a hold on a person covers every channel identifier they have ever had. That is defensible disposal rather than an assertion.

Should you build custom or configure what you already own?

If you are a single jurisdiction firm on email plus one chat platform, with a headcount where a principal can genuinely review a meaningful sample, and no voice obligation, buy end to end. Smarsh or Global Relay will cover capture, retention and review, and building would be an expensive route to the same place. Push your vendor first on policy configurability, sampling and reporting, because some of what firms describe as a product limit turns out to be an unconfigured module.

Build the layer above capture when two or more of these are true. You run more than about five channels across more than one capture vendor. Your reviewers close flagged items in bulk and everyone privately knows the review is a formality. You cannot produce a complete communication history for a named individual across all channels in under a day. You have jurisdictional retention conflicts resolved by somebody's judgement rather than by a rule. Or you already run trade surveillance and cannot link a message to a trading case without manual work.

The strategic line is simple. Capture is a commodity and should be bought. Supervision is your policy, your risk appetite and your accountability, and each of those is firm specific.

How do hidden costs get into the quote?

Voice priced as another channel. Transcription quality on a noisy floor, speaker separation on turret lines, tickers and code words, and multiple languages all degrade accuracy, and voice is reliably the most expensive channel to do properly. Budget it as its own phase.

Channel count priced per connector. Each capture vendor exports in its own shape, with its own identifiers, its own attachment handling and its own idea of what an edited or deleted message is, so normalising five sources is five small projects rather than one.

Multilingual review. A classifier tuned on English trading chat does not transfer to another language, and each language is a tuning exercise with its own reviewer time attached.

Volume at production speed. Storing and searching years of messages with attachments is a real engineering problem once the corpus is large, and search performance requirements have a habit of arriving after the design.

Model tuning generally. The first version of any classifier produces the wrong volume, and correcting it needs reviewer feedback loops and compliance time in the plan.

From Digital Heroes delivery experience, a focused first release covering ingestion of your existing capture feeds, identity resolution, policy defined review populations with lexicon and classifier scoring, reviewer workflow with documented sign off and defensible export runs $95,000 to $210,000 over 14 to 20 weeks. A full platform adding voice review, cross channel risk scoring, legal hold, jurisdictional retention and capture assurance runs $260,000 to $700,000 across 9 to 16 months.

What separates a build that works from one that fails here?

A review queue people believe in. Keyword matching cannot distinguish someone guaranteeing to call back after lunch from a promise about performance, so reviewers learn the pattern in a week and close in bulk. A classifier scoring messages against the behaviours your policy names, running alongside the lexicon rather than replacing it, reduces reviewed volume while raising the share of reviewed items worth reading. It stays advisory: the model prioritises, the qualified principal decides and signs.

Immutability that can be demonstrated rather than asserted. An append only store, cryptographic integrity on stored content, and a complete audit trail of every access and every deletion with the rule that authorised it. Record keeping rules allow approaches beyond traditional write once media, but whichever route you take you have to be able to show the controls working.

Capture assurance running from day one, measured against the source platform rather than against the archive.

Production as a routine operation rather than a project. If assembling a full history for four named individuals across eleven months takes more than a day, that is the number to improve before anything else.

Ownership settled before kickoff, including the classifier training data and the review policy definitions. At Digital Heroes the client owns all of it from the first commit. Your review policy expressed as code is a supervisory artefact, and a programme you cannot open and explain is one you cannot fully defend when it is examined.