NERC CIP Compliance Software Problems: The 7 That Cost Real Money, and How to Avoid Them

Why does the asset inventory turn into the whole project?

The proposal describes an evidence platform. Six weeks in, the team is doing a physical reconciliation at three substations because the inventory does not match what is on the wall. This is the most common scope failure in Critical Infrastructure Protection (CIP) work, and it is not a sign of a sloppy entity. It is structural.

Asset records here come from capital project handovers, protection settings databases, maintenance systems and somebody's spreadsheet, each built for a different purpose. The protection engineer's list is organised by relay function, the maintenance system by functional location, and the compliance list was typed from a commissioning package. None carry the CIP classification chain, meaning the Bulk Electric System (BES) Cyber System and the associated Electronic Access Control or Monitoring Systems, Physical Access Control Systems and Protected Cyber Assets that inherit from it. That chain has to be built, and building it means talking to people.

The fix is to scope the first reconciliation run as a deliverable in its own right, before any cadence or reporting work is priced. Run it against two or three representative locations, count the discrepancies, and use that rate to price the rest. Entities that skip this step consistently discover equipment at substations nobody recorded, and by then the schedule has already been published to the executive sponsor.

What goes wrong when you migrate years of evidence out of SharePoint?

Historical artifacts are worth carrying forward, because your next audit will sample dates that predate your new system. Migrating them is where the quiet damage happens.

• Screenshots with no capture time. A file created date in SharePoint is not a capture time, and copying a file changes it. If the artifact does not carry a defensible time inside itself, migration cannot manufacture one, and pretending otherwise is far worse than recording it as undated.
• Attribution loss. Three engineers captured evidence over fifteen months. The folder does not say who captured what. Attribution matters because an auditor may ask whether the person who performed an evaluation was authorised to.
• Assets that no longer exist. Evidence for decommissioned equipment still needs a home, because the audit period may include the time it was in service. Migrations that only import current assets orphan that evidence.
• Version ambiguity. Folders routinely hold a report, a corrected report and a final report with names that do not indicate order. Without a content hash and an ingest time, you cannot later say which one was the artifact of record.
• Requirement mapping done by folder name. A folder called CIP-007 is not a mapping to a requirement part. Evidence has to be mapped to the specific part it supports, or the export will not assemble.

Import everything with the honest metadata you have, flag what is undated, and treat the undated set as a known risk register rather than laundering it through a new system that makes it look reliable.

Why do substation collectors stop working after go live?

Collection into an Operational Technology (OT) estate is not like collection in an information technology estate, and the failures show up months after launch rather than during testing.

The architectural rule is fixed by CIP-005: nothing reaches inward. A collector lives inside the electronic security perimeter, runs read only queries against what the OT systems already expose, writes structured output to a controlled drop and pushes outward through the reviewed path your operations traffic already uses. Any design involving an agent installed on a relay, or an inbound connection through the perimeter, will be rejected by your own cyber security review before an auditor ever sees it.

Given that constraint, three things break in service. Firmware and configuration changes from a capital project alter what a device exposes, so a read path that worked in June returns nothing in October. Credential rotation on the intermediate system silently kills collection, and because the collector pushes rather than being polled, an absence of data looks identical to a quiet period. And new substations arrive without anyone adding them to collection, so the estate grows and coverage does not.

Monitor for silence, not just for errors. Every asset with an expected collection cadence needs a heartbeat, and a missing artifact should raise an alert to a named person the day it is missed, not at audit prep. Pair that with a monthly reconciliation between the asset inventory of record and the collection roster, so a new location cannot be in scope and uncollected at the same time.

What happens when the evidence system itself is not treated as in scope?

This one catches teams late and it is expensive when it does. The repository you build holds network diagrams, addressing, access lists and asset inventories. That makes it BES Cyber System Information under CIP-011. The system whose purpose is proving your compliance is itself inside your compliance program.

The consequences are practical. The hosting decision is no longer a free choice, and a cloud arrangement needs documented handling before go live rather than as a retrofit. Access to the evidence system needs its own controls and reviews. And the firm building it becomes a vendor in your supply chain, which is a CIP-013 question you will have to answer with documentation you should be collecting during the project rather than reconstructing afterwards.

The second uncovered gap is the access revocation clock. CIP-004-6 R5.1 requires revoking unescorted physical access and interactive remote access by the end of the next calendar day following a termination. Most entities do this correctly and cannot prove it, because the proof is a human remembering to check three lists. The fix is a daily reconciliation between human resources (HR) terminations, OT account lists and badge systems, where the reconciliation record itself is the evidence. That control also catches the case everyone dreads, a contractor whose access was never tied to a termination event at all.

Should you build custom or configure what you already own?

Some registered entities should not commission software, and we say so. If your obligations sit under CIP-003 for low impact assets with one control centre, a well maintained document set plus a managed compliance service such as Certrec will cost less than the annual maintenance on anything custom. A build at that scale retires less risk than it creates.

If you are a medium impact entity with a stable footprint and no acquisitions coming, buy the point tools aimed at your worst standards and accept the seams between them. FoxGuard Solutions is built for the hardest input to CIP-007 R2, which is knowing what patches exist for industrial assets from vendors who do not publish machine readable feeds. Network Perception is aimed at firewall ruleset verification and segmentation analysis, which maps onto CIP-005 evidence. Dragos gives you OT asset visibility that is genuinely useful upstream of CIP-002 classification and CIP-010 baselines. Each is good at its own job and none of them owns your evidence chain.

Archer deserves a specific caution. It will model the standards, and if you already run it for enterprise risk it can serve as a system of record. The catch is that its data model and connectors were written for enterprise information technology risk, so the CIP specific parts, applicability by asset type, per requirement cadence and worksheet aligned export, get built inside their toolkit by consultants. That is a custom build carrying a licence fee, and OT collection remains unsolved because those connectors do not reach into a substation.

Build when scale is against you: more than roughly thirty in scope locations, a high impact control centre, or an estate that keeps changing through capital projects and acquisitions so scope reconstruction dominates every audit. Build also when you are already under a mitigation plan and your regional auditor has told you your controls are not evidenced. At that point the system is the mitigation.

How do hidden costs get into the quote?

• OT vendor platform count. Each relay family, gateway and historian is its own read path with its own quirks. A quote priced on asset count rather than platform count will be wrong, usually by a lot.
• Multiple Regional Entities. Expectations on evidence presentation differ, so you maintain more than one convention. This is rarely mentioned before contracts and never free.
• On premises hosting. Running inside the perimeter removes the straightforward hosting answers and adds infrastructure, patching and backup responsibilities that a hosted deployment would have absorbed.
• The first reconciliation. Cleaning up an asset list that turns out to be materially wrong is client side effort measured in weeks of engineering time, and everything downstream waits on it.
• Ongoing collector maintenance. Capital projects will change what devices expose. Budget for a standing maintenance allocation, because a collector estate is not a one time build.

What separates a build that works from one that fails here?

Ask a prospective developer to describe the collection path into a substation before they show you an interface. If the answer involves an agent on a relay or an inbound connection, they have not worked in this environment and your own security team will stop the project.

Ask how effective dated scope is represented. The correct answer involves asset records carrying commissioning, classification change and decommissioning dates, with the ability to query the inventory as of any prior day. If they describe a current state table, you will still be reading old commissioning emails during sampling.

Ask what an evidence object contains. Source system, collector identity, capture time and a content hash, written to an append only log, is the answer that survives a question about whether an artifact could have been backdated. Anything less is a file store with a nicer front end. Ask too whether they know the repository holds BES Cyber System Information and what that means for hosting, access and their own position in your supply chain. Someone who has done CIP work raises this before you do.

Settle ownership in writing before kickoff: the repository, the infrastructure accounts and the unrestricted right to hire another firm. At Digital Heroes the client owns the code from the first commit. For a system inside your CIP program, a vendor holding the repository is also a supply chain question you will have to answer.

One practical starting move costs nothing. Pull your last two audit findings and your last three self reports, and map each to the clock that was missed. That list is your first release scope, and it can be handed to any developer as a brief.