Network Configuration and Change Compliance: How Do You Prove 4,000 Devices Still Match the Standard?

The change nobody wrote down

02:40 on a Wednesday. A core switch is dropping traffic for one business unit and a senior engineer fixes it by adjusting a policy map and a couple of access list entries. It works. The incident closes. He is on leave for the next nine days and the change never reaches the change record, because at 02:40 the priority was the outage and by 09:00 it was someone else's problem. Seven months later an audit samples forty devices and finds this one differs from the standard build in three ways. Nobody can say why any of them are there. Removing them is a risk because one of them is presumably load bearing. Keeping them is a finding. The engineer, when asked, genuinely does not remember.

Multiply that across a few thousand devices and a decade, and the standard build stops being a standard. It becomes a description of what devices looked like when they were commissioned. Everything since is drift, and drift is not a tidiness problem: it is why a firewall rule set contains entries nobody can explain and nobody dares remove, and it is why a failure on device A cannot be diagnosed by comparing it against device B, because they were never actually the same.

Why the regulated version of this is harder

If you are a bank, a healthcare system, a utility or a carrier, the drift problem carries an evidence requirement on top of the operational one. NERC CIP obligations for electric utilities include configuration change management and monitoring, meaning you have to show baselines, detect changes against them and document what happened. PCI DSS expects firewall and router rule sets to be reviewed on a defined periodic basis. Auditors do not accept a screenshot of a diff tool. They want to see the baseline, the deviation, the approval, the ticket and the reviewer, joined up, for a device they choose rather than one you choose.

That is a documentation problem shaped by your specific regulator and your specific exception process, which is exactly the kind of requirement that never fits a product's built in reports. Teams end up exporting from the configuration tool into a spreadsheet and reassembling the evidence by hand every audit cycle, which costs weeks and produces a document that is out of date the moment it is signed.

What SolarWinds, Cisco NSO, BackBox and the rest actually do

Be clear about what is already solved so you do not pay to rebuild it. Configuration backup, versioning and diffing is a solved problem. SolarWinds Network Configuration Manager, BackBox and Restorepoint all collect configurations on a schedule, store versions and show you what changed. If that is your requirement, buy one. It is inexpensive relative to a build and it works.

Cisco NSO is a different class of tool, a network services orchestrator with a strong model driven approach, and it is genuinely powerful where the estate suits it. Itential and Nautobot sit in the automation and source of truth space, NetBrain in network documentation and diagnostics. All real products with real users.

The constraints show up in three places. First, multi vendor reality: every product has a device support matrix, and estates that grew through acquisition contain vendors and firmware generations that sit outside it, including the ones running your most sensitive segments. Second, the exception model: real golden configurations are not one standard, they are a standard plus a legitimate variation per site, per role, per hardware generation and per regulatory zone, and expressing that in a product usually means either creating dozens of near duplicate templates or giving up on enforcement for the awkward devices. Third, evidence: products produce their reports, not your regulator's.

The result is a familiar pattern. The product handles the tidy majority, and the devices that actually worry you, the aging firewall pair in the payment segment, the industrial gateway, the vendor managed appliance, stay outside it with a manual process and a note in a runbook.

The golden configuration is a set of assertions, not a file

The single most useful design decision in this category is to stop treating the standard as a template to compare against and start treating it as a set of assertions about a device. Not this device should look like this file, but this device must have exactly these authentication servers, must have logging directed to these collectors, must not permit these management protocols on any interface facing this zone, must have this password policy, must have no access list entry permitting any to any on a boundary interface.

Assertions compose. A device inherits the global set, the set for its role, the set for its site and the set for its compliance zone, and each assertion carries its own severity and its own remediation. This solves the exception problem cleanly, because a legitimate variation becomes an assertion that does not apply to that class, recorded as a decision, rather than a template fork that nobody maintains. It also means the compliance report reads as a list of statements that are true or false about a named device, which is exactly the artefact an auditor wants and exactly what a template diff cannot produce. Build the assertions as versioned artefacts under change control, so the question of when a standard changed and who approved it has an answer.

Detecting the change is only half of it

Polling configurations nightly tells you a change happened within 24 hours. For regulated environments and for serious operational safety you want to know sooner, which means consuming device change notifications and syslog change events as well as scheduled collection, and reconciling both.

The more valuable step is correlation. An unattributed change is the problem, so the platform should attempt to attribute every detected change to an approved change record automatically by matching time window, device and requester, then push everything it cannot attribute into a review queue with a named owner. That queue is the actual product. It converts drift from an annual audit discovery into a daily two minute task, and it means the 02:40 fix gets documented on Wednesday morning while the engineer still remembers what he did, which is the only time that documentation is ever accurate.

Rollback deserves the same seriousness. Restoring a previous configuration wholesale is often unsafe because other legitimate changes have happened since. What teams actually need is targeted rollback of the specific lines that changed, with a preview of exactly what will be sent to the device, a dry run against a lab or a simulation where possible, and an approval gate for anything touching a device classified as critical.

What a custom build must include

• Multi vendor collection across the device types your estate actually contains, including the generations that fall outside product support matrices, with credential handling that satisfies your privileged access rules.
• Assertion based standards composed from global, role, site and compliance zone layers, versioned and under change control.
• Near real time change detection combining scheduled collection with device notifications and syslog events.
• Automatic attribution of changes to approved change records, with an unattributed queue that has a named owner.
• Targeted rollback with preview, dry run and approval gates for critical devices.
• Audit evidence generation in your regulator's expected format, produced on demand rather than assembled quarterly.
• Rule set analysis for firewalls, identifying shadowed, redundant and overly permissive entries with the ticket that created them where that history exists.
• Full audit logging of the platform itself, since a system that can change every device in the network is a high value target and will be treated as one by your own security team.

What it costs and how long it takes

From the projects Digital Heroes has delivered, a first release covering multi vendor collection, the assertion engine, drift detection and the unattributed change queue runs $80,000 to $160,000 and ships in 12 to 18 weeks. That version stops drift accumulating and gives you a defensible daily process. The full platform adding targeted rollback with approvals, regulator specific evidence generation, firewall rule set analysis and change record integration runs $200,000 to $450,000 phased over 6 to 12 months.

What drives cost up in this category: the number of vendors and firmware generations, since each collection path has its own quirks and the old ones have no API at all. Air gapped or segmented environments needing distributed collectors with their own approval paths. Any device where credentials must come from a privileged access management system rather than being stored. Regulatory evidence formats, which sound like reporting and are actually a data modelling requirement. And the discovery of your own standards, which in most organisations exist as a design document plus institutional memory and have to be written down properly before they can be encoded.

What keeps it down: start with one device class and one compliance zone, usually the one your auditor cares about most. The assertion framework built once extends to the rest at much lower cost.

When to buy instead

A few hundred devices from one or two mainstream vendors, no regulatory evidence requirement, and a straightforward standard: buy SolarWinds Network Configuration Manager, BackBox or Restorepoint. You will get backup, versioning, diffing and alerting quickly and cheaply, and a custom build would be an expensive way to reach the same place.

If your estate is largely Cisco and you want service level orchestration rather than compliance checking, look at NSO seriously before considering a build.

Build when your vendor mix falls outside product support in the segments that matter most, when your standards need exceptions that a template model cannot express honestly, when an auditor wants evidence in a shape no product produces, or when you already own a configuration tool and still assemble your compliance evidence in a spreadsheet. That last case is the most common trigger we see and it is usually cheaper to fix properly than to keep paying the quarterly cost of assembling it.

How to choose a developer

Ask how they will collect from your three most awkward devices by name, including anything with no usable API. The answer separates people who have automated a real estate from people who have used a vendor SDK.

Ask how they model exceptions to the standard. If the proposal is a template per variation, you will have unmaintainable sprawl within a year and the enforcement will lapse.

Ask how a detected change is attributed to a change record, and what happens to the ones that cannot be attributed. If there is no owned queue, drift will accumulate exactly as it does now, only with better graphs.

Ask how rollback is made safe: preview of the exact commands, dry run, approval gate, and what happens if the device is unreachable halfway through. A platform that can push configuration to every device in your network is either carefully designed or it is your next major incident.

Get code and infrastructure ownership written into the contract before kickoff. At Digital Heroes the client owns the code from the first commit. A system with write access to your entire network is not something to rent from a supplier you cannot replace.