Turning a Compensating Control Into a Test That Fails CI

NIST SP 800-53A Rev. 5 Appendix F describes the mechanism plainly: the organization writes the desired state as data, the assessment compares it with actual state, and a mismatch indicates a defect, with the worked example being a policy that locks accounts after three failed logons against a device configured to allow five, which points at a problem with AC-7, AC-2 or CM-2. That is the same shape as a unit test, and it means the first artifact you write is not code. It is a record.

A workable control record for testing carries six things, which line up with the six items a test definition records: the control and the requirements it maps to, the scope and system of record, the expected state as a rule, a frequency matching how fast the evidence changes, the owner who acts on failure, and a version with the name of whoever approved that version. The last item matters more than it looks, because a test whose rule can be changed without approval can be made to pass.

Here is a filled-in example. A team cannot enforce phishing-resistant MFA for a set of administrative accounts on a legacy platform, so the acceptance record names an alternative: a named-admin allowlist, just-in-time elevation, session logging and a monthly access review, with the residual exposure accepted until a remediation milestone. The worked vendor example shows the same shape with concrete conditions: remediation in 60 days, final implementation in 120, monthly evidence, immediate escalation after any privileged-account incident, and automatic expiry at 120 days unless a fresh decision is approved.

For CI purposes, that memo collapses into:

Two design decisions follow from this table. First, the expiry is a test, not a calendar reminder. An expired acceptance should fail the build the same way a broken control does, because both mean the organization is carrying risk it has not re-approved. An annual review tests whether the compensating control still matches the current system, threat and business context, and the conditions that justified a workaround drift over time. Second, the threshold has a second boundary. "Zero unexplained events" catches the control breaking. "At least one review record in the last 31 days" catches the control going quiet, which is how a manual review process dies without anyone noticing.

A note on evidence freshness. NIST SP 800-53A gives each determination one of two findings, satisfied or other than satisfied, and notes that other than satisfied can also mean the assessor could not obtain enough information to decide. Keep the distinction: a control that failed and a control nobody could read need different owners and different responses. A dashboard that counts an undecided result as a pass looks greener than the environment behind it.

The test has three jobs: fetch the signal, compare against the threshold, and produce a failure message a human can act on without opening a dashboard. Keep it in the same language and framework as the rest of the suite so it inherits the existing runner, reporting and retry behavior.

A minimal check, in the shape most suites already accept:

import datetime
import os
import sys
import requests

CONTROL = {
    "id": "CC-014-privileged-elevation",
    "signal": "elevation_events_without_session_log",
    "max_value": 0,
    "min_value": 0,
    "evidence_max_age_days": 1,
    "review_max_age_days": 31,
    "owner": "identity-platform-oncall",
    "expires_on": datetime.date(2026, 3, 14),
}

def fetch(path):
    r = requests.get(f"{os.environ['EVIDENCE_API']}{path}",
                     headers={"Authorization": f"Bearer {os.environ['EVIDENCE_TOKEN']}"},
                     timeout=30)
    r.raise_for_status()
    return r.json()

def test_compensating_control_holds():
    if datetime.date.today() > CONTROL["expires_on"]:
        sys.exit(f"FAIL {CONTROL['id']}: acceptance expired on {CONTROL['expires_on']}. "
                 f"Re-approve or remediate. Owner: {CONTROL['owner']}")

    data = fetch("/elevation/summary")
    event_count = data["unexplained"]
    evidence_age = data["hours_since_read"]
    review_age = data["days_since_last_review"]

    problems = []
    if event_count != CONTROL["max_value"]:
        problems.append(f"{event_count} elevation events with no session log "
                        f"(threshold {CONTROL['max_value']})")
    if evidence_age > CONTROL["evidence_max_age_days"] * 24:
        problems.append(f"evidence stale by {evidence_age}h")
    if review_age > CONTROL["review_max_age_days"]:
        problems.append(f"last access review {review_age}d ago "
                        f"(max {CONTROL['review_max_age_days']})")

    if problems:
        sys.exit(f"FAIL {CONTROL['id']}: " + "; ".join(problems) +
                 f". Owner to notify: {CONTROL['owner']}")

Three details are deliberate. The failure exits with a message that names the control, the measured value, the threshold and the owner, so the build log alone is enough to route the alert. The evidence-freshness check is separate from the value check, because a stale feed returning yesterday's clean number is the most common way a control test silently passes. And the expiry check runs first, so an expired acceptance cannot be masked by a passing signal.

Resist the temptation to make the interlock write to production. The test reads; a human or the owner's workflow changes state. Back-porting a fix closes the CVE so no exception paperwork is needed, and the same principle applies here: the best long-term result of a failing compensating-control test is usually the underlying fix, not a renewed exception.

A control test that runs on a nightly schedule reports drift after it has already shipped. A control test that runs in the pull-request pipeline blocks the change that caused the drift from reaching production. The pipeline already has the gates; the control test just needs to sit inside one of them.

The CI stage runs automated tests on every commit or pull request, and if something fails at that stage the developer is notified right away, which is the feedback loop you want: the person who changed the elevation policy finds out in minutes, with context still fresh, rather than a reviewer finding it in a diff. Two practical constraints apply. First, latency: a test suite that takes 45 minutes to run defeats the purpose of continuous integration, so a control check that calls three APIs should be its own job that runs in parallel, not a step bolted onto the slow end-to-end block. Second, flakiness: an API timeout is undecided, not failed, and should retry or report separately so the owner does not learn to ignore the check.

Two triggers, two jobs:

• On pull request. Run the control test when the diff touches any file the control depends on (the policy definitions, the identity configuration, the evidence pipeline config). A path filter keeps it fast and keeps the signal scoped to changes that could plausibly move it.
• On schedule. Run it once a day regardless of code changes, because a control can break from an out-of-band console change that never appeared in a diff. This is the check that catches drift from working.

Make the required-status check branch-protecting. If the control test is advisory, it will be merged past, and the acceptance record will keep claiming coverage on a control that no longer holds. A test whose rule can be changed without approval can be made to pass, so protect the control record file itself with a CODEOWNERS entry that maps to the approving authority named in the memo.

The failure message is only half the alert. The other half is where it lands. Three channels, in order of how directly they reach the accountable person:

1. The pull request itself. The required check fails, the author sees the message, and the merge button is disabled. This reaches whoever caused the change, which is usually the fastest path to a fix.
2. A direct page or ticket to the named owner. The owner field in the control record maps to an on-call rotation or a queue. When a test fails, an exception opens with a named owner and a due date, and it is linked to the risk it affects, and the ticket stays linked through the retest.
3. The risk register. A failure that is not fixed within the exception window becomes a register entry with a fresh decision, because the original acceptance did not contemplate a broken compensating control.

The routing rule matters because of what happens without it. Unless controls are documented and reviewed, the organization cannot show what exception was approved, why the primary control was waived, or what specific safeguard was accepted in its place, and the practical consequence is that teams treat an exception as permanent even when it was meant to be temporary. A failing check routed to a general security channel produces exactly that outcome: everyone sees it, nobody owns it, and it stays red until the branch protection is quietly disabled.

A worked failure makes the routing concrete. The elevation summary API returns three unexplained elevation events at 09:14. The pull request that added a new admin to the allowlist fails the required check with the message from the code above. The author cannot merge. The check also opens a ticket assigned to the identity platform on-call, linked to CC-014 and to the risk it maps to in the register. The reviewer who would otherwise have approved the diff on the strength of "the allowlist is still there" never sees it, because the merge is already blocked. An hour later the cause is found: the new admin's session logging was misconfigured because the allowlist entry was added to the IdP but not to the logging policy. The fix is a second commit, the check re-runs, the count returns to zero, and the exception closes with a retest record attached.

That is the whole point. A compensating control is only as useful as its tested status, and listing compensating controls without stating whether they have been tested, what period they cover and whether any deficiencies exist tells the signer that coverage exists, not that coverage works. The CI test is what converts the second claim into the first.

Not every compensating control deserves a CI test. Score each one on two axes and pick the high-high corner first.

The minimum fields for an exception include an expiry date and a plan for closure or renewal, and exception governance rules that prevent decay require every renewal to carry evidence that the underlying blocker still exists. A CI test is the cheapest possible way to produce that evidence on a rolling basis: the passing check is the renewal evidence, dated, versioned and attached to a commit.

One more rule that keeps the program honest. If a compensating control has no artifact the test can read, that is not a reason to skip the test. It is a finding about the control. A compensating control must achieve the same security outcome or reduce risk to an acceptable level, as validated through documented risk assessments, and a control nobody can measure has not been validated. Send it back to the owner to name the enforcing artifact before the next review.

If you are building this across many controls and several teams, the shared plumbing (a private library of control checks, one place to hold the evidence credentials, and per-person keys so failures route to the right owner) is where team plans usually pay off: Quanta for Teams covers the shared credit pool, private skill library and admin controls. If you would rather design the control-testing approach against your own acceptance records first, a free 30-minute discovery call is a reasonable place to start.