A Reviewer Queue for Regression Fixtures That Keep Failing

The entire mechanism fits in one small record. Keep it in a JSON file in the repo, a table in your tracker, or a database row. The format matters less than the discipline of filling in all five fields.

Stable ids matter more than they look. QA guidance on quarantine recommends a unique marker in every test title, such as @QF-101, as "a stable machine-readable test ID, not a quarantine tag" (QAJobFit on quarantining flaky tests). The tag identifies the fixture. A separate manifest decides its status. Never encode status in the test name, or you will be editing tests to change policy.

The owner field is where most implementations fail, because teams replace it with a team alias. Do not. A named owner is the difference between a ticket that gets read and a ticket that gets routed. If you cannot name one person, the fixture does not have an owner yet, and you should say so explicitly rather than filling the field with a group handle.

The stop condition is the field that ends the loop, and it should be written as an outcome, not a verb. Acceptable stop conditions are finite in number:

• Quarantine the fixture until a named review date, then fix or delete it.
• Revert or rewrite the fixture because it asserts the wrong thing.
• Fix the product bug the fixture correctly caught.
• Retire the fixture because the behavior it tests no longer exists.

Anything else ("look into it", "monitor") is a non-condition and should be rejected by whoever reviews the manifest.

The failure class does more work than any other field. It picks the owner, sets a realistic due date and narrows the list of legal stop conditions. Getting it wrong sends a real bug to the flaky queue, which is how product regressions get quarantined and forgotten.

The single most useful test is the one ContextQA states plainly: re-run on the same commit without code changes; passes on retry means flaky, fails consistently means real regression (ContextQA). Make that rerun automatic before the gate opens. A fixture that fails twice in a row on an unchanged commit is far more likely to be a product regression or a data drift than a true flake, and it should not be handed to the flaky queue by default.

Fixing the flaky class properly means naming the mechanism, not adding a retry. The common causes are async timing (fixed sleeps), shared mutable state between tests, uncontrolled external dependencies, environment inconsistency, resource contention under parallel execution, and fragile UI locators (ContextQA). A skill built for this workflow makes the same point: "Adding retries or increasing timeouts masks flakiness without fixing it," and a test must be proven flaky through repeated execution before it is quarantined, because "a single failure does not make a test flaky; it might indicate a real bug" (Flaky Test Quarantine skill). That is exactly why the gate sits at the second failure and why the class is mandatory.

The rule that makes this work is small: the second failure of the same fixture within a fixed window opens a record. Pick a window and keep it. Seven days is a reasonable default for most teams, because it catches a fixture that is red on Monday, green Tuesday, and red again the following Monday, without catching a one-off infrastructure blip.

Before the gate fires, you should have already confirmed the failure is real. Here is the sequence, written as pseudocode you can drop into a CI step:

on fixture_failure(fixture_id, commit, run_id):
    record = load_failure_history(fixture_id)          # last 30 days
    record.append(failure at commit, run_id, timestamp)

    if same_commit_passes_on_retry(fixture_id, commit):
        class = "flaky"                                 # suspected, not yet confirmed
    elif fails_on_every_run_same_commit(fixture_id, commit):
        class = "product regression or data drift"      # needs triage
    else:
        class = "environment"                           # single runner or job only

    if count_failures(record, window="7d") < 2:
        return "notify only, no ticket"

    if open_record_exists(fixture_id):
        return "append evidence to existing record"

    open_record(
        fixture_id = fixture_id,
        failure_class = class,
        owner = codeowners_for(fixture_id),
        due_date = today + sla_days(class),
        stop_condition = default_stop_condition(class)
    )

    quarantine(fixture_id, until = due_date)            # leaves the blocking suite
    return "record opened, fixture quarantined until review"

Two details make this safe. First, the gate writes the record once and appends evidence to it on later failures, so you do not get five tickets for one fixture. Second, quarantining is a status change, not a code change. Quarantine "does not require introducing a code change and can be done immediately when a flake is detected," while manually disabling a test "requires merging and pulling a code change into all open branches" (Trunk). That distinction is why an out-of-band manifest beats commenting out a test.

The escalated record should carry the evidence, not just the fact of failure. Attach the run ids, the two failure timestamps, the failure message, and whether the fixture passed on retry. A record with evidence gets triaged in minutes. A record that says "CHK-114 failing" gets triaged never.

Quarantine is routinely confused with deletion. They are opposites. Quarantined fixtures keep running in a separate, non-blocking job and keep publishing evidence, which is "a temporary control that protects delivery while keeping every flaky test measurable and accountable" (QAJobFit). Deleted or skipped fixtures produce no signal at all, and a skipped fixture "is not doing its job as a bug detector, which could introduce regressions into the system" (Trunk).

The expiry date is the part teams skip, and it is the part that keeps the queue from growing without limit. Trunk's guidance is explicit: set an SLA on how quickly quarantined tests must be resolved, decide what happens to a test quarantined beyond it, and "if a test is quarantined but not deemed a high priority to fix within the determined SLA, it should be disabled or deleted" (Trunk). Treating quarantine as a solved problem rather than a TODO "leads to an ever-growing backlog of flaky tests" that erodes trust exactly as the original flakes did.

Build that policy into CI. A small check that runs on every pull request keeps the manifest honest:

• Fail if any record has a due date in the past.
• Fail if a record references a fixture id that no longer exists.
• Fail if two records name the same fixture id.
• Fail if a record has no owner, or an owner that resolves to a group alias.
• Fail if a quarantined fixture has not produced a result in the last week, which usually means it quietly stopped running.

The same shape appears in the QA skill's project layout, with a quarantine/quarantine-registry.json file and a verification step that re-runs a fixed test "in isolation and in the full suite multiple times," because "some flakiness only manifests under specific ordering or parallel execution conditions" (Flaky Test Quarantine skill). A fixture that passes in isolation and fails in the full suite is not fixed. It is relocated.

One more rule earns its keep: mark your highest-value fixtures as never quarantine. Trunk notes that some tests validate critical behavior, and "for very high-value tests, incurring the hit to productivity when they become flaky is more valuable than allowing them to be quarantined" (Trunk). A checkout flow or a payment path should open a ticket and keep blocking. The gate still fires; the stop condition is just restricted to fixing the fixture or the product.

Suppose a suite of 400 blocking fixtures runs 200 times a week, and 2% of fixtures are flaky at any time, which is conservative against the one-in-seven figure reported from Google's research (ContextQA). That gives you 8 flaky fixtures. If each fails half the time it runs, you see 8 × 200 × 0.5 = 800 red runs a week from flakes alone.

Now assume a developer spends 6 minutes per red run deciding whether it is real, rerunning, and moving on. That is 800 × 6 = 4,800 minutes, or 80 hours of engineering time a week, spent on decisions that mostly end in "not real." Cut the same number by routing repeat failures instead. The gate opens at the second failure, so the first failure still costs 6 minutes and the second produces a record. With the fixture quarantined after the second failure inside seven days, the red runs from that fixture stop blocking anyone. The 80 hours does not vanish, because some of those failures are genuine and some investigation still happens, but the portion spent on known-flaky reruns drops toward zero for quarantined fixtures, and the remaining time is attached to a named owner with a due date.

The worked numbers also show why the window matters. A 30-day window would open records for fixtures that fail twice in a month by coincidence, and the queue would fill with records that resolve themselves. A one-day window would open almost nothing, because most flakes fail less often than daily. Seven days catches the weekly-cycle fixture that fails on Monday builds and misses the monthly one, which is a tradeoff worth stating out loud when you pick the number.

If your fixtures now call models, whether as the system under test or as a judge, the same loop extends naturally. Trace data is what lets you tell which step failed: an incorrect answer "may result from poor retrieval, a tool error, a missing state update, or a model call that received the wrong input" (Braintrust on agent tracing). A diagnosed trace can be promoted into a regression dataset row with a failure label, which is the model-side equivalent of your fixture id (Braintrust on turning production failures into regression tests). Teams running many fixtures across several model providers often route those calls through one endpoint, and Quanta Inference offers text, image, video, audio and embedding models behind a single OpenAI-compatible key, which keeps the failure record's provider field simple.

The queue itself can decay into a rubber stamp. That failure mode is well documented elsewhere: when reviewers face a high volume of items that almost always deserve approval, attention drops and "the 93% approval rate is the evidence that people have stopped reading the prompts" (Beri on human-in-the-loop approval gates). A fixture queue has the same shape: if 90% of records end in "quarantine and forget," reviewers stop reading them.

Three habits keep the review honest. First, review the manifest weekly, not daily, and review it as a list of decisions rather than a list of failures. Second, plant a known-bad record occasionally (a fixture that genuinely needs a product fix) and check that it gets triaged rather than rubber-stamped, the same technique recommended for approval gates (Beri). Third, keep the review short. Accuracy in sustained review falls as sessions lengthen, and teams do better with short, frequent review blocks than long ones (CodeAnt).

The deeper reason to keep the queue small is the asymmetry of attention. Reviewers defer to systems that appear authoritative, and that deference grows when the system supplies a confident explanation; a Harvard Business School study of 228 evaluators found reviewers were 19 percentage points more likely to align with AI recommendations when explanations were provided (The HITL rubber stamp problem). Your failure record is an explanation. If it says "flaky, suspected timing," reviewers will believe it and quarantine without checking. That is why the failure class must be a triage output, not a guess written by the CI bot at the moment of failure. Have the bot propose a class from evidence (passed on retry, failed on one runner only) and require a human to confirm it before the stop condition takes effect.

The loop closes when the record teaches something. After a quarter, count records by failure class. A high flaky count points at test design. A high environment count points at your runners. A high data drift count points at shared fixtures that should be frozen. That breakdown is the output nobody gets from reruns, and it is the reason to build the queue at all.