Detecting Rubber-Stamp Approvals in an Agent Decision Log

You cannot detect reflex approval after the fact without the right columns. Six of the seven below are cheap to add at request time, and the seventh (quality) is the one most teams skip and later regret.

Two design notes. First, store timestamps in UTC at millisecond resolution, because the discriminating signal for a habitual clicker is often a few seconds, and second-resolution clocks round away exactly the behavior you are hunting. Second, request_quality should come from a rubric, not a gut feeling: does the request state the action, the reason, the blast radius, and what was tried? The HITL design literature recommends giving reviewers an evidence pack so the decision is obvious in 10 to 30 seconds, which implies you can score whether the pack was present (StackAI).

If you are routing approvals through an inference layer, log the model call and the human decision as separate events with their own timestamps, so that model latency never contaminates reviewer latency.

The analysis divides into four tests. Run them in order; each one narrows the suspect set.

Test 1: the floor share. Define a reading floor, the minimum seconds a competent review of a given complexity could take, set with your own reviewers rather than borrowed from a benchmark. Then compute, per reviewer, the share of decisions made under that floor. A reviewer at 40% or more under the floor is a candidate for reflex clicking, not a proven one; human review genuinely is fast for routine items, so treat the floor as a starting hypothesis to calibrate against your own data, which is the same discipline the HITL guidance applies to confidence thresholds (Galileo).

Test 2: the latency distribution. Per reviewer, compute the median, the 10th percentile (P10), and the interquartile range of latency. The tell is a distribution squeezed against zero: a P10 of 3 seconds with a median of 9 seconds means this reviewer is not reading much of anything. A healthy reviewer shows a wide spread because genuinely complex requests slow them down.

Test 3: latency versus request length. Correlate latency with request length per reviewer. A reviewer who reads should show a positive relationship: longer requests, longer decisions. A correlation near zero or negative, on a sample with real length variation, is the strongest single signal of rubber-stamping.

Test 4: approve-rate extremes. Compute each reviewer's approve share and compare it to the team's. A reviewer approving 95% or more when the team hovers near 70% is either exceptionally well-served by good requests or not rejecting anything.

A minimal implementation in Python is enough to run all four on a CSV export:

import pandas as pd

df = pd.read_csv("decision_log.csv", parse_dates=["request_ts", "decision_ts"])
df["latency_s"] = (df["decision_ts"] - df["request_ts"]).dt.total_seconds()

FLOOR_S = {"low": 10, "medium": 30, "high": 60}  # calibrate to your team
df["under_floor"] = df.apply(
    lambda r: r["latency_s"] < FLOOR_S.get(r["complexity_tag"], 30), axis=1
)

by_reviewer = df.groupby("reviewer_id").agg(
    n=("decision", "size"),
    approve_rate=("decision", lambda s: (s == "approve").mean()),
    median_latency=("latency_s", "median"),
    p10_latency=("latency_s", lambda s: s.quantile(0.10)),
    under_floor_share=("under_floor", "mean"),
)

length_corr = df.groupby("reviewer_id").apply(
    lambda g: g["latency_s"].corr(g["request_length"])
    if g["request_length"].nunique() > 5 else float("nan")
)
print(by_reviewer.join(length_corr.rename("len_corr")))

Sample size matters. Below roughly 30 decisions per reviewer, correlation estimates are too noisy to accuse anyone; report the numbers as "watch," not "flag." And the whole pipeline depends on a validated measurement system: measurement systems analysis exists precisely because gauges and operators introduce error and bias into the data a decision rests on (Majeske, International Review of Business and Economics). Your timestamps are a gauge. Calibrate them before you trust them.

Consider four reviewers on a change-approval queue, 120 to 400 decisions each over a quarter. The table below is synthetic but built to the pattern the tests described above are designed to surface.

Reviewer A looks like a reader: a median of 52 seconds, only 8% of decisions under the floor, and a strong positive relationship between latency and length. The wide gap between median and P10 shows that hard requests do slow A down.

Reviewer B is the classic pattern. A 94% approve rate, a median of 8 seconds, and 82% of decisions under the floor on requests that include a complexity tag. The near-zero correlation with length is the clincher: B is not reading more when asked to review a 40-page change than when asked to review a one-paragraph one. The pattern matches the description of approval fatigue making oversight meaningless.

Reviewer D is a subtler case. The negative correlation means D decides faster on longer requests, which is the opposite of reading behavior and hints at a different problem: skim the first line, approve, move on, with longer requests actually going quicker because the decision was made before scrolling. This is the case that a team average would hide entirely.

The insight a first search result will not give you is the decision rule. Rank reviewers by under-floor share first (it is the most interpretable), then confirm with the length correlation, then check approve-rate extremes against the team. A reviewer who trips only one test is a watch item. A reviewer who trips under-floor share and near-zero correlation together is a review-routing problem.

Here is the counterintuitive part. A request that reads well can still be approved without thought, and a beautifully written request is more likely to trigger the reflex, because the reviewer's summary instinct is satisfied in the first two lines. The measurement literature on reviewer judgment is blunt about this: building quality into a review practice is not just about improving what is submitted, it is about the quality measures of the review itself, including timeliness and the assessment of the assessment (IntechOpen).

That is why request_quality belongs in the log as a control variable, not as a headline metric. The practical rule is to slice the under-floor share by request-quality band:

• If under-floor share is high in the well-written band but low in the poorly written band, the reviewers are trusting quality as a proxy for safety. That is reasonable for low complexity and dangerous for high.
• If under-floor share is high in both bands, you have a reflex problem independent of request quality.
• If under-floor share is high only in the poorly written band, you have a request-generation problem upstream, not a reviewer problem. The agent is producing unreadable requests.

This distinction saves you from the most common misdiagnosis: firing a training initiative at reviewers when the real fix is that the agent is emitting 4,000-word requests with the decision buried in paragraph six.

Detection without design is just blame. Once you have a suspect list, three interventions change the numbers.

Instrument the gate, don't just log it. Build the pause as a real workflow state with a persistent record, so the reviewer sees the proposed action, its evidence, and its reversibility before deciding. The permit-and-approve loop pattern is exactly this: the agent proposes, the human reviews context, the agent resumes only on approval (Permit.io). A structured approval record that survives the session is also the audit trail that answers who approved what and why, which is what audit requirements actually ask for (Permit.io).

Make the decision obvious, not just available. Provide an evidence pack that lets the reviewer reach a confident answer in 10 to 30 seconds (StackAI). Paradoxically, speeding up legitimate review is how you make latency a valid discriminator: once a good review takes 20 seconds, a 3-second decision becomes unmistakably a click.

Change the queue, not the person. If a reviewer handles 400 decisions a day, reflex clicking is a capacity symptom. Route by risk so that high-blast-radius items reach a reviewer with room to think, and batch low-risk items into a single sampled screen (StackAI). The escalation-matrix literature has the vocabulary for this: functional escalation routes a case to the person with the right expertise, hierarchical escalation routes it to the person with authority, and automatic escalation moves it on rule-based triggers (Dialpad). An approval queue is a routing problem before it is a discipline problem.

One more field pays for itself here: record whether the reviewer saw the full request or a condensed view. If condensed views correlate with under-floor decisions, the UI, not the reviewer, is the cause.

If your queue runs on mixed models and you need the agent calls logged alongside the human decisions, one API key for text, image, audio, and embedding models keeps the trace in one place, which is what Quanta Inference provides.