How to Audit an Internal AI Service Catalog

Every entry is checked against four claims. Each test produces a pass or a fail with evidence attached.

The four tests fit together because they map to the four things a consuming team relies on: what to send, what it costs, who to call, and whether the description is current. A failure in any one of them sends builders into the same trap: they plan against a false picture and discover it in production.

Auditing every entry every quarter is wasteful. Auditing too few gives you false confidence. Use a risk-weighted sample.

The rule: sample 20 percent of entries per cycle, minimum five, rounded up. Within each entry, sample one week of logged traffic, and within that week, pull at least 200 requests stratified across consumers. Add to the sample any entry that changed since the last audit, any entry whose downstream deployment changed a model or pipeline, and any entry that generated a support ticket in the period.

The 200-request floor comes from the same logic as sampling batches of 50 to 100 tasks per workflow type when measuring AI agent cost, where the mean, median and 90th percentile are computed from the batch (Keito). Two hundred requests across a week is usually enough to surface a field that only one consumer sends, or a size limit that a single heavy user is breaching every day.

Why weight by change? Because change is where drift enters. Audit logs in AI systems are supposed to track system changes, model deployments and configuration updates, with rollback events recorded alongside (Latitude). If you log deployments, you can feed that log into the sampling rule directly: any entry touched by a deployment in the period goes into the sample automatically.

This is the test most likely to find something. Open the catalog entry and list every declared input: field name, type, whether it is required, and any stated maximum length or size. Then pull a week of request payloads for that entry and build a frequency table of every field key that actually appears.

You are looking for two mismatch types:

• Used but undeclared. A consumer sends a field the entry does not mention. Downstream teams do not know it exists, so they do not populate it, and the service silently falls back to a default. Sometimes that is harmless. Sometimes it is a fraud flag or a tenant identifier, and the default is wrong.
• Declared but never sent. The entry promises a field that no consumer has sent in a week. Either the field is dead weight or, worse, the entry describes a capability that was removed from the service and never removed from the catalog.

A system log review is the natural source for this evidence, because structured logs give you per-field records rather than free text (Appian). If your logging captures only that a call happened and not what was in it, fix that first; audit logging for AI workloads has to capture inputs and outputs with enough metadata to reconstruct what was sent (LoginRadius).

Worked example: a classification endpoint

Suppose the catalog entry for an internal text-classification service declares three fields: text, locale and max_labels. The audit pulls 400 requests from one week.

Two undeclared fields. trace_id is infrastructure noise and should be added to the entry as an optional passthrough. tenant_id is serious: every consumer sends it, so the service clearly keys its behavior on it, but the catalog never tells a new team to send it. A new consumer that omits it gets treated as a shared default tenant. That is a data isolation bug wearing a documentation costume.

The fix is two-part. Update the entry to declare tenant_id as required, and add a validation rule that rejects requests without it. If you only fix the entry, old consumers keep working by accident; if you only fix validation, you break consumers who were never told. Do both in the same change.

Worked example: a declared field that is never sent

The same service declares priority_hint with a note that it lets consumers skip the queue. The audit finds zero requests carrying it in a week. Either the field was never implemented, or it was removed and the entry stayed behind. The audit finding is not "delete the field." It is "find out why nobody uses a capability the catalog advertises, and either implement it or strike it." An advertised capability nobody can use is worse than no capability, because it trains consumers to distrust the catalog.

The second test compares what the entry says with what the meter actually charges. AI billing is harder than cloud billing here, because the unit has shifted. Cloud allocation relies on tagging resources, but an AI API call is a transaction that leaves a usage record, not a taggable asset, and the billing unit is tokens rather than calls, so the same call can cost very different amounts depending on prompt and output length (Amnic). An entry that says "charged per request" is almost always wrong for a token-billed endpoint.

Start from the invoice. Pull the billing records for the service for the audit period and read three things: the billing unit, the rate applied, and the line items that make up the total. Then reconcile a sample of requests to the invoice. Multiply the sample's token counts by the rate for the model that actually served each call and check that the result lands on the right invoice line.

Why the model matters: once routing is in play, the rate applied to a request depends on which model handled it, and that varies per request. Attribution that assumes a single rate drifts from the invoice as the routing mix shifts (GMI Cloud). If you cannot see which model served a call in your logs, you cannot reconcile, and the audit should record that as a finding in its own right.

Worked example: the per-request entry that bills by tokens

An entry says the service costs a flat internal rate per request. The meter charges per thousand tokens. The audit pulls 1,000 requests from the week. The total input and output tokens come to 2.4 million. At the app's documented per-thousand rate, the expected charge is 2,400 rate units, but the invoice line shows 2,400 units of something else entirely. The number matches by coincidence and not because the entry is right.

The fix is a decision, not just an edit. Either change the entry to state the token-based unit and show a worked example, or change the meter to charge per request. Do not leave the entry saying "per request" when the invoice says tokens; the next team that budgets on request counts will be off by an order of magnitude. This is the finance side of the catalog audit, and it connects to a broader principle: showback of AI cost gives teams visibility before chargeback moves money, and the allocation model has to be trusted before anyone is billed for it (Mavvrik). An entry with the wrong cost unit is a cost model nobody should trust.

A reconciliation like this also protects the reverse case. If the entry says tokens but the meter charges a flat platform fee, consumers will over-optimize token counts that do not affect the bill. Both directions are failures, and both are fixed the same way: make the entry and the meter describe the same unit.

Catalogs list a contact and a response target. Testing them is uncomfortable, which is why it rarely happens. Do it anyway. The test is simple: during business hours, page the listed escalation contact through the channel the entry names, describe a realistic failure, and record the time to a human response that is not an auto-reply.

Structure the finding as a small dataset, not a yes or no.

Two findings usually emerge. First, the entry names a person rather than a rotation, so the contact is stale the moment that person changes teams. Second, the backup path exists in the entry but not in practice. The fix is to make escalations point at a monitored rotation and to state the response target as a measured commitment, not an aspiration. This is the same discipline as documenting SLAs and service level targets per catalog entry, which is a standard component of a well-built catalog (Giva).

Time the response, write the number into the entry, and set the re-audit to check it again. If the measured time is far worse than the target, either improve the path or lower the stated target so the entry stops lying.

Score each entry out of 100 so the audit produces a ranked list instead of a pile of notes. Use five criteria with fixed weights.

A worked result for the classification service from the input test: inputs 30/30 after the tenant_id fix, cost unit 0/25 because the entry still says per request, escalation 10/20 because the primary answered but the backup could not diagnose, freshness 15/15, ownership 10/10. Total 65. That lands as "conditional," which means the entry stays available but gets a fix ticket with a deadline and goes back into the next audit sample. A score below 50 should restrict the entry to existing consumers until it is fixed, so new teams do not build against a false claim.

The point of the score is comparison over time. If the average score rises across cycles, the catalog is getting honest. If it falls, drift is winning. Compare the numbers to a simple threshold: any entry scoring below 50 that has an external consumer or touches money should be treated as a control failure and escalated, not just queued for a doc edit.

A fixed calendar is not enough. Certain events should force an entry back into the audit sample immediately, because change is where drift enters. An audit and assurance program is expected to review policies at least annually or upon significant changes (Cloud Security Alliance), and the same change-driven logic fits a service catalog.

Trigger a re-audit of a specific entry when any of these happens:

• The model behind the endpoint is swapped or upgraded.
• The pipeline that builds the prompt, retrieves context or post-processes output changes.
• The billing meter, rate, or invoice coding changes.
• The service owner or escalation rotation changes.
• A consumer files a support ticket that traces back to a mismatch between the entry and the service.
• A consumer is onboarded onto the entry for the first time.

That last trigger matters most. The moment a new team builds against an entry is the moment a stale claim does real damage, so the entry gets checked before the new consumer ships. Automate the trigger where you can: deployment logs and ownership systems can flag entries, and the audit sample pulls from that flag list first (Latitude).

If you want an expert to run the first cycle with you, Quanta Experts can take a line of business, read your context and hand back finished deliverables, including a scored catalog audit and fix tickets.