Using an AI Agent to Audit a Topic's Search Results

The prompt does three jobs: it fixes the query set, it fixes the extraction fields, and it fixes the output format so two runs can be diffed. Write it once, save it, and change only the query list between runs.

You are auditing search results for a topic. Do not summarize the topic.

QUERIES (audit each one separately, in order):
1. <query>
2. <query>
...

For each query, retrieve the top 10 organic results. Exclude ads and
video carousels, but note them in a column called serp_features.

For each result, output one row with these fields:
- rank: integer 1 to 10
- url: full URL
- page_type: one of [definition, listicle, comparison, product page,
  documentation, forum thread, news, video, template]
- angle: the promise the page makes, in one sentence, in your own words
- ranking_reason: your best evidence-based guess, tagged with why
  (e.g. "exact-match title", "matches a result feature", "reads as a
  direct answer to the query"). If you cannot tell, write "unclear"
  and say what evidence is missing.
- content_read: state whether you read the page body or only the snippet
- publish_or_update_date: as shown on the page, or "not shown"
- word_count_estimate: integer, or "unknown"

Then, for the query as a whole:
- dominant_page_type
- dominant_angle
- intent_read: what the searcher most likely wants, one sentence
- what_is_missing: gaps in coverage by the top 10, as a bullet list

Return a single markdown table per query. Do not invent fields. If a page
blocks retrieval, write "blocked" in content_read and leave other fields
as "unknown".

Two design choices matter more than the wording. First, content_read forces the agent to declare whether it read the page or only the snippet, which is the difference between an audit and a guess. Second, ranking_reason demands evidence and permits "unclear." An agent told it must explain every ranking will invent explanations. An agent allowed to say "unclear" hands you a list of things to check by hand, which is more useful than confident fiction.

If you run the agent through a search and extract pipeline, evaluate the whole stack end to end rather than judging the search layer in isolation, because the answer you care about is the product of model, search and extraction together (Parallel's guide to evaluating web search for agents). Before you build any automated scoring on top, read 20 to 50 real outputs by hand: reviewer time on real traces reveals failure patterns faster than any metric (LangChain's agent evaluation readiness checklist).

The audit is only as useful as the checklist you score it against. Keep the checklist fixed across topics so results stay comparable, and score each query pass or fail rather than on a scale. A 1 to 5 scale invites arguments about whether a gap is a 3 or a 4.

A working checklist for a topic:

1. Direct answer present. Does at least one top 3 page answer the query in the first two paragraphs, or does the reader have to scroll?
2. Page type match. Does the dominant page type match what you can credibly build? If the top 10 are all forum threads, a polished guide may not be the shape that wins.
3. Original evidence. Do ranking pages cite numbers, screenshots, tests or first-party data, or are they summarizing each other?
4. Recency signal. Is there a visible update date on the ranking pages, and does the query reward freshness?
5. Depth gap. Is there a subtopic the top 10 mentions but none covers properly? Ask the agent explicitly, then verify yourself.
6. Format gap. Is there a template, calculator, checklist or comparison table the top 10 lacks?
7. Follow-up coverage. Does the top 10 answer the question the searcher will ask next?

Then compute two numbers per topic. Coverage is the share of queries where at least one top 10 page clearly answers the query. Opportunity is the count of queries failing three or more checklist items. A topic with 18 queries, 4 covered and 5 opportunities is a different decision than one with 18 queries, 14 covered and none. Write both numbers down with the pull date, or the next audit will not be comparable.

This is where most teams fall down. CMI's survey of 1,015 B2B marketers found that 37% of B2B marketers have a documented content marketing strategy, while 62% of organizations successful at content marketing have one (Content Marketing Institute's B2B trends research). The audit checklist is the documented part of competitive research that most teams skip, and it is the part that makes the next content cycle a decision instead of a mood.

The agent is reliable for extraction and coverage. It is unreliable in four specific ways, and each has a cheap test.

Stale or partial index data. The search layer behind your agent is not the search engine your buyers use, and the two drift. A study comparing AI answers to Google rankings found that roughly 80% of URLs cited in AI responses do not rank in Google's top 100 (OutreachBloom's AI visibility audit checklist). Treat any single source of ranking data as one sample, and re-pull from a second source before you act on a gap.

Surface-level summaries. If the agent read snippets rather than page bodies, its angle field is a restatement of the meta description. The content_read column tells you which rows are affected. Open every snippet-only row for the queries that decide your next content bet.

Misread intent. Agents flatten query intent. A "how do I" query can be informational or commercial depending on the modifier, and the page features are the tell. Ahrefs data cited by CXL puts Google AI Overviews on about 21% of keywords, with informational searches driving nearly all of them, which means the agent may be auditing a summary rather than the pages beneath it (CXL's AI Overview gap analysis). Note the result features yourself for the five most important queries in the set.

False confidence. The most dangerous output is a clean table with a plausible ranking_reason on every row. Run a calibration: pick 10 rows at random, read the pages yourself, and score whether the stated reason holds up. If it is right on 8 of 10, use the table. If it is right on 5 of 10, use only the URL list and do the rest by hand. Calibrate once per agent configuration, not per topic, and recalibrate whenever the model or the search tool changes.

There is one more limit worth naming. The agent judges pages, not outcomes. It cannot tell you whether a trending query will still matter in a quarter, or whether a ranking page converts. That judgment stays with you.

Suppose you sell scheduling software and you audit the topic "team scheduling." You define 16 queries: 4 definition, 4 comparison, 4 recommendation, 4 use case. The agent returns 160 rows.

The first run shows 13 of 16 queries covered by a page that answers the query in the first two paragraphs, and 9 of 16 where the dominant page type is a comparison page. The content_read column marks 22 rows as snippet-only, all of them listicles where the snippet was long enough to look complete. You open those 22 by hand and find that 6 have an original pricing table the snippet did not reveal. The agent's angle for those six was too generic to use.

Final score: coverage 13 of 16 (81%), opportunity 4 queries failing three or more checklist items, and 2 queries where the top 10 are dominated by forum threads, meaning a polished guide is the wrong format.

The decision rule most first search results will not give you: spend the next content cycle only on queries where coverage is low, the dominant page type matches what you can credibly build, and at least one ranking page lacks original evidence. That is usually 2 to 4 queries out of 16, not 10. The rest of the list is not a to-do list. It is evidence that you should not write there yet.

If the audit has to run across several topics every month, the cost is mostly orchestration and model calls. Quanta Inference (https://quantaintelligence.ai/inference) puts text, embedding and other model types behind one OpenAI-compatible key, which keeps a multi-query audit run simple to schedule and cheap to repeat.

An audit is a snapshot. Re-run it, do not extend it. Keep the query set and the prompt identical, change nothing but the pull date, and diff the output. The diff is the signal: pages that dropped, pages that appeared, and rows where ranking_reason flipped from "exact-match title" to "unclear."

Three rules keep re-runs honest:

• Re-run after a confirmed change in how the target engine selects and ranks sources, not on a calendar alone. If the answer surface changes what it cites, your stored audit describes a world that no longer exists.
• Recalibrate your 10-row sample on every re-run that changes the agent's model or tools. A better model can shift from "unclear" to confident, and confident is not the same as correct.
• Store the raw output, including the unknown and blocked rows. The rows the agent could not read are often the pages that matter most, and deleting them hides the gap.

For teams with limited resources, the pressure to produce more content without lowering the bar is real, and repurposing what already performs is one answer to it (Multiview's guide to content repurposing). The audit tells you which assets are worth repurposing into the page types the results reward, which is cheaper than starting from zero on every query.

The workflow that survives a real calendar is small: pick the queries, run the agent, score the checklist, open a sample by hand, write down two numbers and the pull date, and repeat only when the index moves. If the manual scoring and write-up is the part that keeps slipping, Quanta (https://quantaintelligence.ai/) is a place to hand the build out: describe the audit report you need, and a specialist builds it with you.