Using an AI Agent to Classify Search Queries by Intent

The prompt does three jobs: it carries the rubric, it fixes the output shape, and it tells the model how to behave when the rubric does not decide. Keep the rubric in the system message and the queries in the user message so the expensive part is cached or reused across batches.

System:
You are a search intent classifier. Apply the rubric below exactly.
Return only JSON matching the schema. Do not add commentary.

RUBRIC
informational: searcher wants to learn. Signals: how, what, why, guide, meaning, difference.
commercial: searcher is comparing before deciding. Signals: best, vs, review, alternatives, pricing.
transactional: searcher is ready to act. Signals: buy, order, sign up, book, coupon, near me.
navigational: searcher wants a specific site. Signals: brand name, login, official site.
mixed: two intents carry equal weight and the rubric does not separate them.

EXAMPLES
"what is a crawl budget" -> informational
"best crm for small teams" -> commercial
"buy running shoes size 10" -> transactional
"facebook login" -> navigational
"crm pricing comparison and how to implement" -> mixed

RULES
1. Judge intent from the query text only. Do not guess the searcher's history.
2. If two buckets fit and neither clearly wins, use "mixed" and set confidence below 0.6.
3. Confidence is 0 to 1. Use 0.9 or higher only when a rubric example is nearly identical.
4. reason is 12 words maximum.

User:
Classify each query. Return JSON: {"results":[{"id":1,"query":"...","intent":"...","confidence":0.0,"reason":"..."}]}
Queries:
1. ...

Four details make this prompt hold up in production:

1. Constrain the labels. Without an explicit list, models drift into synonyms ("research", "buying") that break your downstream joins.
2. Put the schema in the prompt. Describing the JSON keys, including id, lets you match results back to rows when the model reorders anything.
3. Demand a reason. A 12-word justification makes human review fast and surfaces rubric gaps: if reviewers keep reading the same reason, the rubric is missing a rule.
4. Define the confidence scale. Models default to high numbers unless you anchor them with words like "use 0.9 or higher only when a rubric example is nearly identical".

If your stack supports structured output or a JSON mode on the model, use it on top of the prompt rather than instead of it. The prompt explains the judgment; the schema enforces the shape.

Sending one call per query means paying for the rubric hundreds of times. Send batches of 20 to 50 queries per call and the rubric cost is amortized across every query in the batch Lucidworks. The trade-off is granularity: a malformed response wastes the whole batch, so add one retry that splits a failed batch in half.

A simple batching loop in Python:

import json

def build_batch(rows, start, size):
    chunk = rows[start:start + size]
    lines = "\n".join(f'{i+1}. {r["query"]}' for i, r in enumerate(chunk))
    return chunk, f"Classify each query. Queries:\n{lines}"

def classify(rows, call_model, size=40, max_retries=2):
    out = []
    for start in range(0, len(rows), size):
        chunk, prompt = build_batch(rows, start, size)
        for attempt in range(max_retries + 1):
            try:
                payload = json.loads(call_model(prompt))
                results = payload["results"]
                if len(results) != len(chunk):
                    raise ValueError("length mismatch")
                out.extend(results)
                break
            except (json.JSONDecodeError, KeyError, ValueError):
                if attempt == max_retries:
                    out.extend({"query": c["query"], "intent": None,
                                "confidence": 0.0, "reason": "parse_failed"}
                               for c in chunk)
                size = max(10, size // 2)  # shrink and retry
    return out

Two operating rules keep batching cheap and honest. First, log the raw request and response for every batch, so a wrong label can be traced to a prompt version rather than argued about. Second, do not mix languages or very different query types (branded commerce terms and long informational questions) in the same batch at low batch sizes, because a single ambiguous example tends to pull neighboring labels with it.

Before scaling, run 100 to 200 queries twice with the same prompt and measure agreement with your hand labels. If agreement on the transactional bucket is weak, add near-miss examples there and rerun; rerunning 200 rows is far cheaper than auditing 20,000.

A confidence threshold is a cutoff that decides which predictions are accepted automatically and which are flagged for a person LlamaIndex. Set the threshold from your data rather than accepting a default: LlamaIndex notes that accepting the default 0.5 in a binary classifier is one of the most common production mistakes, and that different fields or use cases need different cutoffs LlamaIndex.

A worked example. Suppose 5,000 queries, with the model labeling 62 percent informational and setting 0.9 or higher on 80 percent of rows:

That is about 1,320 reviews, roughly a quarter of the set, before you decide whether to lower the acceptance bar. Excluding the sample, you accept the high-confidence rows on the strength of a 5 percent audit. If the audit shows more than a small number of errors on a bucket you care about (transactional is usually the one), stop accepting that bucket automatically and review it in full until the rubric improves.

Useful operational details:

• Review transactional and commercial first. Transactional intent is the most valuable for marketers because it captures demand, and wrong labels send buyers to blog posts instead of pricing pages Siteimprove.
• Sample randomly, not by convenience. Reviewers who pick "interesting" rows will over-report errors and send you chasing edge cases.
• Feed corrections back into the rubric, not just the output. Every mislabel should produce either a new example row or a new rule. If a correction cannot be expressed as a rule, the label set is missing a bucket.
• Audit the audit. Have a second person review a slice of the reviewer's decisions on the first pass. Inter-rater disagreements tell you which definitions are still vague.

There is a documented failure mode here: teams hand downstream writers or page-builders a single word like "informational" with no further context, then wonder why the resulting pages do not rank Grow and Convert. Store the reason string alongside the label so the intent survives into the brief.

Hand-label 200 rows yourself, blind to the model output, and score the model against them. Report two numbers: overall agreement, and agreement per bucket. A pipeline that is right on aggregate but wrong on transactional is worse than useless, because the errors land on your highest-value queries.

Three checks catch most problems:

1. Confusion between commercial and informational. This is the hardest boundary in practice. "Best project management software" and "what is a project management tool" are both about the same topic but sit in different buckets Siteimprove. If your confusion concentrates here, add paired examples to the rubric.
2. Transaction leaking into navigational. Queries with a brand name plus "pricing" are usually commercial, not navigational, since the searcher is still deciding rather than navigating to a page they already know SE Ranking.
3. Confidence that does not move. If nearly every row comes back above 0.9, the confidence score carries no signal. Tighten the anchor language and test again.

Finally, treat the label as an input to a decision, not the decision. The point of classifying queries is to map each bucket to a page type: guides for informational, comparisons for commercial, product and pricing pages for transactional, brand pages for navigational Siteimprove. Reviewing a handful of labels against a page inventory is a more useful acceptance test than a single accuracy number.

If you would rather concentrate on the rubric and the review queue than on batching retries and provider keys, Quanta Inference puts text, image, video, audio and embedding models behind one OpenAI-compatible API key with prepaid US dollar pricing (https://quantaintelligence.ai/inference). That keeps the classifier and any later embedding or relevance work on one integration.

Informational, commercial and transactional will cover most of a commercial keyword list, but two additions are worth planning for. Local intent expects map results, accurate business details and reviews, so a local label should route to location pages rather than articles SE Ranking. The newer generative intent, where people ask an assistant to create or calculate something, is a distinct pattern that behaves more like a task than like a search SE Ranking.

Crawling and indexing mechanics are worth a short mention because they explain why intent classification rarely maps one to one with rankings. Google crawls, indexes and then serves pages in three stages, and not every page survives all three Google Search Central. Technical problems such as noindex directives, redirects and blocked resources can keep a well-matched page out of the index entirely Keywords Everywhere. So when a page built for a transactional query does not perform, check crawlability and indexing before you blame the label.

A quick sanity check on any URL:

# Check index status
site:example.com/pricing

# Check directives
# view-source:https://example.com/pricing and look for
# <meta name="robots" content="noindex">

Keep the taxonomy small enough that your team can hold it in their heads. Four buckets plus mixed, with local or generative added only if your data justifies it, is enough to route hundreds or thousands of queries into page briefs.