How to Build a Support Knowledge Base That Agents Can Search

A heading is not just navigation for humans; it becomes part of the chunk text and often part of the chunk's title. "Troubleshooting" tells a retrieval system nothing. "Why a renewal payment fails when the card is still valid" carries the query vocabulary itself.

Three rules make headings do retrieval work:

1. State the problem in the user's words, not the internal feature name.
2. Avoid headings that are pure labels ("Overview", "Details", "More") unless nothing better fits.
3. Repeat key terms naturally, even if it feels redundant inside a category where every article is about the same product.

Keep the hierarchy shallow. General help center guidance recommends two levels of nesting and a top-level category count in the single digits: most knowledge bases work best with five to eight top-level categories, and every extra layer is another place a reader can get lost (HelpDocs on knowledge base structure). The same logic applies inside an article. If a section needs three sub-levels, it should probably be its own article.

For readers scanning a results list, a descriptive heading is the preview. For a retrieval system, it is a term-rich anchor. Both benefit from the same rewrite.

The first paragraph under the title should state the answer in plain language, in two to four sentences, without requiring any setup. This does two jobs.

For a human agent under time pressure, it is the answer. Kayako's knowledge base guidance anchors on exactly this: customers arrive mid-problem and frustrated, so state the solution up front, then give the step-by-step detail for those who want it (knowledge base practices). Salesforce's article-writing guide makes the same point about leading with the answer before the detail (Salesforce knowledge base guide).

For a machine, the summary is the highest-signal chunk in the article. It usually sits directly under the heading, contains the entity names and the symptom words, and reads in the same register as a customer question. If your retrieval returns a list of snippets, the summary paragraph is what renders as the preview, and it is what determines whether the agent clicks.

Write it as prose, not as a bullet fragment. A summary that reads "Renewal declines: card expired, fraud hold, insufficient funds" is navigable but stylistically unlike a query. "A renewal payment can decline even when the card is valid. The three common causes are a fraud hold, a currency mismatch, and a card network block on recurring charges" reads close to the question a customer actually typed.

The single biggest cause of missed retrieval is vocabulary mismatch: the customer says "cancelled," the article says "terminated," or the customer types the name of the old feature while the article uses the new one. Retrieval depends on lexical and semantic overlap, and semantic models close some of that gap but not reliably for product-specific jargon.

The fix is to write the alternate phrasings into the body text rather than hiding them in a tag. A line like "Customers sometimes describe this as a chargeback, a dispute, or a payment reversal; in this product all three refer to the same workflow" costs a sentence and gives every one of those terms a home in the chunk. This is how prompts and context instructions work for generative models as well: the FedPayments Improvement blog describes grounding a model with clear context and industry-standard classification definitions so its output stays aligned with approved practice, rather than drifting toward irrelevant detail (guiding LLMs toward better classification).

A useful habit is to maintain a synonym block per article, either as a visible "Also known as" line or as a metadata field. Do not let it read as keyword stuffing; a natural sentence that mentions the common alternates is enough.

Keep the same table for terminology across the whole base. When one team writes "workspace" and another writes "organization" for the same object, a query using one word misses the article using the other.

Metadata is the difference between "here are forty results" and "here are three results from the right product for the right audience." A small, disciplined set of fields is enough:

Retrieval providers expose this directly. AWS Bedrock knowledge bases support filtered retrieval over metadata attached to source chunks, and describe reranking options for ordering the returned chunks by relevance (querying a knowledge base). The pattern is consistent across vendors: filter first, then rank, then optionally rerank.

The payoff is that a query such as "admin cannot invite user to payroll" gets filtered to product Payroll and audience account admin before any similarity scoring happens. The right paragraph is now competing against a handful of candidates instead of the whole corpus.

Metadata also governs freshness. The Kapa.ai guidance on keeping a retrieval index in sync recommends per-source refresh schedules rather than one clock for everything, with different intervals for tickets, wikis, code and web crawls (keeping a RAG knowledge base in sync). A last-reviewed field lets you apply the same idea inside the knowledge base: articles tagged to a fast-moving product get reviewed more often than policy pages.

You cannot improve what you cannot measure, and the most common measurement mistake is blending retrieval and answer quality into one score. Evaluate them separately. Retrieval evaluation uses ranking metrics such as recall at k with ground truth, or judged relevance of the retrieved context; generation evaluation checks whether the final answer is grounded in what was retrieved (RAG evaluation guide).

A practical test loop for a support knowledge base:

1. Collect 100 real queries from search logs and ticket subjects.
2. For each, record the article and section that should have been returned.
3. Run the queries against the index and record whether the correct section appeared in the top five.
4. Fix failures by rewriting the heading, adding the missing synonym, or adding metadata, not by reindexing.
5. Re-run after each batch of edits and watch the top-five hit rate move.

Content fixes and index fixes fail in different ways, and separating the measurement tells you which one to spend effort on. If a query misses because the article never uses the customer's word, no amount of reindexing helps. If it misses because the chunk spans three topics, restructuring helps.

Clinical triage research shows the same principle at scale: a retrieval-augmented system built around a structured case database outperformed a prompt-only model on multi-level triage classification, with the authors attributing gains to multi-evidence retrieval over case records alongside the model (multi-evidence clinical reasoning study). Structured source content is the variable, not model size.

Every knowledge team eventually builds a refresh pipeline: change detection per source, incremental delta updates, validation before publish, and monitoring. Kapa.ai's breakdown of that work is sobering: incremental updates keep a base current far more cheaply than full re-indexing, and "embedding rot" sets in when content or models age out, with a suggested re-embed once roughly 10 to 15 percent of the corpus changes (sync guidance).

Staleness has its own cost. Atlan's analysis reports that stale embeddings can degrade retrieval accuracy by up to 20 percent with no uncertainty signal reaching the user, and argues that faster re-indexing just propagates bad information more efficiently when the source itself is ungoverned (LLM knowledge base staleness). That is the argument for content design as a freshness strategy: a base whose chunks are small, headed and metadata-tagged tolerates slower, cheaper refresh cycles because each changed section maps to a small, identifiable delta.

The decision rule is straightforward. If a query fails because the answer is not in the corpus, fix the corpus. If it fails because the corpus is out of date, fix the refresh cadence. If it fails because the answer is buried in a 1,500-word wall with vague headings, fix the article, and you will not need to shorten your reindex interval to compensate.

A quick checklist before publishing

• One topic per article, with a title that states the problem.
• A two-to-four sentence summary at the top that answers the query in plain language.
• Four to eight descriptive headings per 1,000 words, each section self-contained.
• Alternate phrasings for the terms customers actually type, written into the body.
• Product, version, audience and intent metadata filled in, plus an owner and a review date.
• Consistent terminology with the rest of the base.

Turning the checklist into a pipeline

Once the template is agreed, enforcing it is a parsing job, not a writing job. A small script can flag articles that break the rules before they reach the index:

import re

MAX_WORDS, MIN_HEADS, MAX_HEADS = 1200, 3, 8

def lint_article(md: str, meta: dict) -> list[str]:
    problems = []
    body = md.split("\n", 1)[-1]                 # drop the H1
    words = len(body.split())
    heads = re.findall(r"^##\s+(.+)$", body, re.M)
    if words > MAX_WORDS:
        problems.append(f"{words} words, split the article")
    if not MIN_HEADS <= len(heads) <= MAX_HEADS:
        problems.append(f"{len(heads)} headings, target {MIN_HEADS}-{MAX_HEADS}")
    for h in heads:
        if len(h.split()) < 3:
            problems.append(f"vague heading: {h}")
    for field in ("product", "version", "audience", "intent", "owner"):
        if not meta.get(field):
            problems.append(f"missing metadata: {field}")
    return problems

Run it in CI on every pull request to the docs repo. Failures are cheap to fix at authoring time and expensive to fix after six months of drift.

When you are building the search front end or an agent that queries the base, the model choice is the easy part to vary. Quanta Inference exposes text, image, video, audio and embedding models through one OpenAI-compatible API key with prepaid US-dollar pricing, which lets you swap an embedding model without rewriting the pipeline (Quanta Inference).