Keeping a Support Index Fresh as Help Pages Change

You cannot re-embed "only what changed" if your chunks are arbitrary token windows. A 512-token fixed window that straddles a heading boundary will change whenever either side of the boundary changes, and its neighbors will shift too, cascading re-embeds across the page. Research on chunking for retrieval-augmented generation shows that the choice of strategy matters, with a common practical range of roughly 400 to 512 tokens with modest overlap for general corpora, and document or section-level chunking preferred where the source has clear structure an overview of chunking strategies for RAG.

Help pages have excellent structure: headings, steps, tables, FAQs. Hierarchical text segmentation work shows that splitting documents along their natural structure, rather than fixed windows, produces chunks that hold more semantic meaning and retrieve more precisely research on hierarchical text segmentation chunking. A comparative evaluation of chunking methods for clinical decision support reached the same practical conclusion: structure-aware chunks beat naive fixed windows on retrieval quality the chunking comparison study in Bioengineering.

So the rule is: one chunk per headed section, split further only if a section exceeds your token budget. Then a stable section key, derived from the page URL plus the heading path, becomes the primary key in your index. When the "Refunds" section on /help/billing is re-embedded, it overwrites the previous vector under the same key. Nothing else moves.

The job has six stages. Run it on a schedule, log every stage, and make each stage idempotent so a crash mid-run is safe to retry.

1. Fetch. Pull each help page from its canonical URL or your CMS API. Follow redirects and record the final URL; a redirected page needs its index entries remapped, not duplicated.
2. Extract and normalize. Strip navigation, footers, cookie banners, and anything templated that changes without meaning (relative timestamps, "last updated" widgets). The Stack Overflow thread on hash comparison flags exactly this trap: a page that prints today's date will hash differently every day unless you exclude it page-change hashing. Normalize whitespace, resolve relative URLs to absolute, and drop tracking parameters from inline links.
3. Segment. Split on headings into sections. Assign each a key: sha1(page_path + "#" + heading_path).
4. Hash. Hash the normalized section text, not the raw HTML.
5. Compare. Load the previous manifest of section_key -> content_hash, embedding_id, embedded_at. Any key whose hash differs, or that is missing, goes into the re-embed queue. Any key present in the manifest but absent from the page goes into the delete queue.
6. Embed and commit. Send only the queued sections to the embedding model, write new vectors with their keys and hashes, delete removed keys, and write the new manifest in the same transaction as the vector writes so the two cannot diverge.

import hashlib, json

def section_hash(text: str) -> str:
    norm = " ".join(text.split()).lower()
    return hashlib.sha256(norm.encode("utf-8")).hexdigest()

def plan_run(current: dict[str, str], previous: dict[str, dict]) -> dict:
    to_embed = [k for k, h in current.items()
                if k not in previous or previous[k]["hash"] != h]
    to_delete = [k for k in previous if k not in current]
    unchanged = len(current) - len(to_embed)
    return {"embed": to_embed, "delete": to_delete, "reused": unchanged}

Run this against a manifest of 8,000 sections and a typical daily edit pattern of 40 changed sections, and the job embeds 40 vectors instead of 8,000. That is a 99.5% reduction in embedding calls on a quiet day, which is why incremental jobs are worth the engineering. When you need to route those calls, one API key for text and embedding models removes a vendor-specific client from the loop; that is what Quanta Inference is for Quanta Inference.

The most useful single number is stale share: the percentage of indexed sections whose stored hash no longer matches the live page at check time.

stale_share = stale_sections / total_indexed_sections * 100

Compute it at the end of every run, before the commit. A run that repairs everything will report zero; the number that matters is the reading taken immediately after the previous run's commit, because that shows how much drift accumulated in one interval. Track it as a time series and watch the shape, not the absolute value.

Three companion metrics fill out the picture. Content age is how long since a section's source was last changed. Embedding lag is the delay between a page edit and the new vector landing in the index; on a daily job that ceiling is your run interval. Stale retrieval rate is the fraction of live queries that return a section whose stored hash no longer matches the live page. A data governance write-up on knowledge base freshness scoring proposes the same four dimensions and recommends a composite freshness score with an automated alert threshold Atlan on LLM knowledge base freshness scoring.

A worked example makes the thresholds concrete. Suppose the index holds 8,000 sections. Daily runs find 30 to 60 changed sections, a stale share of 0.4% to 0.75% at measurement time. Set the alert at 2%, roughly 160 sections. That tolerates a busy content day but catches a broken pipeline: if the job fails silently for four days, stale share climbs past 2% and pages someone. Set a second, harder threshold at 10% for paging out of hours.

The sharp-jump case deserves its own alert. A stale share that goes from 0.5% to 15% overnight almost never means editors got busy; it usually means a template change altered the normalized text of every section on every page, or a URL migration changed the page paths. Both need a human before you re-embed 8,000 sections and possibly duplicate them under new keys.

Incremental hashing catches content edits. It does not catch four other failure classes:

• Chunker changes. You altered how sections are split, so every key is now wrong.
• Embedding model changes. Vectors from two models are not comparable; you must re-embed everything so distances are meaningful within one space.
• Normalization changes. A new rule in your cleanup step changes every hash at once.
• Silent retrieval failure. The section is present, its hash matches, but the vector no longer surfaces for queries that should hit it.

The first three are deterministic: version your chunker config, normalization rules, and embedding model in the manifest, and if any version differs from the stored value, run a full re-index automatically. No judgment call needed.

The fourth needs a counter. Define a retrieval miss as a query where the expected section does not appear in the top results. Log misses per query string. When the same query or the same section misses on three consecutive scheduled runs, promote it to a full re-index trigger. The reason for a counter rather than a single miss is noise: one bad phrasing of a query should not cost you a re-embed of the whole corpus, but a query that fails three runs in a row is a signal the index is structurally wrong, not unlucky.

Full re-indexes are rare if the incremental job is healthy. Budget for them: they are the only time you should expect embedding spend to spike, so cap them to an off-peak window and keep the previous index readable until the new one passes your evaluation set.

Two things break when a section is re-embedded under a new key. First, any citation your support bot showed a customer ("see the refund policy, section 2") now points at a key that may not exist. Second, your internal links between help pages, the kind that help both crawlers and readers find related content, may point at anchors that were renamed Yoast on internal linking. Google's own guidance on crawlable links is worth applying to your help center too: if an anchor is generated by script or has no real href, you cannot rely on either a crawler or your own fetcher resolving it Google's link best practices.

The fix is to separate identity from content. Store the section key (stable, derived from URL and heading path) separately from the content hash (volatile). Citations reference the key plus a character offset within the stored section text. When a section is re-embedded, you update the hash, the text, and the offsets, but the key survives, so the citation resolves to the new text. If a heading is renamed, the key changes; write a redirect row mapping the old key to the new one and resolve citations through it rather than breaking them. Fine-grained citation work in production RAG makes the same point: metadata that anchors a chunk to its source location has to be preserved at index time, and the storage overhead is small relative to the value citation-aware RAG pipelines.

Once the daily job is stable, the interesting questions shift from "is the index fresh" to "is the index right."

Watch the re-embed volume per page. A page that re-embeds every day is either genuinely volatile (a live status page) or flapping because your normalizer is not stripping something. Sort pages by weekly re-embed count and inspect the top ten.

Watch the miss counters by section, not just by query. A section that misses repeatedly is a chunking problem, not a freshness problem: it may be embedded but too small, or buried in the middle of a long chunk. The chunking literature is clear that semantic cohesion within a chunk drives retrieval quality more than chunk size alone chunking strategies for RAG.

Watch the citation resolution rate. Sample the citations your bot produced yesterday and check that each still resolves to a live section with a matching key. Dip below your tolerance and you have a key-stability bug, not a freshness bug.

Keep an evaluation set that includes stale-prone queries. The staleness gap write-up makes the sharpest point in the whole area: standard RAG evaluation measures whether an answer is faithful to what was retrieved, not whether what was retrieved is current, so a system can score well while serving instructions for a process that no longer exists the staleness gap analysis. Build your golden set from queries whose answers changed at least once, and re-run it after every full re-index.

If you would rather have the pipeline built and operated for you than assembled from a scheduler and a manifest, a Quanta specialist can scope it with you Quanta.