When the Fact Sheet Becomes Retrieved Context: A Sizing Rule

Two ratios answer the question. You do not need a benchmark harness to compute either.

Ratio one: fact-sheet tokens divided by usable context window. Usable window means the model's total window minus what you reserve for conversation history, the current question, retrieved chunks, and the output. If you reserve half the window for dialogue and answers, a fact sheet at 8,000 tokens against a 128,000-token window is 6 percent of the usable 64,000. That is fine inline. At 30,000 tokens it is 47 percent, and every turn is carrying a third of a novella the question may not touch.

Ratio two: sections touched per session divided by total sections. Instrument your logs. Count how many distinct sections of the fact sheet appear in the retrieved evidence or the answers across a week of real traffic. If a 40-section sheet has 11 sections that anything ever references, you are carrying 29 sections of dead weight on every request.

The decision rule falls out of those two numbers:

The thresholds are engineering heuristics, not laws. The point is to have a rule you can defend in a design review rather than an argument about whether the prompt "feels long."

The mistake people make when they split a fact sheet is moving all of it. Some of a fact sheet is not reference material at all; it is behavior. That part must stay in context permanently, because retrieval is probabilistic and behavior cannot be.

A rules block holds:

• Formats. Output shapes, field names, units, date formats, currency conventions.
• Thresholds and arithmetic rules. "Discount applies only above the tier boundary," "eligibility requires two consecutive qualifying months."
• Definitions. Terms that appear in retrieved chunks and would be ambiguous without a fixed meaning.
• Refusal conditions. When to say "not in the provided material" instead of guessing.
• Pointers. A list of the section titles that exist in the index, so the model knows what it could ask for and can tell the user when a question falls outside the sheet.

Constraint instructions of this kind are what most teams call guardrails, and the enterprise literature treats them as the control point between human intent and machine interpretation, enforcing policy and limiting how the model may respond per the Cloud Security Alliance's prompt guardrail guide. Writer's guardrail writeup makes the same case for output-level controls: safety, factual accuracy, and brand fit are properties you specify, not properties you hope emerge in Writer's guardrails overview. The refusal condition matters most. SUSE's hallucination guidance recommends constraining output and prompting for verification as countermeasures, and a plain "answer only from the rules block and the retrieved sections, otherwise say you do not have it" is the cheapest version of both per SUSE's prompt guidance.

A rules block should be short. Hundreds of tokens, not thousands. If it grows past that, some of what you put in it is really reference material wearing a rule's clothing, and it belongs in the index.

Take a 4,800-token fact sheet for a business that sells tiered service plans. Suppose your usable window after reserving dialogue and output is 40,000 tokens. The sheet is 12 percent of it. That is inside the borderline band, and the usage data decides.

Suppose the logs show that of the sheet's 32 sections, nine come up in real questions: pricing tiers, overage rates, contract term, cancellation window, data residency, support response times, onboarding steps, the discount policy, and the definition of "active seat." The other 23 sections cover integrations, historical notes, and legal boilerplate.

The split:

Rules block (stays inline, about 400 tokens). Definition of active seat, the tier boundary arithmetic, the currency and date formats, the discount threshold, the cancellation-window rule, the refusal condition, and a list of the nine section titles.

Indexed sections (about 4,300 tokens total). The nine active sections, chunked one per question answered, plus a decision already made: the 23 dormant sections go into the index too, but as separate chunks that simply never rank. They cost nothing at query time and cost a few cents to embed once.

Here is the shape of the assembled prompt:

[system]
You are a plan advisor. Use only the RULES and RETRIEVED SECTIONS below.

RULES
- active seat = a seat with at least one login in the trailing 30 days
- tier boundary: overage applies only to seats above the contracted tier
- discount applies only when contract term >= 12 months
- cancellation window: 30 days before renewal
- formats: USD, ISO dates, no thousands separators
- if the answer is not in RULES or RETRIEVED SECTIONS, say so and stop

RETRIEVED SECTIONS
<<chunk: pricing-tier-overage>>
...

[user]
We have 14 seats contracted, 18 active last month, annual term. What do we pay?

The arithmetic that makes this worth doing: 4,800 tokens inline became roughly 400 fixed plus three or four retrieved chunks at 150 to 400 tokens each. Call it 1,400 tokens on a typical question. That is a 70 percent cut in the reference payload, and the rules that govern the answer did not move.

Chunking by question answered is the structural choice that makes those chunks rank well. Mixed-subject chunks return weaker similarity scores than separated ones, and concise chunks improve both performance and accuracy, which is why semantic and paragraph-based strategies beat fixed-length splits for prose documents per Meilisearch's chunking survey. A useful test for a chunk: can you read it alone and still answer one specific question? If not, split or merge it.

The practical procedure, in order:

1. Write the questions first. For each section of the fact sheet, write the one question a user would ask that lands there. If you cannot, the section is dormant; index it anyway, but do not tune for it.
2. Make each chunk self-contained. Include the section title inside the chunk text so a retrieved fragment carries its own label.
3. Allow overlap but keep it small. Trailing text from the previous chunk helps when a rule spans a boundary. Overlap is a real strategy with a cost, so keep it to a sentence or two rather than a percentage of the chunk per Meilisearch's survey.
4. Keep numbers attached to their conditions. A chunk that says "the overage rate is 4 per seat" without the tier boundary is a hallucination waiting to happen. Conditions travel with the number.
5. Store the rules block in the same repository as the chunks, versioned together. The failure mode to fear is a chunk describing an old threshold while the rules block describes the new one.

For teams building the retrieval side for the first time, Daniel Bourke's from-scratch walkthrough of a local RAG pipeline (load, chunk, embed, retrieve, augment, generate) is a useful mental model of where each step can break in Bourke's RAG tutorial. Aaron Tay's caution is the complement: prompt-engineering tricks tuned on plain chat models do not automatically transfer to RAG systems, because the retrieved context changes what the model is actually doing per Tay's RAG prompting notes. The lesson for fact sheets is to test the split against real questions, not to add more instructions.

Run this on a copy of your prompt and a week of real logs. It takes an afternoon.

1. Count. Measure fact-sheet tokens with your tokenizer, not by dividing characters.
2. Measure the window. Subtract the maximum conversation history you allow, the output cap, and the retrieval budget you would need. That is your usable window.
3. Compute the share. If the fact sheet is under 10 percent and used broadly, stop. Keep it inline.
4. Tag usage. For each answer in the logs, record which fact-sheet sections the answer required. Aggregate over a week.
5. Split if either threshold trips (share above 15 percent, or usage under a third of sections), or if the sheet's change rate is high. A sheet that changes weekly and lives in the system prompt forces a prompt redeploy on every edit; as chunks, only the affected chunks are re-embedded.
6. Re-measure. After the split, confirm that the average reference payload per request dropped and the share of correctly grounded answers did not. Precision and recall are the standard measures for this, and both are computable from labeled sample questions per Meilisearch's evaluation notes.

Two secondary signals push the decision earlier. First, latency and cost: if your bill is dominated by input tokens and most requests touch a small corner of the sheet, the split pays for itself in the first week. Second, agent loops: an agent that re-reads its context several times per task pays the fact-sheet cost on each pass, which is why the same rule that makes a chat prompt acceptable can make an agent budget unacceptable.

If you would rather not run the embedding and retrieval plumbing yourself while prototyping, Quanta Inference exposes text, embedding, and other model types through one OpenAI-compatible API key, so you can test both the inline and the retrieved version of the same sheet against the same traffic: https://quantaintelligence.ai/inference.

A few failure patterns recur.

Splitting a rules sheet. If the fact sheet is mostly thresholds and refusal conditions, it is already a rules block and should stay inline. The split only applies to reference material.

Chunking that orphans the condition. As above: numbers without their conditions, definitions separated from the terms they define, exception clauses cut away from the rule they modify.

Assuming the index knows what the prompt knows. If a retrieved chunk contradicts the rules block, the model has to choose. Version them together and add a line to the rules block naming it as authoritative.

Skipping the eval set. Build twenty to fifty real questions with known correct answers before the split, run them after, and compare. Without that, you cannot tell whether a drop in quality came from retrieval missing a chunk or from the model ignoring one it received.

Treating the split as permanent. Sheets grow. The nine active sections from the worked example become fourteen. Re-run the usage count once a quarter and let the dormant sections stay dormant; the threshold does not change, only which sections are on the active list.

The underlying discipline is the same one worth applying to prompt-level safety controls in general: specify the behavior you require, test it against real inputs, and keep the reference data separate from the instructions per Writer's guardrails overview. A fact sheet in a system prompt is a decision about your context budget made once and never revisited. Making it twice a year, with two numbers in hand, is the whole technique.