What a Referral Drafting Pipeline Costs in Model Tokens

Here is a concrete plan you can copy into a spreadsheet. Every rate below is illustrative arithmetic from published list prices, not a quote. Verify current rates on the provider page before you commit.

Assumptions:

• 1,000 referral asks per month.
• A referral corpus of 2,000 documents (product notes, past successful referral emails, reward rules, customer profiles), averaging 500 tokens each: 1,000,000 tokens.
• The corpus is rebuilt fully twice a month (fresh customer data), and only changed documents are re-embedded on other days.
• Each ask retrieves 8 passages of 300 tokens each: 2,400 tokens of context.
• The prompt wrapper (instructions, requester profile, output schema) is 600 tokens.
• The draft output averages 350 tokens, capped at 500.
• One retry happens on about 10 percent of asks.

Embedding the corpus. At $0.02 per million input tokens for the small embedding model (CostGoat), one full build of 1,000,000 tokens is $0.02. Two rebuilds a month plus change-driven updates land well under a dime. Run the initial build through a batch tier and it halves (CostGoat). Embedding is essentially free at this scale, which is why caching it is the first thing to get right and the last thing to worry about.

Query embedding. 1,000 queries at roughly 30 tokens each is 30,000 tokens a month, or about $0.0006. Round it away.

Generation. Using $2.00 per million input and $10.00 per million output as the rate pair in this example (OpenAI API pricing):

• Input per ask: 600 wrapper + 2,400 retrieved = 3,000 tokens.
• Output per ask: 350 tokens.
• Input cost per ask: 3,000 / 1,000,000 x $2.00 = $0.006.
• Output cost per ask: 350 / 1,000,000 x $10.00 = $0.0035.
• Total per successful ask: $0.0095.
• With a 10 percent retry rate, effective cost per ask: about $0.0105.

1,000 asks a month therefore lands near $10.50 in generation, plus a few cents of embedding. That is the low-tens-of-dollars figure. Now the sensitivity test that actually matters:

The lesson: doubling retrieved passages costs you more than a 30 percent retry rate. Retrieval depth, not model selection, is the first lever.

A token bill is not one price times visible words. It is the sum of uncached input, cache writes, cache reads, output, and any separate tool charges, across every billed attempt (BenchLM). Build the estimate as a ledger:

• Uncached input: wrapper, retrieved passages, few-shot examples, schema.
• Cache writes: the reusable prefix on the request that first introduces it.
• Cache reads: the same prefix on later requests.
• Output: the draft, plus hidden reasoning tokens if you use a reasoning mode, which providers may bill as output (BenchLM).
• Retries: every retried call is another billable attempt unless the provider says otherwise (BenchLM).

Two practical traps. First, unit errors: an off-by-ten in query volume turns a $3 estimate into a $30 one, and the arithmetic looks fine either way (BenchLM). Second, assuming the cache applies when it does not. A cached rate only helps if the prompt prefix is identical, long enough to qualify, unexpired, and scoped the way the provider requires (OpenAI prompt caching).

The decision rule: if your estimate and your invoice differ by more than 20 percent, do not adjust the estimate. Find the bucket where the gap lives. Nine times in ten it is retries or a cache that never hit.

There are three cache layers in a referral pipeline, and they save different things.

Layer 1: corpus embeddings. Store vectors keyed by document hash. Re-embed only when the document or the embedding model changes. This makes the embedding line effectively a one-time cost and is why embedding drops out of the monthly conversation after month one (CostGoat).

Layer 2: prompt caching. Most asks share a long stable prefix: system instructions, tone rules, the reward policy, the output schema, and a few exemplar drafts. Put all of that at the front, unchanged, and let per-ask material follow. Caching reuses work when requests share the same prompt prefix and cuts the cost of recomputing it (OpenAI prompt caching). The failure mode is a builder who injects the requester's name, timestamp or a random ID into the system prompt, which invalidates the prefix on every call and silently bills the full input rate.

Layer 3: draft cache. If two asks are near-identical (same referrer, same target company, same product), retrieve the previous draft and have the model adapt it instead of generating from scratch. That cuts output tokens, which is the most expensive bucket in this workload.

A worked comparison for layer 2, using the same rate sheet: if the stable prefix is 1,500 tokens and cached input is billed at a fraction of the uncached rate, those 1,500 tokens cost a tenth of their uncached price on every hit (the cached input column on OpenAI's pricing page shows the multiplier for each model). Over 1,000 asks, moving 1,500 tokens per call from uncached to cached input at a 10x discount saves roughly $2.70 a month at these rates. Modest, but it also cuts latency, which is where users actually notice it. The true prize is correctness: a stable prefix is easier to evaluate and version than one assembled ad hoc per request.

A budget is not a policy until the code enforces it. A reliable pattern is a per-ask cap with three knobs, checked before the call and after the response.

• Max retrieved passages: hard cap on the number of chunks, sorted by similarity, with a lower cap for "simple" asks detected by a cheap classifier or by the presence of a known target.
• Max context tokens: count the assembled prompt with the provider's tokenizer before sending, then drop the lowest-ranked passages until it fits.
• Max output tokens: set the completion cap per ask type, not globally.

CAPS = {
    "passages": 8,
    "context_tokens": 3000,
    "output_tokens": 500,
    "usd_per_ask": 0.02,  # your ceiling, not a list price
}

def assemble(passages, wrapper):
    keep = []
    used = count_tokens(wrapper)
    for p in sorted(passages, key=lambda x: -x.score):
        t = count_tokens(p.text)
        if used + t > CAPS["context_tokens"]:
            break
        keep.append(p)
        used += t
        if len(keep) >= CAPS["passages"]:
            break
    return keep, used

def billed_cost(input_tokens, output_tokens, in_rate, out_rate):
    return (input_tokens / 1e6) * in_rate + (output_tokens / 1e6) * out_rate

Then log every call with its token counts and computed cost, and alert when the rolling daily average per ask crosses the cap. The cap should be a few cents, not a fraction of a cent, because a draft that misses the reward terms costs far more in support time than the tokens saved. Set the ceiling from your economics: if a referred customer is worth enough that a few cents is noise, you can afford generous retrieval. If referral volume is high and conversion is thin, argue over the cap explicitly with a number rather than letting the retry rate argue for you.

The generation model is where the visible quality lives, so start there and work backward. A workable decision rule:

• Drafting quality first. Test two or three candidate models on 50 real asks with a human rating: does the draft name the right target, state the reward correctly, and read like the referrer wrote it?
• Then price the token mix. Your mix is input-heavy (about 8.5:1 in the baseline above), so input rates dominate more than a quick skim of headline output prices suggests. The rate table from OpenAI's pricing page makes this visible: cached input is priced at a small fraction of uncached, and output at several times input.
• Then batch and off-peak. If drafts are queued and reviewed hourly rather than sent instantly, batch or flex tiers can cut the price substantially (OpenAI API pricing). Off-peak scheduling is another lever worth pricing explicitly (BenchLM).
• Keep a fallback. A provider outage mid-month is a business problem; a routing layer with a second model lets you fail over at a known cost premium.

This is also the point where a single gateway pays off. Quanta Inference puts text, image, video, audio and embedding models behind one OpenAI-compatible API key, priced in US dollars and prepaid, so you can swap the generation model without rewriting the client, and keep the embedding calls on the same key (Quanta Inference). For a pipeline whose cost is spread across two model families, one ledger is worth more than a clever discount.

• Version the corpus. Embedding is cheap; a stale vector index is expensive because every draft then reads wrong facts. Re-embed on document hash change, not on a calendar.
• Instrument the four buckets. Log input, cached input, output and retry flag on every call, then reconcile against the invoice a week later (BenchLM).
• Evaluate, do not vibe-check. Ground answers in retrieved sources, keep citations available for reviewers, and run a small judge or rubric over a sample each month (Evidently AI).
• Write prompts that constrain. Clear role, explicit constraints, an output schema and an instruction to state when context is missing are exactly the practices prompt-engineering guides recommend, and they reduce retries, which is a cost control, not just a quality control (Clio, Champlain College).
• Track a cost per draft, not a cost per month. Monthly totals hide mix changes. Cost per ask tells you immediately when retrieval depth or retries drift.
• Review the human loop. Referral asks that read as transactional perform worse than ones that read as favors (Referral Rock, Indeed), so sample drafts for tone, not just facts. A cheap draft that nobody sends is the most expensive outcome available.