What Source Pages Cost to Fetch and Read

Count every HTTP request your pipeline issues, not every URL you intended to visit. That includes redirect hops, pagination, sitemap reads, retry attempts and requests that return an error. A crawl loop that follows every visible link will happily wander into faceted navigation, calendars and session ID duplicates, which is why crawl controls exist in the first place: allowed domains, path rules, URL normalization, depth and stopping conditions (Webscraper.io).

The practical counter is simple. Record three numbers per page attempt: requests issued, requests that returned a usable response, and retries. Then:

requests_per_thousand_fields =
  (requests_issued_with_retries / fields_extracted) * 1000

A pipeline that fetches 20,000 pages, issues 27,000 requests including retries and redirects, and extracts 250,000 fields spends about 108 requests per thousand fields. That is the number to put in the budget, not the page count.

Retries are the quiet multiplier. A site that rate-limits aggressively can turn one page into three or four requests, and each retry usually also re-triggers rendering if the fetch layer renders before it decides the response is bad. Set retry limits and, more importantly, record why a retry happened. If most retries are 429 or 403 responses, the fix is pacing or a different request path, not a bigger budget.

Managed fetch services expose the same arithmetic from the other side. Firecrawl's crawl documentation, for example, states that each page crawled consumes one credit, with additional credits for JSON mode and PDF parsing (Firecrawl docs). That page-based pricing is easy to reason about but hides the field-level question: if a page yields four useful fields, the per-field cost is four times the per-page number.

A static HTTP fetch returns whatever the server sends. If that is a thin shell that loads content through JavaScript, the fields you want are not in the body, so a plain client discovers nothing and parses nothing (Crawlbase). The fix is a headless browser, and the cost is time.

Render cost is best counted in seconds of browser occupation, not in pages. A fast static page may settle in a few hundred milliseconds; a script-heavy page that waits on several network calls can hold a browser instance for many seconds. Trendyol's account of rendering two billion pages with a headless browser describes the operational reality of running that stack at scale, including the difference between server-rendered pages that need a single request and client-rendered pages that need execution (Trendyol Tech).

Count it as:

render_seconds_per_thousand_fields =
  (sum_of_browser_seconds / fields_extracted) * 1000

The insight most teams miss is that render seconds, not requests, decide whether you can run on a small worker pool. If you extract 250,000 fields and each page takes 4 seconds of browser time at 8 fields per page, you spend 31,250 pages times 4 seconds, which is 125,000 browser seconds, or about 35 browser hours. At 8 concurrent browsers that is roughly 4.3 hours of wall-clock work. Double the render wait and you double the pool or the latency, whichever you prefer to lose.

Two controls move this line. First, try the static reader before the browser: many "JavaScript" pages still ship the target values in the HTML, in JSON-LD or in an embedded state object. Second, render once and extract many fields from the single rendered state rather than re-rendering per field. Every additional field taken from an already-rendered page is nearly free, which pushes your per-field render cost down sharply.

Raw HTML is mostly boilerplate. Navigation, cookie banners, related links and ad slots can easily outweigh the article or listing you want. Cleaning strips that away and returns readable text, and it costs compute whether you do it with a parser, a readability library or a hosted service that returns markdown.

Firecrawl's crawl endpoint exposes an only_main_content option alongside its format choices, which is exactly the kind of switch that changes this line (Firecrawl docs). Turning it on usually reduces downstream token counts as well, because the verifier and extractor no longer see the navigation.

Count clean compute in the same currency as rendering (worker milliseconds) so the two can be added:

clean_ms_per_thousand_fields =
  (strip_and_clean_worker_ms / fields_extracted) * 1000

This line is the one most likely to be dominated by a cache. Cleaned text for an unchanged page is identical on every request, so if your source has a stable URL and a version or hash, you can store the cleaned result and skip both the fetch and the clean. The Firecrawl documentation notes a max_age option in scrape options, which is a freshness window expressed as a cache lifetime, and it also notes that crawl results are retained via the API for a limited period (Firecrawl docs). Both facts point the same way: repeated reads of the same page should not pay full price twice.

The verifier is the step that checks an extracted value against the page it came from. It is the most misunderstood cost line, because it is priced per token while the work it does is per field.

Verification is not one technique. Research on schema-guided extraction pipelines shows that strict schema validation plus domain rules plus a post-processing recovery module caught most failures, and that a large share of extraction failures were systematic formatting issues rather than semantic misunderstanding (Algorithms, MDPI). That result is the argument for gating: run deterministic checks (format, range, allowed values, cross-field consistency) before you spend a token.

When you do spend tokens, the established pattern is an evaluation prompt that receives the source material and the candidate output and returns a label or score. Arize's guide describes a judge that compares a response with the supplied evidence and applies a rubric, and recommends validating the judge against reviewed examples before trusting its scores (Arize). Evidently's guide covers the same mechanics and the same caveat: judge quality depends on the prompt, the model and the task (Evidently AI).

Count verification in tokens per checked field, then convert to tokens per thousand fields:

verifier_tokens_per_thousand_fields =
  (total_verifier_input_tokens + total_verifier_output_tokens)
  / verified_fields * 1000

The decision rule that matters: verify only fields where a wrong value is expensive, and pass the smallest evidence that lets the judge decide. Sending the whole cleaned page for every field multiplies your token line by the number of fields. Sending the paragraph or table row that contains the value keeps the line small and the judgment easier.

Structured outputs also reduce verifier volume, because a schema that enforces types and required keys removes a class of failures before verification starts. The OpenAI structured outputs documentation describes schema-enforced responses as a way to avoid omitting required keys or producing invalid enum values (OpenAI docs), and practitioner write-ups make the same case: enforce structure at generation time, then verify meaning (Agenta).

Take a pipeline over public company records. Sources include state business registries and federal filings, which vary widely in structure. The Colorado Secretary of State business search, for instance, is a form-driven registry that returns entity details, while SEC EDGAR full text search covers filings since 2001 (Colorado SOS, SEC EDGAR). A federal incorporation lookup is not available, so incorporation records live at the state level, and each state portal has its own interface and may charge for certified copies (Commerce Research Library). Your fetcher therefore faces many different page shapes, which is exactly when per-field costs drift.

Assume 50,000 pages, 10 fields per page, so 500,000 fields:

• Requests issued with retries: 62,500. That is 125 requests per thousand fields.
• Browser seconds: 50,000 pages times 3 seconds, of which 60 percent need a browser. That is 90,000 browser seconds, or 180 seconds per thousand fields.
• Clean compute: 40 worker milliseconds per page, so 2,000,000 milliseconds, or 4,000 milliseconds per thousand fields.
• Verifier: 25 percent of fields verified at 900 input tokens and 40 output tokens each. That is 125,000 verified fields at 940 tokens, so 1,175 tokens per thousand fields overall.

Now change one variable. Suppose a cache serves 70 percent of pages unchanged, so only 15,000 pages are fetched. Requests fall to about 18,750 (37.5 per thousand fields), browser seconds fall to 27,000 (54 per thousand fields), and clean compute falls to 600,000 milliseconds (1,200 per thousand fields). Verification volume stays the same if you still verify every extracted field, which is the point: the cache cuts the first three lines and leaves the fourth untouched. If verification dominates your bill, caching is not your lever.

This is also why the choice of fetch stack matters less than the shape of the workload. A cheap static reader that handles even half of your pages removes render seconds entirely for those pages. A headless browser, however well priced, cannot beat zero.

The decision rule is about which line dominates.

If requests or render seconds dominate, invest in fetch strategy: static-first with browser fallback, URL rules that stop the crawler wandering, dedupe across tracking parameters, and a cache keyed on URL plus content hash. If clean compute dominates, look at what you are cleaning: only_main_content, a cheaper parser, or caching the cleaned text. If verifier tokens dominate, change what you verify and how much evidence you send, not how you fetch.

Two practical tests before you buy anything:

1. Sample 200 pages and record, per page, whether a plain static fetch contained every target field. The share that do is your static ceiling, and it caps how much render time you can remove.
2. Take 100 extracted fields, check them with deterministic rules alone, and count how many pass. The remainder is your true verification demand, and it is usually far smaller than the field count.

There is a trap in optimizing the cheapest line. Teams often spend weeks reducing request counts when requests are a minor cost and verifier tokens are the majority. Measure all four lines for a week before changing anything. The meters are cheap to add: a request counter, a browser timer, a worker clock and a token tally per field.

When you want one place to run the model side of this, including the verifier calls across many providers, Quanta Inference exposes text and embedding models through one OpenAI-compatible API key with prepaid pricing in US dollars (https://quantaintelligence.ai/inference). And if you would rather hand the whole fetching and extraction pipeline to a specialist who returns finished deliverables, Quanta Experts is the natural next step (https://quantaintelligence.ai/agents).