Choosing a Reader for Each Page Type: HTML, PDF and JS-Rendered

The table below is the core of the article. It assumes the URL is known and the goal is clean text, not screenshots or interaction.

Two rows deserve emphasis. The SPA row is where teams waste the most money, because a browser is the only reader that works but a JSON endpoint is often cheaper still. The PDF-in-a-viewer row is where teams waste the most time, because the URL they logged is not the URL that contains the text.

Numbers make the choice concrete. The figures below are illustrative unit costs you should replace with your own measured latency and your provider's billing, but the arithmetic is the point. Suppose a plain fetch takes about 0.2 seconds of wall clock and a few hundred kilobytes, a headless render takes several seconds and loads many subresources, and a PDF parse takes well under a second per page for born-digital files.

If 70 percent of a 10,000 page batch is static HTML, 20 percent is PDF and 10 percent truly needs a browser, routing by page type keeps the browser fleet at roughly 1,000 pages instead of 10,000. That is a tenfold cut in the most expensive resource. A managed crawl service prices the same distinction explicitly: each crawled page consumes one credit, with PDF parsing billed per PDF page and structured JSON extraction billed as additional credits on top (Firecrawl). The lesson generalizes whether you self-host or buy: the render is the line item, so avoid it where a cheaper reader wins.

There is a second cost that rarely appears in budgets: blocked requests. Sites watch for datacenter IP ranges, missing browser headers and inhuman request rates, and a naive loop around a bare HTTP client gets rate-limited, IP-blocked or served a CAPTCHA before reaching the content (Crawlbase). A browser does not fix blocking by itself. Stealth patches, proxy rotation and managed fingerprint handling each close part of the detection gap and none closes all of it (Browserless). Budget for proxies separately from readers.

Cheap readers sometimes return text that is technically non-empty and completely useless: cookie banners, navigation, a "enable JavaScript" notice, or a garbled column dump. The fix is a deterministic gate that runs on every extraction before it is stored, and it should be code, not a model. Deterministic checks are the right tool when a rule can express the expectation, and models are for judging meaning (Arize).

A practical gate uses four checks and a points system:

• Length: body text characters after boilerplate removal. A product page under roughly 200 characters is suspicious.
• Unique-word ratio: unique words divided by total words. Very low ratios indicate repeated menus or template text.
• Boilerplate share: characters inside nav, header, footer and aside elements divided by total characters. High share means the main content did not load.
• Required fields: for structured jobs, the presence of the fields your schema demands, such as title and date.

Here is a compact implementation of that gate in Python, scoring each extraction from 0 to 4 and routing accordingly:

import re
from urllib.parse import urlparse

BOILERPLATE = re.compile(r"<(nav|header|footer|aside)[^>]*>.*?</\1>", re.S | re.I)
TAGS = re.compile(r"<[^>]+>")

def text_of(html: str) -> str:
    return re.sub(r"\s+", " ", TAGS.sub(" ", html)).strip()

def quality(html: str, required=("title",)) -> tuple[int, dict]:
    stripped = BOILERPLATE.sub(" ", html)
    body = text_of(stripped)
    full = text_of(html)
    words = body.split()
    unique_ratio = len(set(w.lower() for w in words)) / max(len(words), 1)
    boiler_share = 1 - (len(body) / max(len(full), 1))
    score, detail = 0, {
        "chars": len(body),
        "unique_ratio": round(unique_ratio, 3),
        "boilerplate_share": round(boiler_share, 3),
    }
    if len(body) >= 200:
        score += 1
    if unique_ratio >= 0.30:
        score += 1
    if boiler_share <= 0.50:
        score += 1
    lower = body.lower()
    if all(tag in lower for tag in required):
        score += 1
    return score, detail

def route(url: str, fetch, render, parse_pdf):
    path = urlparse(url).path.lower()
    if path.endswith(".pdf"):
        result = parse_pdf(url)
        if quality(result["html"])[0] >= 3:
            return "pdf", result
    fetched = fetch(url)
    score, detail = quality(fetched["html"])
    if score >= 3:
        return "http", fetched
    rendered = render(url)
    if quality(rendered["html"])[0] > score:
        return "render", rendered
    return "quarantine", fetched

Two design notes. First, the thresholds are starting points, not laws: calibrate them on a labeled sample of a few hundred pages and watch the false escalation rate. Second, the gate belongs in the extraction layer, not the retrieval layer, because extraction accuracy is measured at the field level, against a verified ground truth, and is task specific, so an invoice benchmark does not transfer unchanged to contract clauses (LlamaIndex).

The parsing layer deserves its own warning. Optical character recognition converts visual content to machine-readable text imperfectly, and character errors, layout misinterpretation and inconsistent table handling propagate straight into extraction, corrupting data before your field logic ever runs (LlamaIndex). A scanned PDF that produces 400 characters of gibberish should fail the gate and escalate, not enter the index.

PDF is not a web page. It stores instructions for character and line placement rather than tagged structure, which is why parsing is a reconstruction problem: word identification, word order, paragraph integrity and tables all break in predictable ways (Adhikari and Agarwal). That property drives reader choice inside the PDF family.

The largest comparative study of the question evaluated ten popular tools across six document categories using the DocLayNet dataset. PyMuPDF and pypdfium generally outperformed other text extractors, all parsers struggled with scientific and patent documents, learning-based tools such as Nougat did better on those hard categories, and table detection quality varied by document type, with Table Transformer strong on financial, patent, law and scientific categories and Camelot best on government tenders (Adhikari and Agarwal). A separate hands-on test of seven Python extractors on a one-page business document with a six-column table found pypdfium2 returned clean text in about 0.004 seconds, fast and readable but without table structure (Aman Kumar).

The decision rule that follows is simple. For born-digital PDFs where you need prose, start with a fast rule-based parser. When the gate reports low length, broken word order or missing tables on scientific, patent or financial files, escalate within the PDF family to a layout-aware or learning-based tool before you consider a browser. A browser should almost never be your PDF reader, because rendering a PDF in a browser gets you pixels or a viewer, not structured text.

You cannot tune a routing table you do not measure. Emit one log record per URL with enough fields to recompute the decision later. A minimal schema looks like this:

{
  "url": "https://example.com/report",
  "page_type": "pdf",
  "reader": "pdf_parser",
  "escalated_from": "http",
  "latency_ms": 310,
  "bytes_in": 812344,
  "chars_out": 18422,
  "quality_score": 4,
  "required_fields_found": 2,
  "required_fields_total": 2,
  "http_status": 200,
  "outcome": "kept"
}

Then review three aggregates weekly: escalation rate by page type, quality score distribution by reader, and cost per thousand clean pages. If the escalation rate for a site climbs, the site changed its rendering and your page-type classifier is stale. If quality scores for the HTTP reader drop while escalations stay flat, your boilerplate selectors broke. Keep function and evaluator versions next to results when comparing runs, so a change in scoring is not mistaken for a change in extraction quality (Arize).

For open-ended downstream tasks, such as summarizing a page or answering questions over it, deterministic gates are not enough. There, an LLM judge with a rubric can score relevance or groundedness against the task input and any reference material, returning labels and scores you can aggregate (Arize). Judges are a flexible, fast, cost-effective approximation of human judgment, and their usefulness depends on the prompt, the model and the task complexity (Evidently AI). Validate the judge on reviewed examples before you trust its scores, and keep it out of the hot path where a regex would do.

Finally, structure your outputs so a bad extraction cannot silently corrupt a database. When a page yields text, hand it to a model with a JSON schema and require the response to conform, so required keys are present and enum values are valid rather than hoped for (OpenAI). Where a provider lacks native schema enforcement, schema validation plus a recovery pass still catches most failures, because the majority of extraction failures are systematic and correctable formatting problems rather than semantic misunderstanding (MDPI Algorithms). That is the same philosophy as the reader gate: fail loudly, escalate once, and log why.

If you are running this at scale across mixed sources, the natural next step is to keep the reader logic and push the model calls through one endpoint. Quanta Inference (https://quantaintelligence.ai/inference) serves text, image, video, audio and embedding models through one OpenAI-compatible API key with prepaid US dollar pricing, so the judge and the schema-bound extractor do not each need their own integration.