Whose Content Is Loaded: A Source Allowlist for Fetched Text

A system prompt that says "only fetch from docs.example.com" changes the model's behavior on average. It does not change what the process can do. The model is a probabilistic component, and OWASP is explicit that prompt injection happens when input alters a model's behavior in unintended ways, including content carried in from external sources like websites and files (LLM01: Prompt Injection). The same page notes that retrieval augmented generation and fine-tuning do not fully mitigate prompt injection, which means the defense has to live somewhere other than the model.

The practical consequence: if the fetch tool accepts a URL and the model supplies that URL, then the model is the authority on which URL gets fetched. Whatever the prompt says, the worst case is a URL that the model was talked into supplying by a document it read a minute earlier. Indirect injection is exactly this path, where an LLM accepts input from an external source and that content alters its behavior (OWASP LLM01).

OWASP's mitigation list puts the pieces in order. It recommends enforcing privilege control and least privilege by handling sensitive functions in code rather than giving them to the model, segregating and identifying external content, and conducting adversarial testing (LLM01). A gateway allowlist is the privilege control. The prompt is the documentation of intent.

There is a second reason to put the check in code: consistency. A model that follows the instruction 99 times out of 100 has 99 correct fetches and one that you will find in a log if you are lucky. A gateway that blocks unlisted destinations has zero unlisted fetches, because the request never leaves.

Injection tests and allowlists are complementary, not competing. An injection test suite tells you whether behavior changed under adversarial input. An allowlist shrinks the set of content that can change behavior in the first place. Run both, but do not use a green injection suite as evidence that you can skip the allowlist, because the suite tests the variants you thought of.

Put the allowlist in a file, a policy store, or a config service, and have the fetching code load it on startup and on change. Nobody should edit a Python list to grant access. Version the file, review it in a pull request, and keep a record of who added which entry and why.

JSON Schema is a reasonable way to express the shape of the policy so that bad entries fail at load time rather than at fetch time. The JSON Schema validation vocabulary defines keywords for asserting what a valid document must look like, including the required keyword for property presence (JSON Schema Validation). A policy with a typo in a hostname should be rejected before the service starts.

{
  "version": 7,
  "default": "deny",
  "web": [
    {"host": "docs.example.com", "path_prefix": "/help/", "methods": ["GET"]},
    {"host": "status.example.net", "path_prefix": "/api/v2/", "methods": ["GET"]}
  ],
  "database": [
    {"table": "billing.invoices", "columns": ["id", "amount", "status", "issued_at"],
     "row_filter": "tenant_id = :tenant"}
  ],
  "documents": [
    {"collection": "policies", "filter": {"audience": "customer"}, "max_bytes": 200000}
  ],
  "limits": {"per_request_bytes": 500000, "per_run_requests": 40}
}

Three details matter more than the rest of the file. The default key is deny, so an entry that does not match anything is blocked. Every web entry has a path_prefix, so the grant is narrow. The database entry names columns rather than *, because column lists are the cheapest form of least privilege and they survive schema changes better than a blanket select.

Keep the policy small enough to read in one sitting. If your web section has more than twenty hosts, the task is probably too broad, or you are granting hosts you have not justified. The honest move is to cut it down.

The gateway is the process that actually opens the connection. It could be a dedicated service, an egress proxy, or the fetch client inside your agent runtime, as long as it is the single code path through which every outbound request passes. If two paths exist, the allowlist is enforced on one of them.

The validation order matters, and it is the part most implementations get wrong. Resolve, normalize, then check the resolved destination against the list, then connect to that exact resolved address. Checking the string the model supplied is not enough, because the resolved destination can differ from the text after DNS resolution, redirects, protocol handlers, and alternative address encodings. The OWASP guidance on request forgery prevention is blunt about this: URL denylists and hostname-only checks can be bypassed by alternate IP address formats, DNS resolution to private addresses, redirects, protocol handlers, and DNS rebinding, so effective validation restricts approved schemes, hosts, and ports while checking the resolved destination at connection time (Sweet Security on SSRF).

The same source notes that cloud environments make this worse, because a vulnerable workload may reach instance metadata services and obtain temporary credentials (Sweet Security). An agent that can be talked into fetching http://169.254.169.254/ is not a content problem, it is a credential problem.

A simple gateway check reads like this:

ALLOWED_SCHEMES = {"https"}
MAX_BYTES = 500_000

def fetch(url: str, policy: dict, tenant: str) -> bytes:
    parts = urlparse(url)
    if parts.scheme not in ALLOWED_SCHEMES:
        log_block(url, "scheme_not_allowed")
        raise Denied("scheme")
    host = parts.hostname or ""
    path = parts.path or "/"
    if not any(host == e["host"] and path.startswith(e["path_prefix"])
               for e in policy["web"]):
        log_block(url, "host_or_path_not_listed")
        raise Denied("allowlist")
    resolved = resolve_and_pin(host)  # single IP, no re-resolution on connect
    if is_private_or_link_local(resolved):
        log_block(url, "resolved_to_private_range")
        raise Denied("private_range")
    return get_pinned(resolved, path, query=parts.query,
                      max_bytes=MAX_BYTES, follow_redirects=False)

Two choices in that sketch are deliberate. follow_redirects=False means a redirect is not followed at all, so the client never silently changes destination. Instead, the redirect location is returned to the gateway, which runs the whole check again against the same policy. Second, resolve_and_pin returns one address and the connection is made to that address, so a DNS answer cannot change between the check and the connect.

If you need to follow redirects for usability, cap the chain (three hops is a common choice), re-check every hop, and log each hop with its verdict. Never let a hop through just because the first URL passed.

The database path has its own version of the same mistake. An agent given one owner-level database credential has ambient authority over every row that credential can reach, and prompt injection exploits that authority. Running the agent as the end user, with the database filtering rows against a verified identity claim, is the pattern that holds under injection; the alternative relies on application code adding the right tenant filter to every query, which agent-generated SQL can route around (Powabase on row-level security for AI agents).

Two PostgreSQL statements make the difference between row-level security that filters rows and row-level security that is decoration: ENABLE ROW LEVEL SECURITY turns policies on for everyone except the table owner, and FORCE ROW LEVEL SECURITY applies policies to the table owner as well (Powabase). If the agent's role is a table owner or holds BYPASSRLS, the policies are skipped, so check both flags:

SELECT rolname, rolsuper, rolbypassrls
FROM pg_roles WHERE rolname = current_user;

Both flags must be false for the agent role. That check belongs in your startup validation, not in a wiki.

For documents, the allowlist entry is a collection plus a metadata filter, and the query layer adds that filter to every retrieval, whether the model asked for it or not. Chunk-level filtering is the only version that survives a model that constructs its own filters. If you use a managed retrieval service, confirm that the filter is applied in the query, not as a post-filter after retrieval, because post-filtering still reads the disallowed chunks.

The worked example that makes the volume difference concrete: suppose an agent run makes 40 fetch calls on average and 200 in the tail. With a 200 KB per-document cap and a 500 KB per-request cap, a run cannot bring in more than 20 MB of text, and the tail run is bounded by the same numbers. Without caps, one archived page of 12 MB can occupy the context and the bill. The caps are part of the allowlist, not an afterthought.

The allowlist needs its own tests, separate from the injection suite. Record real fetch traffic and replay it against the gateway with expected verdicts. Recording and replay as an HTTP technique means capturing requests, sending them to a target, and checking behavior, with the tooling supplying capture and replay while your assertions decide pass or fail (GoReplay on record and replay testing). The same article is careful about what a capture represents: it covers only the traffic observed during its collection window, so keep explicit tests for cases that did not occur.

That limitation is the reason to hand-write the adversarial cases. A capture from a normal week contains no fetch to http://169.254.169.254/, no redirect to an unlisted host, no hostname that resolves to a private range. Write those by hand and add them to the replay set. Replay tooling can also generate mocks from real dependency responses, which keeps the test fast and offline (Mokapi on recording and replaying API traffic).

For the model side of the harness, a deterministic mock server removes the cost and variance of real calls. A mock LLM server that behaves as a real HTTP server, streams in provider formats, routes fixtures by substring, regex or predicate, and injects tool calls lets you test the full agent loop without burning budget or fighting non-deterministic output (CopilotKit on LLMock). Point that mock at your gateway with fixture tool calls whose arguments include unlisted hosts and private-range IPs, and assert that the gateway blocks them before the connection.

Two testing practices carry over from general CI hygiene. Keep the test suite fast, since fast feedback is what makes frequent runs practical (GitLab on continuous integration best practices). And treat failures as process signals rather than blame signals, because a block that surprises you is a gap in the policy, and the fix is a reviewable change to the policy file (GitLab).

The test matrix to keep in CI:

Every decision the gateway makes should produce a log line with the URL or table, the verdict, the reason code, and the policy version. Without the policy version, you cannot tell whether a block was correct at the time or caused by an edit. Without the reason code, you cannot tell a missing entry from a malformed one.

Make policy review part of the normal change process. A new host entry is a new trust decision, so the pull request should state which task needs it, which path prefix is required, and when it can be removed. Entries added for a one-off debugging session are the ones that persist for years.

OWASP's broader guidance points the same way. It recommends restricting the tools an LLM can access, applying least privilege to data, APIs and tools, and logging and monitoring suspicious prompt patterns (Mend on the OWASP Top 10 for LLM Applications). The allowlist is where least privilege for data becomes a fact rather than a diagram. The project itself has grown into a wider GenAI security initiative with published Top 10 material you can align internal reviews to (the OWASP project page).

A short cadence keeps the list honest. Once a month, list the entries that were never matched by an allowed request in that period, and ask whether each one is still needed. That review takes minutes and shrinks the surface more reliably than any audit.

If you are building this kind of agent and want the fetching layer to be a managed component rather than a service you patch, a specialist can build it with you at Quanta.