Skip to main content

​
URL Scraper SSRF Guard

Status: active as of 2026-05-12 Owner: services/url_scraper.py

​
Threat model

POST /api/topics/seed-url is auth-required, but before this guard shipped, services/url_scraper.py accepted any http(s):// URL and followed redirects with no IP filter. The route fetched the page, extracted title + content, and returned them in the task metadata response. An authenticated operator (or anyone who minted a token via the prior open Dynamic Client Registration before audit P0 #3 closed it) could drive scraping against internal-only endpoints exposed on the worker host or its tailnet: The title + first few KB of the response body are exfiltrated through the API response, regardless of whether the operator owns the internal endpoint or just shares the tailnet. A second attack: an attacker-controlled public page can 302 redirect to one of the above IPs. Without per-hop re-checking, httpx’s follow_redirects=True would chase the 302 and serve the internal response to the caller.

​
Defense

Layer 1 — IP denylist before every HTTP request. _resolve_and_check(url, site_config) runs before the initial GET and again at every redirect. It:
  1. Parses the URL, extracts the hostname.
  2. Short-circuits on literal IP hostnames (http://127.0.0.1 is refused without a DNS round-trip).
  3. Refuses the well-known loopback names localhost / localhost.localdomain regardless of /etc/hosts.
  4. Resolves the hostname via socket.getaddrinfo (DNS-aware, IPv4 + IPv6) — never socket.gethostbyname (IPv4-only).
  5. Checks every returned IP against the denylist. Any blocked IP short-circuits the request with SSRFBlockedError.
Resolved IPs and unparseable input both default to “blocked” — the guard fails closed.

​
Denylist

Layer 2 — manual redirect loop with per-hop re-check. _safe_get(client, url) replaces httpx’s built-in redirect follower. For each hop it:
  1. Calls _resolve_and_check on the current URL.
  2. Issues a single client.get (no auto-follow).
  3. If the response is a 3xx with a Location header, computes the absolute target via urljoin, refuses non-http(s) schemes (e.g. file://, gopher://), and loops.
  4. Caps the chain at MAX_REDIRECTS = 5 hops.
Both _fetch (used by the generic + arXiv paths) and _scrape_github (used for the api.github.com calls) construct their httpx.AsyncClient with follow_redirects=False and route through _safe_get.

​
Operator override

app_settings.url_scraper_allow_internal_ips (default false). Flip to true when there’s a legitimate internal-scraping need (testing a staging instance on the LAN, scraping a local doc server on 100.x.x.x). The override short-circuits both _resolve_and_check calls — DNS still happens but no IP is denied. Flip it back off when done — leaving it on undoes the entire guard.

​
DNS rebinding posture

The current implementation resolves the hostname once before each HTTP request and trusts the kernel + libc resolver to reuse that answer when httpx opens the actual socket. An attacker who controls DNS for the target hostname can return:
  • a public IP at our getaddrinfo call (passes the gate)
  • a private IP at the kernel’s connect-time resolve (hits internal)
Closing this fully requires connecting to the resolved IP literal and passing the original Host: header separately (or pinning the connection via a custom AsyncHTTPTransport). That’s deferred — see the TODO in _resolve_and_check’s docstring. The current “resolve + check before each hop” closes ~95% of the attack surface (every redirect-based and literal-IP attack) and is reviewable in ~50 lines.

​
Test coverage matrix

Tests live at tests/unit/services/test_url_scraper_ssrf.py (54 cases). All tests use unittest.mock.patch on socket.getaddrinfo and httpx.AsyncClient — no real DNS or HTTP egress.

​
Operational notes

  • SSRFBlockedError subclasses URLScrapeError, so existing callers (routes/topics_routes.py:71-73) catch it via the existing except URLScrapeError branch and surface a 400 with the explanatory message. No route-handler change required.
  • The error message includes the override-flag instruction so an operator hitting the guard on a legitimate internal URL can self-serve.
  • Run-time cost: one getaddrinfo call per HTTP hop. Negligible compared to the HTTP round-trip itself.