Skip to main content

​
Static Export Pipeline

The public site (web/public-site/) reads its post listings and content exclusively from R2 — there is no API call from Vercel back into the worker for routine reads. The lib/posts.ts → fetchPostIndex helper fetches ${STATIC_URL}/posts/index.json (today https://pub-1432fdefa18e47ad98f213a8a2bf14d5.r2.dev/static/posts/index.json, see How the site finds the bucket) and uses Next.js tag-based caching ({ tags: ['posts', 'post-index'] }) that the worker invalidates on publish. This means R2 is the source of truth for the homepage and archive. A publish that creates a posts row but never refreshes R2 silently freezes the public site until the next successful export. No bucket? With storage_provider=local (the fresh-install default) the same export, and every other R2UploadService call, goes to a folder the worker serves at /site/ with a small reader, instead of a bucket. The keys and file layout below are identical. See local-storage-provider.md.

​
Files on R2 (static/ prefix)

​
Write paths

​
publish_post_from_task (synchronous since 2026-05-11)

services/publish_service.py:publish_post_from_task is the single entry point for materializing a published post. After the posts row INSERT it now awaits export_post(_pool, slug) inline. The return value lands on PublishResult.static_export_success so /approve and /publish callers can surface the failure. Originally (pre-2026-05-11) export_post was scheduled via _spawn_background(...) as a fire-and-forget asyncio task. That worked for the long-lived FastAPI worker but failed silently in two scenarios:
  1. Prefect subprocess auto-publish — the content_generation flow runs in a subprocess that terminates when the flow returns. Background tasks scheduled mid-flow were cancelled before completing.
  2. Worker restart mid-publish — any restart between INSERT and the asyncio scheduler picking up the fire-and-forget task dropped the work.
Between 2026-05-08 and 2026-05-11 four published posts never reached the bucket; the homepage silently lagged the DB by 3 days. The await-inline fix trades ~3-5s of publish latency for guaranteed consistency.

​
StaticExportReconciliationJob (every 15 min)

Safety net at services/jobs/static_export_reconciliation.py. Compares COUNT(*) FROM posts WHERE status='published' against post_count in static/manifest.json. On drift (count mismatch or manifest older than the newest published_at by more than stale_minutes, default 30) it fires export_full_rebuild and emits an r2_static_drift finding via utils.findings.emit_finding. Config keys (under plugin.job.static_export_reconciliation):
  • stale_minutes — manifest staleness threshold (default 30)
  • r2_manifest_url — manifest probe URL (default points at the public R2 endpoint)
  • alert_on_drift — emit a finding alongside the rebuild (default true)
The job is registered in plugins/registry.py next to VerifyPublishedPostsJob; together they cover the two complementary failure modes (R2 missing the post vs. the public URL returning non-200).

​
fire_post_distribution_hooks (gate clearance path)

When approval gates clear on a status='awaiting_gates' post, publish_service.fire_post_distribution_hooks flips the status to published and also calls export_post (still via _spawn_background for now — separate gate-engine path with different lifecycle, not in scope for the 2026-05-11 fix).

​
Credentials

Reads from app_settings (category general): Legacy cloudflare_r2_* keys still resolve as a fallback (see services/r2_upload_service.py:_storage) but are deprecated.

​
How the site finds the bucket

The site names the bucket in one file, web/public-site/lib/static-url.js, and every other file imports from it:
  • STATIC_URL is NEXT_PUBLIC_STATIC_URL, else the default (the Glad Labs bucket’s static/ prefix). Every page, route handler, the sitemap, the RSS feed and the edge proxy (proxy.ts) fetch ${STATIC_URL}/posts/index.json and its siblings from it.
  • STATIC_ORIGIN is that bucket’s origin: scheme and host, no path. next.config.js allows it in the CSP connect-src and as a next/image remote pattern, and the podcast and video feed routes fetch /podcast/feed.xml and /video/feed.xml from it. The backend publishes both beside static/. When NEXT_PUBLIC_STATIC_URL is not an absolute http(s) URL it falls back to the default bucket’s origin.
The worker’s side of the same bucket is the storage_* settings above, and storage_public_url must name the same origin as the site’s STATIC_ORIGIN. Moving the bucket takes one edit on the site: change DEFAULT_STATIC_URL in lib/static-url.js, or set NEXT_PUBLIC_STATIC_URL for the deploy (it is read at build time, so redeploy after changing it on Vercel). The CSP allow-list and the image pattern follow, because next.config.js reads the same module. Nothing else on the site names the bucket, and __tests__/static-url-single-source.test.js fails if something does. That guard exists because the site used to hold a copy of the fallback in every file that read the export, and next.config.js kept one for the CSP. A missed copy sent one surface to the old bucket with no error, and when the CSP allow-list and the fetch URL drifted apart, /search silently returned no results (Gitea #262). The bucket’s host is also spelled outside the site, in the few places that have nowhere else to read it from: the DR re-import tool (it runs when the database is gone, so its default is a literal on purpose), the operator’s settings overlay, and two operator scripts. The brain’s R2 probe used to be one of them. It now reads storage_public_url and reports not_configured when that is unset, so an install no longer probes another operator’s bucket by default. tests/unit/test_bucket_host_agreement.py compares every such spelling with the site’s DEFAULT_STATIC_URL. A bucket move that misses one fails there, and the failure lists each stale file:line, so there is nothing to search for by hand. The DR tool’s default is held to the site’s exactly, path included.

​
Operator runbook

“R2 is stale, what do I do?”
  1. Check the reconciliation watchdog’s last job_runs entry — if it’s firing and the rebuild is failing, the metric in static_export_reconciliation job logs will show rebuild_ok=0.
  2. Run a manual rebuild via the MCP rebuild_static_export tool or directly: docker exec poindexter-worker python -c "..." invoking services.static_export_service.export_full_rebuild.
  3. Trigger ISR revalidation so Vercel re-fetches: await trigger_isr_revalidate(slug, site_config=sc).
“How do I know if a specific publish hit R2?” After 2026-05-11 the publish endpoints return PublishResult.static_export_success — true means R2 got the new index. Pre-2026-05-11 publishes have no per-publish signal; cross-check the manifest’s last_published_slug instead.