How Poindexter itself is tested and deployed
Last Updated: 2026-06-03What this doc is. A transparency record of how Poindexter (the project, not your self-host) is tested and shipped to gladlabs.io. Kept here so that contributors can understand why a PR fails CI, what gets run on every push, and how changes reach production. What this doc isn’t. A recipe for setting up your own CI. If you want to run Poindexter on your own infrastructure, the only supported deployment ispoindexter setupthenbash scripts/start-stack.shon a single machine.
The flow
Glad-Labs/poindexter (the private origin),
NOT the public poindexter repo. The public repo has no deploy
workflow — Release Please is the only thing producing artifacts.
The cross-repo sync is automatic: GitHub Actions workflow
.github/workflows/sync-to-public-poindexter.yml runs on every push
to origin/main and mirrors the filtered subset to the public repo
in ~30s, authenticating with a dedicated GitHub App
(glad-labs-mirror-sync, installed on poindexter with Contents +
Workflows read+write; secrets MIRROR_SYNC_APP_ID +
MIRROR_SYNC_APP_PRIVATE_KEY on glad-labs-stack). Migrated
2026-06-13 from a fine-grained PAT that silently expired and froze
the mirror (which had itself replaced an SSH deploy key 2026-05-09).
Just git push origin main and the public mirror updates itself.
scripts/sync-to-github.sh strips private files (web/public-site,
web/storefront, mcp-server-gladlabs, marketing, premium dashboards,
writing_samples, gladlabs-config, .shared-context, CLAUDE.md,
scripts/bootstrap.sh, and select internal docs — audits, plans,
and the operator-only finance / CI-runner runbooks; the rest of
docs/ ships to Mintlify) before pushing.
The sync filter also performs content-level rewrites:
- docs.json: operator-branded
gladlabs.ioURLs are rewritten to poindexter-neutral GitHub URLs so OSS forks don’t inherit operator branding. - CHANGELOG.md: lines mentioning operator-private values — private
finance-module
app_settingskeys, Tailnet hostnames, or hardware-cost figures — are redacted before the mirror push. - Operator-name regex: the leak guard uses
[Mm]atthew (?:[A-Z]\.\s+)?[Gg]ladding(with optional middle-initial group) to catch both the plain and middle-initial forms of the operator name. Added in the 2026-05-27 security audit — the middle-initial form was slipping past the old[Mm]atthew [Gg]laddingpattern.
[skip-public-sync] in the commit message to
keep a particular commit private (in-progress branches, sensitive
WIP).
Debugging “Vercel is failing”
If you see a notification that Vercel deploy failed:- Check the Vercel dashboard — the deploy runs directly from
the
glad-labs-stackrepo via Vercel’s GitHub integration, not via a GitHub Actions workflow. - Common build failure:
next.config.jsrejects localhost URLs in production. SetSKIP_ENV_VALIDATION=truein Vercel env vars, or ensureNEXT_PUBLIC_API_BASE_URLis set to a real URL (or left empty for static-only builds). - If tests fail locally: reproduce with
docker exec poindexter-worker python -m pytest tests/unit/ -q. Frontend:cd web/public-site && npm run test.
Local vs CI environment differences
A few tests pass in CI but fail inside the worker container because the worker runs withENVIRONMENT=production set and some
middleware evaluates that at import time. Tests that depend on the
brain module or sentry-sdk are skipped in Docker (the modules
aren’t available in the worker container). See the skipif
decorators in test_database_service.py and
test_sentry_integration.py.
Key files
.github/workflows/unit-tests.yml— backend pytest, exposed as thetest-backendstatus check. One of the two branch-protection required checks; adetect-changesstep short-circuits the expensive pytest steps on docs-only changes while still reporting green (a required check must always report — see “CI minutes / cost discipline” below). No deploy step..github/workflows/migrations-smoke.yml— applies every migration against a clean Postgres + pgvector. The other branch-protection required check; fires on every PR + push to main..github/workflows/playwright-e2e.yml— frontend E2E (Playwright),paths:-gated toweb/public-site/**. Non-required. The frontend Jest unit run + JS lint are hook-only, not run in CI (see the workflow header)..github/workflows/security.yml/grafana-panels-lint.yml— non-required scans: gitleaks / trivy / sbom + path-specific lints, and the paths-gated Grafana panel lint, respectively..github/workflows/sync-to-public-poindexter.yml— auto-mirror from glad-labs-stack to poindexter on every push to main.scripts/sync-to-github.sh— filter that runs inside the sync workflow. Strips operator-only files before pushing the public subset..github/workflows/release-please.yml— Release Please onGlad-Labs/poindexter(the source repo — NOT the public mirror; running it on the force-rebuilt mirror broke versioning, see the workflow header). Versioning only. Runs daily at 08:00 UTC (wason: pushto main) so a day’sfeat:/fix:commits batch into one release instead of one-per-merge — the per-merge cadence 3×-amplified Actions-minute usage (each release commit re-ran the full suite AND re-triggered this workflow).workflow_dispatchcuts an ad-hoc release immediately..github/workflows/regen-app-settings-doc.yml— nightly regen ofdocs/reference/app-settings.mdagainst a clean Postgres seeded by the baseline migration. Opens a single PR onchore/regen-app-settings-docwhen the file drifts; the branch is force-pushed every run so the PR always reflects the latest regen. Per poindexter#439.src/cofounder_agent/tests/— Python unit tests (pytest). Thetest-backendcheck runs the full backend suite (several thousand cases; the exact count drifts as agents add tests, so it is not pinned here).web/public-site/next.config.js— has avalidateEnvcheck that rejects localhost URLs in production.SKIP_ENV_VALIDATION=truebypasses for local dev.
CI minutes / cost discipline
Actions minutes are billable on this private repo, and a high PR + push-to-main volume (nightly scheduled agents, release commits, docs bots, dependabot) multiplies fast. The rules that keep the bill down:- Only
test-backendandmigrations-smokeare branch-protection required checks. Required checks can’t bepaths:-filtered — a skipped required check never reports, so it would block the PR forever; they keep firing and gate their expensive steps instead (see thedetect-changesstep inunit-tests.yml). Every other workflow is non-required and ispaths:-filtered freely. playwright-e2eispaths:-gated toweb/public-site/**+ the playwright config + rootpackage*.json. A backend/docs/infra change skips the Chromium build entirely (those specs only exercise the static Next.js site, so they can’t regress on a backend change).security.ymlclassifies changed paths first (thechangesjob), then runs only the relevant file-specific jobs (trivy-config/action-pins/shell-line-endings/poetry-lock).gitleaks/trivy-fs/sbomalways run — a secret or CVE can land in any file. The weekly baseline + manualworkflow_dispatchscans run every job regardless.grafana-panels-lintispaths:-gated toinfrastructure/grafana/**+ the lint script + migrations — the model the others copy.- Release Please batches daily rather than per-merge (see Key files above).
- Deferred: a GitHub merge queue (would run the heavy suite
once at merge instead of PR-then-post-merge-on-main) is intentionally
NOT adopted yet — a merge queue amplifies flaky failures (an evicted
entry rebuilds everything behind it), so it waits until the unit
suite is reliably green. CodeQL is moving to advanced setup
(PR + weekly schedule,
paths-ignorefor docs/infra) to drop its per-push-to-main scan — tracked as the fast-follow to this sweep.
Coverage (#995)
Coverage reuses the existingtest-backend matrix in
unit-tests.yml — we do not add a second test job or a parallel
coverage workflow (that would duplicate the per-dir/--forked split and
drift as test dirs are added). But it is gated to the nightly schedule
(cron: 0 9 * * *) + manual workflow_dispatch only — NOT every PR.
A job-level COV env var holds --cov=cofounder_agent --cov-append --cov-report= on those events and is empty on push/PR, so every
pytest step appends $COV: on a PR that expands to nothing (lean ~8m
run), on the nightly run it turns on coverage. The Initialize coverage data / Coverage report / Upload coverage.xml artifact steps are
likewise gated to schedule/dispatch. Why gated, not per-PR:
coverage instrumentation across the --forked split roughly doubled
test-backend (8m → 17m). Your nightly agents open several backend PRs
a day, so paying that on every PR would erode the CI-minutes win — a
once-a-day trend line gives the signal without the per-PR tax.
Coverage is ADVISORY right now — it never fails the build. There is
deliberately no --cov-fail-under yet:
test-backendis a required branch-protection check. A blind--cov-fail-underwould block every PR before we even know the current percentage.- The plan is a ratchet, not a target: read the baseline % from the
first few CI runs (the
Coverage reportstep log / thecoverage-xmlartifact), then set--cov-fail-under=<baseline>and bump it upward over time as coverage improves. The number only ever goes up — a PR that drops below the current floor fails; one that holds or improves passes. This avoids gating on the long tail while still catching regressions once a floor is set.
coverage.xml — “N tests pass” plus “X% of cofounder_agent
is exercised”, instead of just the test count.
The public release repo is separate
github.com/Glad-Labs/poindexter is the open-source release repo.
It gets a filtered snapshot via the auto-sync workflow above. It
does NOT auto-deploy anywhere. Vercel watches the private origin
(Glad-Labs/poindexter), not the public mirror.
The public mirror has allow_force_pushes: true in its branch
protection — the mirror is rebuilt from scratch on every sync, so
force-push protection on a derived branch would just keep the mirror
permanently stale. Public-side CI (test-backend, migrations-smoke,
Mintlify Deployment, link-rot) still has to pass on the resulting
commit.
Deploying the local worker (bringing prod up to main)
The worker / brain / pipeline-bot / prefect-worker containers bind-mount
the deploy clone (POINDEXTER_DEPLOY_ROOT, defaulting to
~/.poindexter/deploy/glad-labs-stack) — not this dev checkout. The deploy
clone is what the running pipeline actually executes. A merge to main does
not reach the worker until the deploy clone is synced and the containers
restart. Leaving the deploy clone behind is how production silently drifts
behind main.
The canonical one-command deploy:
main in the dev checkout, fast-forwards to origin/main,
syncs the deploy clone (deploy-checkout-sync.ps1) so the containers get
the new code, verifies both checkouts are at origin/main, then restarts the
pipeline containers and waits for the worker healthcheck and
poindexter_worker_up=1. There is no image rebuild — app code is
bind-mounted from the deploy clone, so a sync + restart is the deploy
(dependency / base-image changes still need docker compose build).
Split-brain fix (glad-labs-stack#1295). Before this fix,Routine Python merges now auto-deploy. The 10-mindeploy-worker.ps1only fast-forwarded the dev checkout and left the deploy clone lagging up to 10 minutes behindorigin/main. The script now explicitly syncs the deploy clone before restarting containers, and verifies the deploy clone HEAD matchesorigin/mainbefore proceeding.
deploy-checkout-sync.ps1
scheduled task (above) bounces poindexter-worker + poindexter-pipeline-bot
whenever it advances the deploy clone, so a merged code change reaches the
running worker within ~10 min on its own. deploy-worker.ps1 remains the tool
for an immediate deploy (skip the wait) and is still required for dependency /
base-image changes (which need docker compose build) and for
poindexter-prefect-worker / poindexter-brain-daemon bootstrap-level changes.
Deploy-drift canary (glad-labs-stack#942). Because the worker / brain
bind-mount the deploy clone, “merged on main” does not mean “running in prod”
until you run the deploy above. The brain’s branch_drift_probe closes that
loop: every ~15 min it reads the deploy clone’s HEAD from a read-only .git
mount (${POINDEXTER_DEPLOY_ROOT:-.}/.git:/host-git:ro on the brain-daemon
container — pointing at the deploy clone, not the dev checkout, per
glad-labs-stack#1295), compares it to origin/main via the GitHub API
(gh_token), and pages the operator (Telegram / Discord) when prod is behind.
It is alert-only; the remedy it points at is
pwsh ./scripts/deploy-worker.ps1. Tunables (in app_settings):
branch_drift_probe_enabled, branch_drift_poll_interval_minutes,
branch_drift_repo, branch_drift_dedup_hours, branch_drift_git_dir.
Deploying the canary itself requires a brain image rebuild
(docker compose build brain-daemon && up -d brain-daemon), since the
.git mount + the git binary are new.