Secret Rotation Runbook
Last reviewed: 2026-05-08 Audience: solo operator (Matt) at 2am during an incident, or doing scheduled rotation Prereqs: Local PC online, Docker running, gh CLI authed, poindexter CLI installed,~/.poindexter/bootstrap.toml accessible, POINDEXTER_SECRET_KEY set
Every secret in Poindexter has a rotation procedure. Most live in app_settings with is_secret=true and are encrypted at rest via pgcrypto — see src/cofounder_agent/plugins/secrets.py for how that works. Two secrets (database_url and POINDEXTER_SECRET_KEY itself) are bootstrap-only and live in ~/.poindexter/bootstrap.toml.
This runbook lists every known secret, where to obtain a fresh value, how to set the new value, and what to restart to pick it up.
Scenarios this covers
- Routine 90-day rotation of API keys
- Emergency rotation after suspected compromise
- Re-seeding after the encryption key was lost (see
disaster-recovery.mdCONFIG-2) - Updating a third-party token that the provider expired / regenerated
local-development-setup.md. For incident triage, see incident-response.md.
Quick triage flowchart
How rotation works (quick model)
Every encrypted secret follows this lifecycle:- Generate a new value (via the provider’s UI / API, or
openssl rand) - Set the new value into
app_settingsviapoindexter settings set <key> "<value>" --secretor theset_secret(...)Python helper. Either path encrypts under the currentPOINDEXTER_SECRET_KEY. - Restart any in-process consumers so they re-fetch from
app_settings. Most async consumers callawait site_config.get_secret(...)per-request and don’t need a restart; some sync constructors cache at startup and DO. - Verify the rotation by triggering a real downstream call (test webhook, test alert, etc.).
- Revoke the old value at the provider (when applicable).
poindexter settings set <key> "<value>" --secret CLI (or direct DB UPDATE, or the set_secret Python helper) is your write surface. The poindexter settings get <key> command shows ******* (encrypted) for is_secret=true rows; pass poindexter settings get <key> --reveal to decrypt and print the plaintext (value to stdout, exposure warning to stderr) when you need to read one back.
Inventory — every known secret
Pulled fromapp_settings WHERE is_secret = TRUE plus the two bootstrap secrets. Refreshed 2026-05-08.
| Key | Where it lives | Used for | Rotate via |
|---|---|---|---|
database_url | bootstrap.toml | DB connection | § database_url |
POINDEXTER_SECRET_KEY | bootstrap.toml (env) | Encrypts every other secret | § POINDEXTER_SECRET_KEY |
<consumer>_oauth_client_secret | app_settings (encrypted) | OAuth 2.1 worker auth (per consumer) | § OAuth client secrets |
revalidate_secret | app_settings (encrypted) | Vercel ISR revalidation header | § revalidate_secret |
jwt_secret_key | app_settings (encrypted) | JWT signing | § jwt_secret_key |
secret_key | app_settings (encrypted) | App-wide signing | § secret_key |
openclaw_webhook_token | app_settings (encrypted) | OpenClaw → worker auth | § openclaw_webhook_token |
alertmanager_webhook_token | app_settings (encrypted) | Alertmanager → worker auth | § alertmanager_webhook_token |
telegram_bot_token | app_settings (encrypted) | Telegram bot API | § telegram_bot_token |
discord_bot_token | app_settings (encrypted) | Discord bot API | § discord_bot_token |
discord_voice_bot_token | app_settings (encrypted) | Discord voice bot | § discord_voice_bot_token |
discord_*_webhook | app_settings (encrypted) | Per-channel webhooks | § discord webhooks |
lemon_squeezy_webhook_secret | app_settings (encrypted) | Lemon Squeezy webhook HMAC | § lemon_squeezy_webhook_secret |
resend_api_key | app_settings (encrypted) | Newsletter email send | § resend_api_key |
resend_webhook_secret | app_settings (encrypted) | Resend webhook HMAC | § resend_webhook_secret |
smtp_password | app_settings (NOT yet encrypted as of 2026-04-30) | SMTP fallback | § smtp_password |
uptime_kuma_api_key | app_settings (encrypted) | Uptime Kuma push | § uptime_kuma_api_key |
openai_api_key | app_settings (encrypted) | OpenAI fallback | § openai_api_key |
anthropic_api_key | app_settings (encrypted) | Claude Haiku critic | § anthropic_api_key |
gemini_api_key | app_settings (encrypted) | Gemini fallback | § gemini_api_key |
pexels_api_key | app_settings (encrypted) | Stock image fallback | § pexels_api_key |
cloudinary_api_key + cloudinary_api_secret | app_settings (encrypted) | Image CDN | § cloudinary keys |
storage_secret_key + storage_token | app_settings (encrypted) | S3-compatible object storage (R2) | § storage keys |
redis_url | app_settings (encrypted) | Redis connection (with AUTH password) | § redis_url |
devto_api_key | app_settings (encrypted) | Dev.to cross-post | § devto_api_key |
notion_api_key | app_settings (encrypted) | Notion integration | § notion_api_key |
grafana_api_key + grafana_api_token | app_settings (encrypted) | Self-hosted Grafana HTTP API | § grafana keys |
google_api_key | app_settings (encrypted) | Google APIs | § google_api_key |
elevenlabs_api_key | app_settings (encrypted) | TTS provider | § elevenlabs_api_key |
Generic procedure (covers ~90% of the keys)
Most rotations follow this exact recipe:poindexter settings set --secret? It works, but it puts the secret value in your shell history and the click argv. set_secret(...) via the helper above keeps the secret in a 0600 file that you delete after.
Per-key rotation procedures
database_url
Lives in. ~/.poindexter/bootstrap.toml, key database_url.
When to rotate. Postgres password compromised, or you’re moving the DB to a new host.
Procedure.
POINDEXTER_SECRET_KEY
Lives in. ~/.poindexter/bootstrap.toml (env var name POINDEXTER_SECRET_KEY).
When to rotate. Annually, or after a suspected compromise. THIS IS THE DOOMSDAY KEY. Losing it without rotating means every encrypted app_settings row becomes garbage forever (see disaster-recovery.md CONFIG-2).
Procedure. Use the rotate_key helper from plugins.secrets — it decrypts every secret with the OLD key and re-encrypts with the NEW one in a single transaction.
OAuth client secrets
Lives in.app_settings.<consumer>_oauth_client_id /
app_settings.<consumer>_oauth_client_secret (encrypted, is_secret=true).
One pair per consumer (cli, mcp, mcp_gladlabs, brain, scripts,
openclaw).
Used for. OAuth 2.1 Client Credentials Grant — every consumer mints
short-lived JWTs by hitting POST /token with its client_id +
client_secret. The legacy static-Bearer plumbing
(POINDEXTER_KEY / GLADLABS_KEY / app_settings.api_token /
app_settings.api_auth_token) was removed in
Glad-Labs/poindexter#249 (2026-05-05).
Procedure. The migrate commands rotate the client_secret atomically and
push the new value into the consumer’s config file (~/.claude.json,
~/.openclaw/openclaw.json, ~/.poindexter/bootstrap.toml, etc.).
services/auth/oauth_client.py
and poindexter/cli/auth.py. The umbrella OAuth migration is umbrella issue
Glad-Labs/poindexter#241.
revalidate_secret
Lives in. app_settings.revalidate_secret (encrypted).
Used for. Worker → Vercel ISR revalidation. Sent as x-revalidate-secret header. The Vercel function compares against an env var on the Vercel side.
Procedure.
jwt_secret_key
Lives in. app_settings.jwt_secret_key (encrypted).
When to rotate. Suspected compromise, or after major code change to JWT issuance. Rotation invalidates every existing JWT — anyone holding one will need to re-auth.
Procedure. Generic procedure. Restart poindexter-worker so the new key is picked up before issuing the next JWT.
secret_key
Lives in. app_settings.secret_key (encrypted).
Used for. General-purpose app signing (CSRF tokens, signed cookies). TBD — needs operator confirmation: the exact callsites for this key aren’t fully audited; treat as jwt_secret_key-equivalent.
openclaw_webhook_token
Lives in. app_settings.openclaw_webhook_token (encrypted).
Used for. Worker → OpenClaw gateway auth (Authorization header).
Procedure.
alertmanager_webhook_token
Lives in. app_settings.alertmanager_webhook_token (encrypted).
Used for. Alertmanager → worker webhook auth.
Procedure. Generic. After updating the DB, also update infrastructure/prometheus/alertmanager.yml.tmpl (the matching value lives there; the rendered alertmanager.yml is regenerated by render_alertmanager_config.py) and restart Alertmanager:
telegram_bot_token
Lives in. app_settings.telegram_bot_token (encrypted) AND optionally bootstrap.toml (operator notification fallback).
When to rotate. Suspected leak (the token is in the URL path of every Telegram API call — it’s surprisingly leaky). Or @BotFather forced rotation.
Procedure.
discord_bot_token
Lives in. app_settings.discord_bot_token (encrypted).
Procedure.
discord_voice_bot_token
Lives in. app_settings.discord_voice_bot_token (encrypted).
Procedure. Same as discord_bot_token but for the voice-bot Discord application. Restart poindexter-voice-bot.
Discord webhooks
Keys:discord_ops_webhook_url, discord_alerts_webhook_url, etc. (per-channel webhook URLs that include a secret token in the path).
Procedure.
lemon_squeezy_webhook_secret
Lives in. app_settings.lemon_squeezy_webhook_secret (encrypted as of 2026-04-27).
Used for. HMAC verification of Lemon Squeezy webhooks (subscription created / cancelled / order paid).
Procedure.
resend_api_key
Lives in. app_settings.resend_api_key (encrypted).
Used for. Newsletter emails via Resend.
Procedure.
poindexter newsletter --help.
resend_webhook_secret
Lives in. app_settings.resend_webhook_secret (encrypted as of 2026-04-27).
Used for. HMAC verification of Resend delivery webhooks.
Procedure. Same shape as Lemon Squeezy. New secret comes from Resend dashboard → Webhooks → endpoint → Signing Secret.
smtp_password
Lives in. app_settings.smtp_password. Read via await site_config.get_secret(...) (the newsletter _cfg callsite), so set it with --secret to store it encrypted at rest.
Procedure.
uptime_kuma_api_key
Lives in. app_settings.uptime_kuma_api_key (encrypted; seeded in the baseline).
Used for. Pushing health beacons to Uptime Kuma.
Procedure.
openai_api_key
Lives in. app_settings.openai_api_key (encrypted).
Used for. OpenAI fallback in the model fallback chain (only used when explicitly enabled in app_settings AND gated by cost_guard).
Procedure.
cost_guard_enabled=true and openai_enabled=true only if you intend to actually use it. Per Matt’s policy, local Ollama is the default — paid providers are opt-in fallbacks ONLY.
anthropic_api_key
Lives in. app_settings.anthropic_api_key (encrypted).
Used for. Claude Haiku adversarial QA review — the multi-model QA rail atoms (qa.critic / qa.deepeval) that replaced the retired cross_model_qa stage (#355).
Procedure.
gemini_api_key
Lives in. app_settings.gemini_api_key (encrypted).
Used for. Gemini fallback. High risk — Matt previously spent $300 in one night via Gemini before cost guards. Keep cost_guard_enabled=true and gemini_enabled=false unless you actively need it.
Procedure. Generic. Same as OpenAI.
pexels_api_key
Lives in. app_settings.pexels_api_key (encrypted).
Used for. Stock image fallback when image-gen is degraded.
Procedure.
Cloudinary keys
Keys:cloudinary_api_key + cloudinary_api_secret (both encrypted).
Procedure.
Storage keys
Keys:storage_secret_key + storage_token (both encrypted). Provider-agnostic — works with R2, S3, B2, MinIO. Old cloudflare_r2_* keys are deprecated fallbacks.
Procedure.
redis_url
Lives in. app_settings.redis_url (encrypted — the URL contains the AUTH password).
Procedure.
RedisCache initialization in worker logs should not show “degraded to no-cache mode.”
devto_api_key
Procedure. https://dev.to/settings/extensions → API Keys → Generate new. Generic flow.
notion_api_key
Procedure. https://www.notion.so/my-integrations → select integration → Refresh / Reset. Generic flow.
Grafana keys
Keys:grafana_api_key + grafana_api_token (both encrypted; some duplication for legacy reasons).
Procedure.
google_api_key
Procedure. https://console.cloud.google.com → APIs & Services → Credentials → Regenerate. Generic flow. Note: this single key may grant access to multiple Google APIs depending on which APIs are enabled — review the API restrictions before rotating.
elevenlabs_api_key
Procedure. https://elevenlabs.io → Profile → API Keys → Generate / Revoke. Generic flow.
Re-seeding from scratch (after losing POINDEXTER_SECRET_KEY)
After a CONFIG-2 disaster recovery (lost encryption key), you have an empty app_settings for every secret. Walk through this checklist:
Verification — was the rotation actually applied?
Rotation cadence — calendar
TBD — needs operator to confirm cadence preferences:POINDEXTER_SECRET_KEY— annually (low risk if undisturbed; high blast radius if leaked)- OAuth client secrets,
revalidate_secret,openclaw_webhook_token— every 90 days - Bot tokens (Telegram, Discord) — only on suspected compromise (rotation invalidates active sessions)
- Paid API keys (OpenAI, Anthropic, Gemini, Resend, Cloudinary) — every 90 days, or on bill anomaly
- Webhook secrets (Lemon Squeezy, Resend) — only when the provider rotates them or on suspected compromise
- Cross-post adapter tokens (Dev.to) — yearly
/schedule skill).
See also
disaster-recovery.md— recovery from lost key (CONFIG-2)incident-response.md— alert routingsrc/cofounder_agent/plugins/secrets.py— encryption module referencesrc/cofounder_agent/poindexter/cli/auth.py— implementation ofpoindexter auth migrate-*