Skip to main content

Secret Rotation Runbook

Last reviewed: 2026-05-08 Audience: solo operator (Matt) at 2am during an incident, or doing scheduled rotation Prereqs: Local PC online, Docker running, gh CLI authed, poindexter CLI installed, ~/.poindexter/bootstrap.toml accessible, POINDEXTER_SECRET_KEY set Every secret in Poindexter has a rotation procedure. Most live in app_settings with is_secret=true and are encrypted at rest via pgcrypto — see src/cofounder_agent/plugins/secrets.py for how that works. Two secrets (database_url and POINDEXTER_SECRET_KEY itself) are bootstrap-only and live in ~/.poindexter/bootstrap.toml. This runbook lists every known secret, where to obtain a fresh value, how to set the new value, and what to restart to pick it up.

Scenarios this covers

  • Routine 90-day rotation of API keys
  • Emergency rotation after suspected compromise
  • Re-seeding after the encryption key was lost (see disaster-recovery.md CONFIG-2)
  • Updating a third-party token that the provider expired / regenerated
For initial provisioning, see local-development-setup.md. For incident triage, see incident-response.md.

Quick triage flowchart


How rotation works (quick model)

Every encrypted secret follows this lifecycle:
  1. Generate a new value (via the provider’s UI / API, or openssl rand)
  2. Set the new value into app_settings via poindexter settings set <key> "<value>" --secret or the set_secret(...) Python helper. Either path encrypts under the current POINDEXTER_SECRET_KEY.
  3. Restart any in-process consumers so they re-fetch from app_settings. Most async consumers call await site_config.get_secret(...) per-request and don’t need a restart; some sync constructors cache at startup and DO.
  4. Verify the rotation by triggering a real downstream call (test webhook, test alert, etc.).
  5. Revoke the old value at the provider (when applicable).
The poindexter settings set <key> "<value>" --secret CLI (or direct DB UPDATE, or the set_secret Python helper) is your write surface. The poindexter settings get <key> command shows ******* (encrypted) for is_secret=true rows; pass poindexter settings get <key> --reveal to decrypt and print the plaintext (value to stdout, exposure warning to stderr) when you need to read one back.

Inventory — every known secret

Pulled from app_settings WHERE is_secret = TRUE plus the two bootstrap secrets. Refreshed 2026-05-08.
KeyWhere it livesUsed forRotate via
database_urlbootstrap.tomlDB connection§ database_url
POINDEXTER_SECRET_KEYbootstrap.toml (env)Encrypts every other secret§ POINDEXTER_SECRET_KEY
<consumer>_oauth_client_secretapp_settings (encrypted)OAuth 2.1 worker auth (per consumer)§ OAuth client secrets
revalidate_secretapp_settings (encrypted)Vercel ISR revalidation header§ revalidate_secret
jwt_secret_keyapp_settings (encrypted)JWT signing§ jwt_secret_key
secret_keyapp_settings (encrypted)App-wide signing§ secret_key
openclaw_webhook_tokenapp_settings (encrypted)OpenClaw → worker auth§ openclaw_webhook_token
alertmanager_webhook_tokenapp_settings (encrypted)Alertmanager → worker auth§ alertmanager_webhook_token
telegram_bot_tokenapp_settings (encrypted)Telegram bot API§ telegram_bot_token
discord_bot_tokenapp_settings (encrypted)Discord bot API§ discord_bot_token
discord_voice_bot_tokenapp_settings (encrypted)Discord voice bot§ discord_voice_bot_token
discord_*_webhookapp_settings (encrypted)Per-channel webhooks§ discord webhooks
lemon_squeezy_webhook_secretapp_settings (encrypted)Lemon Squeezy webhook HMAC§ lemon_squeezy_webhook_secret
resend_api_keyapp_settings (encrypted)Newsletter email send§ resend_api_key
resend_webhook_secretapp_settings (encrypted)Resend webhook HMAC§ resend_webhook_secret
smtp_passwordapp_settings (NOT yet encrypted as of 2026-04-30)SMTP fallback§ smtp_password
uptime_kuma_api_keyapp_settings (encrypted)Uptime Kuma push§ uptime_kuma_api_key
openai_api_keyapp_settings (encrypted)OpenAI fallback§ openai_api_key
anthropic_api_keyapp_settings (encrypted)Claude Haiku critic§ anthropic_api_key
gemini_api_keyapp_settings (encrypted)Gemini fallback§ gemini_api_key
pexels_api_keyapp_settings (encrypted)Stock image fallback§ pexels_api_key
cloudinary_api_key + cloudinary_api_secretapp_settings (encrypted)Image CDN§ cloudinary keys
storage_secret_key + storage_tokenapp_settings (encrypted)S3-compatible object storage (R2)§ storage keys
redis_urlapp_settings (encrypted)Redis connection (with AUTH password)§ redis_url
devto_api_keyapp_settings (encrypted)Dev.to cross-post§ devto_api_key
notion_api_keyapp_settings (encrypted)Notion integration§ notion_api_key
grafana_api_key + grafana_api_tokenapp_settings (encrypted)Self-hosted Grafana HTTP API§ grafana keys
google_api_keyapp_settings (encrypted)Google APIs§ google_api_key
elevenlabs_api_keyapp_settings (encrypted)TTS provider§ elevenlabs_api_key

Generic procedure (covers ~90% of the keys)

Most rotations follow this exact recipe:
Why not always poindexter settings set --secret? It works, but it puts the secret value in your shell history and the click argv. set_secret(...) via the helper above keeps the secret in a 0600 file that you delete after.

Per-key rotation procedures

database_url

Lives in. ~/.poindexter/bootstrap.toml, key database_url. When to rotate. Postgres password compromised, or you’re moving the DB to a new host. Procedure.
Verify.

POINDEXTER_SECRET_KEY

Lives in. ~/.poindexter/bootstrap.toml (env var name POINDEXTER_SECRET_KEY). When to rotate. Annually, or after a suspected compromise. THIS IS THE DOOMSDAY KEY. Losing it without rotating means every encrypted app_settings row becomes garbage forever (see disaster-recovery.md CONFIG-2). Procedure. Use the rotate_key helper from plugins.secrets — it decrypts every secret with the OLD key and re-encrypts with the NEW one in a single transaction.
Verify.
Revoke. N/A — the old key isn’t held anywhere external. Just remove it from your password manager once you’ve confirmed the rotation took.

OAuth client secrets

Lives in. app_settings.<consumer>_oauth_client_id / app_settings.<consumer>_oauth_client_secret (encrypted, is_secret=true). One pair per consumer (cli, mcp, mcp_gladlabs, brain, scripts, openclaw). Used for. OAuth 2.1 Client Credentials Grant — every consumer mints short-lived JWTs by hitting POST /token with its client_id + client_secret. The legacy static-Bearer plumbing (POINDEXTER_KEY / GLADLABS_KEY / app_settings.api_token / app_settings.api_auth_token) was removed in Glad-Labs/poindexter#249 (2026-05-05). Procedure. The migrate commands rotate the client_secret atomically and push the new value into the consumer’s config file (~/.claude.json, ~/.openclaw/openclaw.json, ~/.poindexter/bootstrap.toml, etc.).
Verify.
Reference. Consumer wiring lives in services/auth/oauth_client.py and poindexter/cli/auth.py. The umbrella OAuth migration is umbrella issue Glad-Labs/poindexter#241.

revalidate_secret

Lives in. app_settings.revalidate_secret (encrypted). Used for. Worker → Vercel ISR revalidation. Sent as x-revalidate-secret header. The Vercel function compares against an env var on the Vercel side. Procedure.
Verify.

jwt_secret_key

Lives in. app_settings.jwt_secret_key (encrypted). When to rotate. Suspected compromise, or after major code change to JWT issuance. Rotation invalidates every existing JWT — anyone holding one will need to re-auth. Procedure. Generic procedure. Restart poindexter-worker so the new key is picked up before issuing the next JWT.

secret_key

Lives in. app_settings.secret_key (encrypted). Used for. General-purpose app signing (CSRF tokens, signed cookies). TBD — needs operator confirmation: the exact callsites for this key aren’t fully audited; treat as jwt_secret_key-equivalent.

openclaw_webhook_token

Lives in. app_settings.openclaw_webhook_token (encrypted). Used for. Worker → OpenClaw gateway auth (Authorization header). Procedure.

alertmanager_webhook_token

Lives in. app_settings.alertmanager_webhook_token (encrypted). Used for. Alertmanager → worker webhook auth. Procedure. Generic. After updating the DB, also update infrastructure/prometheus/alertmanager.yml.tmpl (the matching value lives there; the rendered alertmanager.yml is regenerated by render_alertmanager_config.py) and restart Alertmanager:

telegram_bot_token

Lives in. app_settings.telegram_bot_token (encrypted) AND optionally bootstrap.toml (operator notification fallback). When to rotate. Suspected leak (the token is in the URL path of every Telegram API call — it’s surprisingly leaky). Or @BotFather forced rotation. Procedure.
Verify.

discord_bot_token

Lives in. app_settings.discord_bot_token (encrypted). Procedure.

discord_voice_bot_token

Lives in. app_settings.discord_voice_bot_token (encrypted). Procedure. Same as discord_bot_token but for the voice-bot Discord application. Restart poindexter-voice-bot.

Discord webhooks

Keys: discord_ops_webhook_url, discord_alerts_webhook_url, etc. (per-channel webhook URLs that include a secret token in the path). Procedure.

lemon_squeezy_webhook_secret

Lives in. app_settings.lemon_squeezy_webhook_secret (encrypted as of 2026-04-27). Used for. HMAC verification of Lemon Squeezy webhooks (subscription created / cancelled / order paid). Procedure.
Verify. Trigger a test webhook from Lemon Squeezy dashboard. Worker logs should show a 200 response, not 401.

resend_api_key

Lives in. app_settings.resend_api_key (encrypted). Used for. Newsletter emails via Resend. Procedure.
Verify. Send a test newsletter to one address — see poindexter newsletter --help.

resend_webhook_secret

Lives in. app_settings.resend_webhook_secret (encrypted as of 2026-04-27). Used for. HMAC verification of Resend delivery webhooks. Procedure. Same shape as Lemon Squeezy. New secret comes from Resend dashboard → Webhooks → endpoint → Signing Secret.

smtp_password

Lives in. app_settings.smtp_password. Read via await site_config.get_secret(...) (the newsletter _cfg callsite), so set it with --secret to store it encrypted at rest. Procedure.

uptime_kuma_api_key

Lives in. app_settings.uptime_kuma_api_key (encrypted; seeded in the baseline). Used for. Pushing health beacons to Uptime Kuma. Procedure.
Verify. Watch Uptime Kuma — heartbeats should resume within 60 seconds.

openai_api_key

Lives in. app_settings.openai_api_key (encrypted). Used for. OpenAI fallback in the model fallback chain (only used when explicitly enabled in app_settings AND gated by cost_guard). Procedure.
Cost guardrails. Confirm cost_guard_enabled=true and openai_enabled=true only if you intend to actually use it. Per Matt’s policy, local Ollama is the default — paid providers are opt-in fallbacks ONLY.

anthropic_api_key

Lives in. app_settings.anthropic_api_key (encrypted). Used for. Claude Haiku adversarial QA review — the multi-model QA rail atoms (qa.critic / qa.deepeval) that replaced the retired cross_model_qa stage (#355). Procedure.
Verify. Run a content task end-to-end; check that a QA rail atom logs a Claude Haiku call rather than skipping.

gemini_api_key

Lives in. app_settings.gemini_api_key (encrypted). Used for. Gemini fallback. High risk — Matt previously spent $300 in one night via Gemini before cost guards. Keep cost_guard_enabled=true and gemini_enabled=false unless you actively need it. Procedure. Generic. Same as OpenAI.

pexels_api_key

Lives in. app_settings.pexels_api_key (encrypted). Used for. Stock image fallback when image-gen is degraded. Procedure.

Cloudinary keys

Keys: cloudinary_api_key + cloudinary_api_secret (both encrypted). Procedure.
Verify. Trigger a Cloudinary-backed image upload and watch worker logs for Cloudinary 200 responses.

Storage keys

Keys: storage_secret_key + storage_token (both encrypted). Provider-agnostic — works with R2, S3, B2, MinIO. Old cloudflare_r2_* keys are deprecated fallbacks. Procedure.
Verify. Trigger a static export rebuild — watch for object writes to land in the bucket.

redis_url

Lives in. app_settings.redis_url (encrypted — the URL contains the AUTH password). Procedure.
Verify. RedisCache initialization in worker logs should not show “degraded to no-cache mode.”

devto_api_key

Procedure. https://dev.to/settings/extensions → API Keys → Generate new. Generic flow.

notion_api_key

Procedure. https://www.notion.so/my-integrations → select integration → Refresh / Reset. Generic flow.

Grafana keys

Keys: grafana_api_key + grafana_api_token (both encrypted; some duplication for legacy reasons). Procedure.

google_api_key

Procedure. https://console.cloud.google.com → APIs & Services → Credentials → Regenerate. Generic flow. Note: this single key may grant access to multiple Google APIs depending on which APIs are enabled — review the API restrictions before rotating.

elevenlabs_api_key

Procedure. https://elevenlabs.io → Profile → API Keys → Generate / Revoke. Generic flow.

Re-seeding from scratch (after losing POINDEXTER_SECRET_KEY)

After a CONFIG-2 disaster recovery (lost encryption key), you have an empty app_settings for every secret. Walk through this checklist:
For each one: go to the section above, generate a new value at the provider, set it in DB, restart the consumer.

Verification — was the rotation actually applied?


Rotation cadence — calendar

TBD — needs operator to confirm cadence preferences:
  • POINDEXTER_SECRET_KEY — annually (low risk if undisturbed; high blast radius if leaked)
  • OAuth client secrets, revalidate_secret, openclaw_webhook_token — every 90 days
  • Bot tokens (Telegram, Discord) — only on suspected compromise (rotation invalidates active sessions)
  • Paid API keys (OpenAI, Anthropic, Gemini, Resend, Cloudinary) — every 90 days, or on bill anomaly
  • Webhook secrets (Lemon Squeezy, Resend) — only when the provider rotates them or on suspected compromise
  • Cross-post adapter tokens (Dev.to) — yearly
A scheduled agent should be set up to remind on this cadence (see /schedule skill).

See also

  • disaster-recovery.md — recovery from lost key (CONFIG-2)
  • incident-response.md — alert routing
  • src/cofounder_agent/plugins/secrets.py — encryption module reference
  • src/cofounder_agent/poindexter/cli/auth.py — implementation of poindexter auth migrate-*