Disaster Recovery Runbook

Last reviewed: 2026-06-15 Audience: solo operator (Matt) at 2am during an incident Prereqs: Local PC online, Docker running, gh CLI authed, poindexter CLI installed, ~/.poindexter/bootstrap.toml accessible (or you have the database_url + POINDEXTER_SECRET_KEY somewhere safe) This runbook covers catastrophic-loss scenarios — the kind where “just restart the container” is not enough. For per-alert triage and routing, see incident-response.md. For known-pattern symptom debugging, see troubleshooting.md.

Scenarios this covers

DB-1. PostgreSQL container is healthy but data is corrupted / wrong / missing rows
DB-2. poindexter-postgres-local Docker volume was wiped or destroyed (full data loss)
DB-3. A migration ran half-way, left the schema in an inconsistent state
HOST-1. New machine — rebuild the entire local stack from scratch (laptop died, fresh OS install)
HOST-2. Docker Desktop / WSL2 corrupted — need to reset Docker but keep the data
CONFIG-1. bootstrap.toml lost — need to reconstruct it
CONFIG-2. POINDEXTER_SECRET_KEY lost — every encrypted secret in app_settings is unreadable

For per-service recovery (worker crashed, image-gen degraded, Vercel down), see the Per-service recovery section at the bottom.

Quick triage flowchart

Is the postgres container running?
  NO  -> Try DB-1 (restart)
  YES -> Can you connect with psql?
            NO  -> Container is broken: DB-1 / DB-2 (volume gone)
            YES -> Are the tables there? (check `app_settings`, `posts`, `pipeline_tasks`)
                     NO  -> DB-2 (volume wiped, fresh DB)
                     YES -> Does data look right? (post counts, recent rows)
                              NO  -> DB-1 (corruption, restore from backup)
                              YES -> Issue is downstream — check incident-response.md

Is `~/.poindexter/bootstrap.toml` present and readable?
  NO  -> CONFIG-1 (reconstruct bootstrap.toml)

Is `POINDEXTER_SECRET_KEY` set in env / bootstrap.toml?
  NO  -> CONFIG-2 (re-generate key + re-seed every secret)

Can you run `docker ps`?
  NO  -> HOST-2 (Docker broken)

Is this a brand new machine / fresh OS?
  YES -> HOST-1 (full rebuild)

Procedure: DB-1 — Restore PostgreSQL from backup

Symptoms. Container is running, you can connect, but data is wrong (missing rows, bad migration, accidental DELETE, etc.).

Step 1 — Stop the worker so it doesn’t fight you

docker stop poindexter-worker poindexter-brain-daemon

Step 2 — List available backups

Two backup tiers write under ~/.poindexter/backups/ — see backups.md for the full design:

In-stack container tier (primary). backup-hourly / backup-daily write ~/.poindexter/backups/auto/{hourly,daily}/poindexter_brain_*.dump (last 24 hourly + 7 daily).
DbBackupJob. The worker’s scheduled job runs scripts/db-backup-local.sh into the flat ~/.poindexter/backups/ (poindexter-db-*.dump, 14-day retention).

ls -lht ~/.poindexter/backups/auto/hourly/ ~/.poindexter/backups/auto/daily/ ~/.poindexter/backups/ 2>/dev/null | head -30
# Expected: poindexter_brain_2026-04-30T0312Z.dump (container tiers)
#       and/or poindexter-db-2026-04-30T0312Z.dump (DbBackupJob)

If every location is empty or stale, your backup automation is broken — go to DB-2 and treat this as a volume-loss event (your last good state is whatever’s in git/Vercel/R2). The restore steps below take whichever dump you picked here.

Step 3 — Verify the backup is restorable BEFORE you drop the live DB

pg_restore --list ~/.poindexter/backups/poindexter-db-<TIMESTAMP>.dump | head -20
# Expected: a list of TABLE / INDEX / SEQUENCE entries.
# If pg_restore errors, the backup is corrupt — try the next-oldest one.

Step 4 — Drop and recreate the DB

# Connect as superuser
docker exec -it poindexter-postgres-local psql -U poindexter -d postgres

# Inside psql:
DROP DATABASE poindexter_brain;
CREATE DATABASE poindexter_brain OWNER poindexter;
\q

Step 5 — Restore from the backup

docker exec -i poindexter-postgres-local pg_restore \
    -U poindexter \
    -d poindexter_brain \
    --no-owner --no-privileges \
    < ~/.poindexter/backups/poindexter-db-<TIMESTAMP>.dump

Expected output: a stream of pg_restore: processing data for table "..." lines, then exit 0. Some warnings about extensions (pgcrypto, vector) are normal — they’ll be re-created.

Step 6 — Re-apply any migrations newer than the backup

poindexter migrate status
# Compare the "applied" list against what's on disk.
poindexter migrate up

Step 7 — Bring the worker back

docker start poindexter-brain-daemon poindexter-worker
sleep 10
curl -s http://localhost:8002/api/health | python -m json.tool

Verification

# Row counts should be in the right ballpark for the backup vintage
docker exec poindexter-postgres-local psql -U poindexter -d poindexter_brain -c \
  "SELECT 'posts' AS tbl, COUNT(*) FROM posts UNION ALL
   SELECT 'app_settings', COUNT(*) FROM app_settings UNION ALL
   SELECT 'pipeline_tasks', COUNT(*) FROM pipeline_tasks;"

Procedure: DB-2 — `poindexter-postgres-local` data volume was wiped

Symptoms. docker ps shows postgres is up, but psql -c "\dt" returns “Did not find any relations” — the database exists but is empty. Or the container won’t even start because the volume mount is missing. Worst case — you lost the volume AND have no backup. The static export on R2 / public-site frontend still has the published posts. The git history has the schema migrations. Everything else is gone.

Step 1 — Confirm the volume is actually gone

docker volume ls | grep postgres
# Expected names: poindexter-postgres-data, postgres-local-data, etc.

docker volume inspect <volume_name> 2>&1 | head -10
# If "No such volume", it's gone.

Step 2 — Stop everything that talks to the DB

docker stop poindexter-worker poindexter-brain-daemon poindexter-grafana \
  poindexter-langfuse-web poindexter-langfuse-worker 2>/dev/null

Step 3 — IF you have a backup, restore it

Same as DB-1 Steps 4-7. If you got here because backups were missing too, continue.

Step 4 — IF no backup exists, rebuild the schema

# Ensure the container has a fresh empty volume
docker compose -f docker-compose.local.yml up -d postgres-local
sleep 5

# Run all migrations from scratch
poindexter migrate up
# Expected: "Applied N migration(s)"

Step 5 — Recreate the secrets that lived in `app_settings`

This is the painful part. Encrypted secrets are gone forever. You need to:

Go to secret-rotation.md and rotate every API key listed in the inventory (Telegram, Discord, OpenAI, Anthropic, Lemon Squeezy, Resend, Cloudinary, Pexels, etc.) — treat all of them as compromised because the encrypted blobs may have leaked with the volume backup.
Re-seed each one with poindexter settings set <key> "<value>" --secret (see secret-rotation.md “Re-seeding from scratch”).

Step 6 — Re-import published posts from R2

The static export on R2 (static/posts/index.json + static/posts/<slug>.json) is the source of truth for what was live. scripts/dr-reimport-posts-from-r2.py replays those JSON files back into the posts table.

# Dry-run first — validates the R2 fetch and logs what would be written
python scripts/dr-reimport-posts-from-r2.py --dry-run

# Then write (idempotent — safe to re-run)
python scripts/dr-reimport-posts-from-r2.py

The script resolves the database URL from bootstrap.toml automatically. Pass --database-url to override. Pass --r2-url if NEXT_PUBLIC_STATIC_URL env var is unset and the hardcoded default is stale. The upsert is keyed on slug so re-running fills gaps without overwriting rows already restored from a backup.

Note: pipeline_tasks rows and pipeline_versions rows are NOT re-imported (they live only in the DB). Only the published posts rows surface in the frontend, so the site is functional after this step. The missing task history means the auto-publish gate and edit-distance tracker start fresh — that’s acceptable for a DR scenario.

Verification

poindexter migrate status     # Should show all migrations applied.
docker exec poindexter-postgres-local psql -U poindexter -d poindexter_brain -c \
  "SELECT COUNT(*) FROM app_settings;"   # Should be > 0 after migration seeds run.

Procedure: DB-3 — Migration ran half-way, schema is inconsistent

Symptoms. A poindexter migrate up errored mid-run. poindexter migrate status shows a migration as “pending” but you can see partial DDL was already applied (e.g., a column exists that shouldn’t be there until that migration completes).

Step 1 — Capture the failure mode

poindexter migrate status > /tmp/migrate-status-before.txt
# Note the LAST applied migration name. The pending one(s) after it are the suspects.

Step 2 — Choose: roll forward or roll back?

Roll forward is preferred when:

The failed migration is small and you can eyeball what didn’t run
Re-running it (after manual cleanup) is safe (the DDL is idempotent or you can drop the half-applied object)

Roll back is preferred when:

The migration is complex
You have a recent backup and can afford to lose a few minutes of activity

Step 3a — Roll forward

# Inspect the migration file
cat src/cofounder_agent/services/migrations/<NNNN>_<name>.py

# Manually drop / undo whatever DDL the failed migration partially applied
docker exec -it poindexter-postgres-local psql -U poindexter -d poindexter_brain
# Inside psql, e.g.:
#   DROP TABLE IF EXISTS half_created_table;
#   ALTER TABLE existing_table DROP COLUMN IF EXISTS half_added_col;
# \q

# Re-run
poindexter migrate up

Step 3b — Roll back

# Roll back to the migration immediately BEFORE the broken one
poindexter migrate down --to <NNNN_previous_known_good>
# Confirms with prompt; pass --yes to skip.

# Verify
poindexter migrate status

If the failing migration has no down()/rollback_migration(), the down command will skip it. In that case, restore from backup (DB-1) to the last good state.

Step 4 — Bring services back up

docker start poindexter-worker poindexter-brain-daemon

Verification

poindexter migrate status   # All applied or all pending — no half-states.
curl -s http://localhost:8002/api/health | python -m json.tool

Procedure: DB-4 — Restore from the off-machine (Tier 2) restic repo

Scenario. The machine is gone — drive failure, theft, ransomware — so the in-stack ~/.poindexter/backups/auto/ dumps went with it. You configured Tier 2 (backups.md), so the daily dumps live in an S3-compatible bucket as an encrypted restic repo. This restores from there.

Hard dependency: the restic password. Tier 2’s password is stored encrypted in app_settings — which is gone in this scenario along with POINDEXTER_SECRET_KEY. You restore from the offline copy the setup wizard told you to save (password manager / fireproof safe). Without it the repo is unrecoverable — restic encryption with a lost password is final. If you don’t have it, this procedure cannot help; skip to DB-2 (rebuild from migrations + R2).

Step 1 — Install restic on the recovery machine

# Linux/macOS: package manager, or the static binary from
# https://github.com/restic/restic/releases
# Windows: `winget install restic.restic` or the release zip.
restic version

Step 2 — Point restic at the repo with your offline credentials

You need four values: the repo URL (the offsite_backup_repository you chose — s3:https://<endpoint>/<bucket>/<path>), the offline restic password, and the S3 access key id + secret. On a fresh machine app_settings is gone, so these come from your offline copy / the bucket provider’s console.

export RESTIC_REPOSITORY="s3:https://s3.us-west-002.backblazeb2.com/<bucket>/<path>"
export RESTIC_PASSWORD="<the password you saved offline>"
export AWS_ACCESS_KEY_ID="<s3 access key id>"
export AWS_SECRET_ACCESS_KEY="<s3 secret access key>"

Step 3 — Confirm the repo is reachable and list snapshots

restic snapshots
# Expected: one or more snapshots tagged `poindexter`, newest last.
# If this 403s, the key/password is wrong. If it hangs, check the endpoint.

Step 4 — Restore the latest snapshot to a local directory

mkdir -p ./offsite-restore
restic restore latest --target ./offsite-restore
# The Tier 1 daily dump lands under ./offsite-restore/daily/poindexter_brain_*.dump
ls -lht ./offsite-restore/daily/

Step 5 — `pg_restore` the recovered dump

Bring up an empty Postgres (HOST-1 Steps 1-5 if this is a fresh machine), then restore the recovered .dump exactly like DB-1 Step 5:

docker exec -i poindexter-postgres-local pg_restore \
    -U poindexter -d poindexter_brain --no-owner --no-privileges \
    < ./offsite-restore/daily/poindexter_brain_<TIMESTAMP>.dump

Then re-apply any newer migrations (DB-1 Step 6) and bring services up (DB-1 Step 7). Because the restored dump’s app_settings still holds the old encrypted secrets, you’ll also need the original POINDEXTER_SECRET_KEY to decrypt them — if that’s lost too, finish with CONFIG-2 (rotate + re-seed).

Verification

restic check                    # repo integrity (optional, slow for full)
docker exec poindexter-postgres-local psql -U poindexter -d poindexter_brain -c \
  "SELECT COUNT(*) FROM posts; SELECT COUNT(*) FROM app_settings;"

Procedure: HOST-1 — Rebuild the entire local stack on a new machine

Scenario. Laptop died, fresh OS install, or you’re moving to new hardware.

Step 1 — Install prerequisites

Tool	Why
Docker Desktop 4.26+ with WSL2 backend	Runs all containers
Git + Git Bash (Windows)	start-stack.sh uses bash
Python 3.13+	poindexter CLI
Node.js 22+	Public site dev/build
Ollama 0.1.40+	Local LLM inference
NVIDIA driver supporting CUDA 12.8+	image-gen + Wan video on RTX 5090

Step 2 — Clone the repo

git clone git@github.com:Glad-Labs/poindexter.git ~/glad-labs-stack
cd ~/glad-labs-stack

Step 3 — Restore secrets

You need the following from a safe place (1Password, encrypted USB, second machine):

~/.poindexter/bootstrap.toml — contains database_url and POINDEXTER_SECRET_KEY
A recent backup from ~/.poindexter/backups/ (or the cloud copy if you set one up)

mkdir -p ~/.poindexter/backups
cp /path/to/backup/bootstrap.toml ~/.poindexter/bootstrap.toml
chmod 600 ~/.poindexter/bootstrap.toml
cp /path/to/backup/poindexter-db-*.dump ~/.poindexter/backups/

If you don’t have either, skip ahead — we’ll generate a fresh bootstrap and rebuild from migrations + R2.

Step 4 — Install poindexter

pip install -e src/cofounder_agent
poindexter --help    # confirm install

Step 5 — Bring up the stack

bash scripts/start-stack.sh
# Expected: postgres-local, worker, brain-daemon, grafana, etc. all start.

docker ps --format "table {{.Names}}\t{{.Status}}"

Step 6 — Restore data

If you have a backup → follow DB-1 Steps 4-7 from “Drop and recreate the DB.” If no backup → run migrations to create empty schema, then go to DB-2 Steps 5-6.

Step 7 — Pull Ollama models

ollama pull nomic-embed-text
ollama pull gemma3:27b           # critic model
ollama pull qwen3:8b
ollama pull gemma-4-31B-it-qat:latest  # writer model (bakeoff winner 2026-06-18)
ollama pull glm-4.7-5090:latest        # reviser model (qa.rewrite rescue cycle)
# Confirm with:
ollama list

Step 8 — Verify everything

curl -s http://localhost:8002/api/health | python -m json.tool
curl -s http://localhost:3000/api/health
curl -s http://localhost:9836/health     # image-gen
curl -s http://localhost:11434/api/tags  # Ollama

Step 9 — Re-pair MCP / OpenClaw / Telegram / Discord

# Telegram pairing — use the /telegram:configure skill in Claude Code
# Discord pairing — use the /discord:configure skill in Claude Code
# OpenClaw — re-install via the existing setup script + paste tokens

Procedure: HOST-2 — Docker / WSL2 broken, keep the data

Symptoms. docker ps errors, Docker Desktop refuses to start, WSL2 is unresponsive. The data is fine — only the runtime is busted.

Step 1 — Try a soft reset first

Note. The Docker Engine Watchdog scheduled task (scripts/docker-watchdog.ps1, every 5 min) now performs this soft reset automatically. When the Docker Desktop process is alive but the engine is wedged (the WSL2-VM HCS_E_CONNECTION_TIMEOUT case), it re-checks after -WedgeConfirmSeconds (default 30) to rule out a transient blip, captures forensics to ~/.poindexter/logs/wedge-<timestamp>/ (host nvidia-smi, Hyper-V/vmcompute event channels, docker diagnose bundle), pings Telegram if telegram_bot_token is set in bootstrap.toml, then runs wsl --shutdown and restores the stack via start-stack.sh. The manual steps below are the fallback if the watchdog is disabled or the recycle doesn’t take.

# PowerShell, as admin
wsl --shutdown
Restart-Service com.docker.service -Force
# Wait 30s, try `docker ps` again

Step 2 — If that fails, reset Docker Desktop (preserves volumes by default)

In Docker Desktop GUI: Settings → Troubleshoot → “Reset to factory defaults” WARNING — this nukes named volumes. Use “Clean / Purge data” only if you have a current backup. The safer path: uninstall Docker Desktop, reinstall same version, restart. Named volumes survive a reinstall as long as you don’t delete the WSL2 distro docker-desktop-data.

Step 3 — Restart the stack

bash scripts/start-stack.sh
docker volume ls    # Confirm postgres volume still exists

Step 4 — Verify data integrity

docker exec poindexter-postgres-local psql -U poindexter -d poindexter_brain -c \
  "SELECT COUNT(*) FROM posts; SELECT COUNT(*) FROM app_settings;"
# Compare against what you remember the counts being.

If the counts are way off, treat as DB-2 (volume loss).

Procedure: CONFIG-1 — Lost `bootstrap.toml`

Symptoms. Worker won’t start. Logs say notify_operator(): no DATABASE_URL resolved, then sys.exit(2).

Step 1 — Reconstruct the file

mkdir -p ~/.poindexter
cat > ~/.poindexter/bootstrap.toml <<'EOF'
database_url = "postgresql://poindexter:<password>@localhost:5433/poindexter_brain"

# Operator notification fallbacks (optional — used when DB is unreachable)
telegram_bot_token = ""
telegram_chat_id = ""
discord_ops_webhook_url = ""

# Symmetric key for app_settings encrypted secrets — REQUIRED if you
# have any is_secret=true rows.
POINDEXTER_SECRET_KEY = "<existing key>"
EOF
chmod 600 ~/.poindexter/bootstrap.toml

You need the postgres password (find it: docker exec poindexter-postgres-local printenv POSTGRES_PASSWORD) and the POINDEXTER_SECRET_KEY (this is the killer — see CONFIG-2 if lost).

Step 2 — Restart worker

docker restart poindexter-worker
sleep 5
curl -s http://localhost:8002/api/health

Verification

# Worker should now start without sys.exit(2)
docker logs poindexter-worker 2>&1 | tail -20

Procedure: CONFIG-2 — Lost `POINDEXTER_SECRET_KEY`

Symptoms. Worker boots, but every get_secret(...) call raises SecretsError: Could not decrypt. Telegram/Discord/Vercel revalidation/Lemon Squeezy webhook verification all 401. app_settings rows with is_secret=true show enc:v1:... blobs that can never be decoded again. There is no recovery for the encrypted blobs. Encryption with AES-256 + a lost key is final. You must rotate every secret.

Step 1 — Generate a new key

python -c "import secrets; print(secrets.token_urlsafe(48))"
# Copy the output, save it RIGHT NOW (1Password / encrypted USB).

Step 2 — Update `bootstrap.toml`

# Edit ~/.poindexter/bootstrap.toml — set POINDEXTER_SECRET_KEY = "<new key>"
chmod 600 ~/.poindexter/bootstrap.toml

Step 3 — Wipe the dead encrypted rows

docker exec -it poindexter-postgres-local psql -U poindexter -d poindexter_brain

-- See what's gone
SELECT key, length(value) FROM app_settings
WHERE is_secret = TRUE AND value LIKE 'enc:v1:%';

-- Wipe them so they don't poison future reads
UPDATE app_settings SET value = '' WHERE is_secret = TRUE AND value LIKE 'enc:v1:%';

Step 4 — Restart the worker so it picks up the new key

docker restart poindexter-worker

Step 5 — Re-seed every secret

Follow secret-rotation.md — go through the inventory and rotate / re-add each one.

Verification

# No more enc:v1: blobs with empty values
docker exec poindexter-postgres-local psql -U poindexter -d poindexter_brain -c \
  "SELECT key FROM app_settings WHERE is_secret=TRUE AND value=''"
# Expected: empty after you've re-seeded all the secrets you actually use.

# Trigger a real downstream that uses a secret (e.g. Telegram alert).
# Mint a fresh JWT for the CLI client first — see `poindexter auth
# mint-token --client-id <pdx_xxx> --client-secret <secret>`.
curl -s -X POST http://localhost:8002/api/test/notify-operator \
  -H "Authorization: Bearer $JWT"
# Expected: Telegram message arrives.

Per-service recovery (subordinate playbooks)

For when a single service is down but the rest of the stack is fine. These were the original disaster-recovery entries — kept here as a quick reference.

Brain Daemon — no Telegram alerts, no `brain_decisions` rows

docker ps | grep brain-daemon
docker compose -f docker-compose.local.yml up -d brain-daemon
cat ~/.poindexter/heartbeat   # Should be < 5 min old

The container has restart: unless-stopped, so Docker relaunches it automatically on crash. (The legacy OS-level watchdog was retired when the brain was containerized.) If it’s up but the heartbeat is stale, it’s hung — docker restart poindexter-brain-daemon.

Content Worker (FastAPI) — pipeline stalled, `/api/health` fails

curl -s http://localhost:8002/api/health
docker compose -f docker-compose.local.yml up -d worker
# Or full stack: bash scripts/start-stack.sh

Ollama — content generation fails, “Ollama Unresponsive” alert

curl -s http://localhost:11434/api/tags
ollama serve   # If not running as service
ollama list    # Verify required models present

image-gen Server — posts publishing with Pexels stock images

curl -s http://localhost:9836/health | python -m json.tool
docker compose -f docker-compose.local.yml up -d image-gen-server
# If torch/torchvision mismatch:
docker exec poindexter-image-gen-server pip install --upgrade torchvision peft
docker restart poindexter-image-gen-server

Vercel — site returns 500 / blank page

curl -sf https://www.gladlabs.io
gh run list --repo Glad-Labs/poindexter --limit 3
gh run rerun <run_id> --repo Glad-Labs/poindexter

Grafana — dashboards down

docker compose -f docker-compose.local.yml up -d grafana
curl -s http://localhost:3000/api/health
# Dashboards are file-provisioned and reload on restart.

The brain daemon alerts via Telegram directly — Grafana down does NOT mute alerts.

OpenClaw Gateway — bot unreachable via Discord/Telegram

curl -s http://127.0.0.1:18789/
# Windows restart:
start "OpenClaw Gateway" cmd /d /c "%USERPROFILE%\.openclaw\gateway.cmd"

openclaw-watchdog.ps1 auto-restarts on health failure.

GPU metrics stale (“GPU Metrics Stale” alert)

curl -s http://localhost:9835/metrics | head -5
# Restart scraper (Windows):
taskkill /IM python.exe /FI "WINDOWTITLE eq nvidia-smi-exporter"
pythonw scripts/nvidia-smi-exporter.py

Backup hygiene — preventing the next disaster

Postgres is backed up automatically in tiers (see backups.md): the backup-hourly / backup-daily containers (24 hourly + 7 daily dumps under ~/.poindexter/backups/auto/) plus the worker’s DbBackupJob (scripts/db-backup-local.sh, flat 14-day dumps). The brain’s backup_watcher restarts a stalled tier before paging, and a nightly restore_test_probe proves a dump actually restores. If you don’t see fresh dumps every morning, that automation is broken — fix it before the next incident, not during.
bootstrap.toml should be in your password manager as a secure note. If it dies, you’re in CONFIG-1 + CONFIG-2 territory — minimum 1 hour of pain.
Enable off-machine (Tier 2) backups so a drive failure / theft / ransomware event doesn’t take your only copy with it. Run poindexter backup setup to ship the daily dumps to an S3-compatible bucket via restic (see backups.md); recover with DB-4 above. Save the restic password offline when the wizard prints it.
POINDEXTER_SECRET_KEY is the doomsday key. Print it on paper, put it in a fireproof safe. If you lose it AND the live app_settings table, you cannot recover the encrypted secrets — only re-issue them.

Contact

Operator: configured via app_settings.support_email / app_settings.privacy_email
Telegram: brain daemon sends alerts to configured chat_id
Discord: ops channel receives Grafana alerts

Backups Secret rotation

​Disaster Recovery Runbook

​Scenarios this covers

​Quick triage flowchart

​Procedure: DB-1 — Restore PostgreSQL from backup

​Step 1 — Stop the worker so it doesn’t fight you

​Step 2 — List available backups

​Step 3 — Verify the backup is restorable BEFORE you drop the live DB

​Step 4 — Drop and recreate the DB

​Step 5 — Restore from the backup

​Step 6 — Re-apply any migrations newer than the backup

​Step 7 — Bring the worker back

​Verification

​Procedure: DB-2 — poindexter-postgres-local data volume was wiped

​Step 1 — Confirm the volume is actually gone

​Step 2 — Stop everything that talks to the DB

​Step 3 — IF you have a backup, restore it

​Step 4 — IF no backup exists, rebuild the schema

​Step 5 — Recreate the secrets that lived in app_settings

​Step 6 — Re-import published posts from R2

​Verification

​Procedure: DB-3 — Migration ran half-way, schema is inconsistent

​Step 1 — Capture the failure mode

​Step 2 — Choose: roll forward or roll back?

​Step 3a — Roll forward

​Step 3b — Roll back

​Step 4 — Bring services back up

​Verification

​Procedure: DB-4 — Restore from the off-machine (Tier 2) restic repo

​Step 1 — Install restic on the recovery machine

​Step 2 — Point restic at the repo with your offline credentials

​Step 3 — Confirm the repo is reachable and list snapshots

​Step 4 — Restore the latest snapshot to a local directory

​Step 5 — pg_restore the recovered dump

​Verification

​Procedure: HOST-1 — Rebuild the entire local stack on a new machine

​Step 1 — Install prerequisites

​Step 2 — Clone the repo

​Step 3 — Restore secrets

​Step 4 — Install poindexter

​Step 5 — Bring up the stack

​Step 6 — Restore data

​Step 7 — Pull Ollama models

​Step 8 — Verify everything

​Step 9 — Re-pair MCP / OpenClaw / Telegram / Discord

​Procedure: HOST-2 — Docker / WSL2 broken, keep the data

​Step 1 — Try a soft reset first

​Step 2 — If that fails, reset Docker Desktop (preserves volumes by default)

​Step 3 — Restart the stack

​Step 4 — Verify data integrity

​Procedure: CONFIG-1 — Lost bootstrap.toml

​Step 1 — Reconstruct the file

​Step 2 — Restart worker

​Verification

​Procedure: CONFIG-2 — Lost POINDEXTER_SECRET_KEY

​Step 1 — Generate a new key

​Step 2 — Update bootstrap.toml

​Step 3 — Wipe the dead encrypted rows

​Step 4 — Restart the worker so it picks up the new key

​Step 5 — Re-seed every secret

​Verification

​Per-service recovery (subordinate playbooks)

​Brain Daemon — no Telegram alerts, no brain_decisions rows

​Content Worker (FastAPI) — pipeline stalled, /api/health fails

​Ollama — content generation fails, “Ollama Unresponsive” alert

​image-gen Server — posts publishing with Pexels stock images

​Vercel — site returns 500 / blank page

​Grafana — dashboards down

​OpenClaw Gateway — bot unreachable via Discord/Telegram

​GPU metrics stale (“GPU Metrics Stale” alert)

​Backup hygiene — preventing the next disaster

​See also

​Contact

Disaster Recovery Runbook

Scenarios this covers

Quick triage flowchart

Procedure: DB-1 — Restore PostgreSQL from backup

Step 1 — Stop the worker so it doesn’t fight you

Step 2 — List available backups

Step 3 — Verify the backup is restorable BEFORE you drop the live DB

Step 4 — Drop and recreate the DB

Step 5 — Restore from the backup

Step 6 — Re-apply any migrations newer than the backup

Step 7 — Bring the worker back

Verification

Procedure: DB-2 — `poindexter-postgres-local` data volume was wiped

Step 1 — Confirm the volume is actually gone

Step 2 — Stop everything that talks to the DB

Step 3 — IF you have a backup, restore it

Step 4 — IF no backup exists, rebuild the schema

Step 5 — Recreate the secrets that lived in `app_settings`

Step 6 — Re-import published posts from R2

Verification

Procedure: DB-3 — Migration ran half-way, schema is inconsistent

Step 1 — Capture the failure mode

Step 2 — Choose: roll forward or roll back?

Step 3a — Roll forward

Step 3b — Roll back

Step 4 — Bring services back up

Verification

Procedure: DB-4 — Restore from the off-machine (Tier 2) restic repo

Step 1 — Install restic on the recovery machine

Step 2 — Point restic at the repo with your offline credentials

Step 3 — Confirm the repo is reachable and list snapshots

Step 4 — Restore the latest snapshot to a local directory

Step 5 — `pg_restore` the recovered dump

Verification

Procedure: HOST-1 — Rebuild the entire local stack on a new machine

Step 1 — Install prerequisites

Step 2 — Clone the repo

Step 3 — Restore secrets

Step 4 — Install poindexter

Step 5 — Bring up the stack

Step 6 — Restore data

Step 7 — Pull Ollama models

Step 8 — Verify everything

Step 9 — Re-pair MCP / OpenClaw / Telegram / Discord

Procedure: HOST-2 — Docker / WSL2 broken, keep the data

Step 1 — Try a soft reset first

Step 2 — If that fails, reset Docker Desktop (preserves volumes by default)

Step 3 — Restart the stack

Step 4 — Verify data integrity

Procedure: CONFIG-1 — Lost `bootstrap.toml`

Step 1 — Reconstruct the file

Step 2 — Restart worker

Verification

Procedure: CONFIG-2 — Lost `POINDEXTER_SECRET_KEY`

Step 1 — Generate a new key

Step 2 — Update `bootstrap.toml`

Step 3 — Wipe the dead encrypted rows

Step 4 — Restart the worker so it picks up the new key

Step 5 — Re-seed every secret

Verification

Per-service recovery (subordinate playbooks)

Brain Daemon — no Telegram alerts, no `brain_decisions` rows

Content Worker (FastAPI) — pipeline stalled, `/api/health` fails

Ollama — content generation fails, “Ollama Unresponsive” alert

image-gen Server — posts publishing with Pexels stock images

Vercel — site returns 500 / blank page

Grafana — dashboards down

OpenClaw Gateway — bot unreachable via Discord/Telegram

GPU metrics stale (“GPU Metrics Stale” alert)

Backup hygiene — preventing the next disaster

See also

Contact