Incident Response Runbook

Last reviewed: 2026-04-30 Audience: solo operator (Matt) at 2am during an incident Prereqs: Local PC online, Docker running, gh CLI authed, poindexter CLI installed, Grafana access When an alert fires (Telegram or Discord), this is the playbook. Find the alert by name in the index, jump to its section, follow the steps. If something doesn’t match a known alert, fall through to Unknown alert — generic triage.

Scenarios this covers

A Telegram or Discord notification just fired and you need to know what to do
You woke up to multiple alerts and need to triage by severity
A symptom is happening but no alert fired (Grafana / brain daemon may be down)
You need to escalate / mute an alert during planned maintenance

For catastrophic-loss scenarios (lost DB volume, lost secret key), see disaster-recovery.md. For known-pattern symptoms, see troubleshooting.md. For routine secret rotation, see secret-rotation.md.

Quick triage flowchart

Alert fired in the last 5 minutes?
  YES -> Check the alert NAME first. Jump to that section below.
  NO  -> Is the issue still happening?
           YES -> Treat as a fresh symptom; check troubleshooting.md by symptom.
           NO  -> Add a postmortem entry to troubleshooting.md and move on.

Multiple alerts at once?
  -> Triage by severity:
     1. critical   -> Worker offline / brain daemon stale / GPU temp high
     2. data loss  -> DB size growing weird / disk space low
     3. warning    -> Stuck tasks, embedding lag, cost spike, traffic anomaly
     4. cosmetic   -> Pipeline stalled (no new tasks) / quality drop / approval rate

Telegram silent but you SEE a problem?
  -> Brain daemon may be dead.
     Check: cat ~/.poindexter/heartbeat
     Then:  docker ps | grep brain-daemon
     Jump to "Brain Daemon Stale" alert section.

Alert index — alert name to runbook section

These are the active Grafana alert rules in infrastructure/grafana/provisioning/alerting/alert-rules.yml (15 rules as of 2026-06-13). Brain daemon also fires a few synthetic alerts directly to Telegram, and host-infrastructure alerts (e.g. disk space) are sourced from Prometheus (infrastructure/prometheus/alerts/). No-data posture (Glad-Labs/poindexter#581). 14 of the 15 rules use noDataState: Alerting: every page-worthy rule returns a row in the healthy state (a count(*) / SUM / AVG always does; GPU Temperature High reads the latest gpu_metrics row), so “no data” can only mean the underlying table/view was renamed or dropped, or the datasource is down — i.e. the rule went blind. Those surface loudly rather than silently resolving green (fail-loud, no silent fallbacks). Query errors (renamed column, dropped table) surface independently via execErrState, which is Error (or Alerting) on every rule — never OK. The single exception is DB Size Warning, intentionally kept on noDataState: OK: it is a non-page-worthy capacity warning whose only no-data condition (datasource unreachable) is already paged by the two critical rules on the same datasource (Worker Offline, Brain Daemon Stale). The posture is pinned by src/cofounder_agent/tests/unit/infrastructure/test_grafana_alert_no_data_state.py.

Alert	Severity	Section
Worker Offline	critical	§ Worker Offline
Brain Daemon Stale	critical	§ Brain Daemon Stale
GPU Temperature High	critical	§ GPU Temperature High
High Error Rate	critical	§ High Error Rate
Stale Tasks	warning	§ Stale Tasks
Embedding Sync Lag	warning	§ Embedding Sync Lag
DB Size Warning	warning	§ DB Size Warning
Daily Cost Spike	warning	§ Daily Cost Spike
Content Quality Drop	warning	§ Content Quality Drop
Traffic Anomaly	warning	§ Traffic Anomaly
Page-View Capture Dead	warning	§ Page-View Capture Dead
Pipeline Stalled	warning	§ Pipeline Stalled
Ollama Unresponsive	warning	§ Ollama Unresponsive
Zero Published Posts This Week	warning	§ Zero Published Posts This Week
GPU Metrics Stale	warning	§ GPU Metrics Stale
Disk Space Low (Prometheus)	warning	§ Disk Space Low
Site DOWN (brain)	critical	§ Site DOWN
Wan Server DEGRADED	warning	§ Wan Server DEGRADED
(Unknown alert)	varies	§ Unknown alert — generic triage

Worker Offline

Means. No brain_decisions row in the last 10 minutes AND no pipeline_tasks activity in the last 30 minutes. The worker is probably crashed. Triage.

docker ps --filter name=poindexter-worker --format "{{.Status}}"
# If "Restarting" or empty -> container is down
# If "Up X minutes" -> worker is alive but not processing

Fix.

# Restart
docker compose -f docker-compose.local.yml up -d worker

# Watch boot logs
docker logs -f poindexter-worker 2>&1 | head -40
# Look for: "Application startup complete." (FastAPI boot)
# Look for: "[flows.content_generation] cycle starting" (worker actually
# dispatching tasks via the Prefect flow — task_executor.py was deleted
# in the 2026-05-16 Stage 4 cutover).
# Cross-check Prefect itself:
#   docker logs poindexter-worker | grep -i prefect

# Verify
curl -s http://localhost:8002/api/health | python -m json.tool

If the container won’t start (crashloop), see troubleshooting.md “Test suite fails in the worker container” or check the logs for a Python traceback. Most common: a recent migration added a column that the deployed code doesn’t expect — roll the migration back or pull/rebuild. Escalation. If the container is healthy but tasks still aren’t processing, jump to Stale Tasks below.

Brain Daemon Stale

Means. No brain_decisions rows in the last 15 minutes. The 5-minute brain cycle has stopped. Triage.

cat ~/.poindexter/heartbeat
# Should be a unix timestamp from < 5 minutes ago.

docker ps --filter name=poindexter-brain-daemon --format "{{.Status}}"

Fix.

docker compose -f docker-compose.local.yml up -d brain-daemon
docker logs -f poindexter-brain-daemon 2>&1 | head -20

The brain runs as the poindexter-brain-daemon container with restart: unless-stopped, so Docker relaunches it automatically if the process crashes. (The legacy OS-level watchdog — the Task Scheduler “Poindexter Brain Watchdog” — predates containerization and was disabled 2026-06-09; Docker’s restart policy is the recovery path now.) If the container is up but the cycle is still stale, it’s hung rather than crashed — force a clean cycle:

docker restart poindexter-brain-daemon

Escalation. If brain restarts but immediately exits, look at the logs for a DB connection error — likely a DATABASE_URL problem. See disaster-recovery.md CONFIG-1.

GPU Temperature High

Means. The latest gpu_metrics row has temperature > 85°C. Alert is stable for 5 min. Triage.

nvidia-smi --query-gpu=temperature.gpu,utilization.gpu,memory.used --format=csv

# What's hammering the GPU right now?
docker stats --no-stream poindexter-image-gen-server poindexter-wan-server poindexter-worker

Fix.

If it’s a runaway pipeline: pause it.

poindexter settings set pipeline_paused true

If it’s an image-gen or Wan render: stop the in-flight job.

docker restart poindexter-image-gen-server   # nuclear; cancels any in-flight render
docker restart poindexter-wan-server

If it’s a third-party workload (gaming, a stray process): kill that process or accept the temperature.
If GPU temp stays high after killing all GPU loads: physical cooling problem. Check fan curve, intake/exhaust airflow, ambient temp.

Verification.

# Wait 60s, re-check
nvidia-smi --query-gpu=temperature.gpu --format=csv
# Should drop below 80°C within 1 minute of removing the load.

Escalation. Sustained > 90°C is hardware damage territory. Power-off the workstation if it stays there.

High Error Rate

Means. More than 5 audit_log rows with severity='error' in the last hour. Triage.

-- What kinds of errors?
docker exec poindexter-postgres-local psql -U poindexter -d poindexter_brain -c \
  "SELECT category, COUNT(*) AS n, MAX(timestamp) AS latest
   FROM audit_log
   WHERE severity = 'error' AND timestamp > NOW() - INTERVAL '1 hour'
   GROUP BY category ORDER BY n DESC LIMIT 10;"

-- Top 5 most recent error messages
docker exec poindexter-postgres-local psql -U poindexter -d poindexter_brain -c \
  "SELECT timestamp, category, LEFT(message, 100) AS msg
   FROM audit_log
   WHERE severity = 'error' AND timestamp > NOW() - INTERVAL '1 hour'
   ORDER BY timestamp DESC LIMIT 5;"

Fix. Depends on the category. Common ones:

category='ollama' → see § Ollama Unresponsive
category='pipeline' → see Stale Tasks / troubleshooting.md
category='webhook' → likely a 4xx from a third-party (Vercel, Telegram, Discord) — check secret rotation

Stale Tasks

Means. One or more pipeline_tasks rows have status='in_progress' and updated_at older than 2 hours. Triage.

docker exec poindexter-postgres-local psql -U poindexter -d poindexter_brain -c \
  "SELECT task_id, LEFT(topic, 50) AS topic, updated_at, NOW() - updated_at AS stuck_for
   FROM pipeline_tasks
   WHERE status = 'in_progress' AND updated_at < NOW() - INTERVAL '2 hours'
   ORDER BY updated_at;"

Fix.

-- Clear the stuck rows (they'll fall out of the alert)
docker exec poindexter-postgres-local psql -U poindexter -d poindexter_brain -c \
  "UPDATE pipeline_tasks
   SET status='failed', error_message='manually cleared — stuck in_progress'
   WHERE status='in_progress' AND updated_at < NOW() - INTERVAL '2 hours';"

Then check what hung — see troubleshooting.md “Pipeline task stuck in_progress for more than 10 minutes.”

Embedding Sync Lag

Means. Newest row in embeddings is more than 6 hours old. Auto-embed daemon is probably dead. Triage + fix.

docker ps --filter name=poindexter-auto-embed --format "{{.Status}}"
docker compose -f docker-compose.local.yml up -d auto-embed
docker logs -f poindexter-auto-embed 2>&1 | head -20

If the daemon is alive but embeddings still aren’t progressing, check Ollama (the embedding model is nomic-embed-text, served by Ollama). See § Ollama Unresponsive.

DB Size Warning

Means. poindexter_brain is now > 1 GB. Triage.

docker exec poindexter-postgres-local psql -U poindexter -d poindexter_brain -c \
  "SELECT relname AS table, pg_size_pretty(pg_total_relation_size(relid)) AS size
   FROM pg_stat_user_tables
   ORDER BY pg_total_relation_size(relid) DESC LIMIT 10;"

Fix. Usual suspects:

audit_log — runs forever by default. Add a retention policy:

DELETE FROM audit_log WHERE timestamp < NOW() - INTERVAL '30 days';

embeddings — high churn. Check whether old post embeddings are being re-generated unnecessarily.
pipeline_tasks — old failed rows accumulate. See poindexter retention --help for archival commands.

Escalation. Sustained growth > 10 GB needs a long-term plan — partition the high-churn tables, archive cold rows to a separate DB.

Daily Cost Spike

Means. cost_logs SUM in the last 24h exceeds $5. Triage.

docker exec poindexter-postgres-local psql -U poindexter -d poindexter_brain -c \
  "SELECT provider, model, SUM(cost_usd) AS spend, COUNT(*) AS calls
   FROM cost_logs WHERE created_at > NOW() - INTERVAL '24 hours'
   GROUP BY provider, model ORDER BY spend DESC;"

Fix. If a paid provider is spiking:

# Disable the paid model — fallback chain takes over
poindexter settings set <provider>_enabled false

# Or tighten the daily budget so cost_guard kills further calls
poindexter settings set cost_guard_daily_limit_usd 1.0

For the budget to actually enforce, the cost guard must be enabled — verify with poindexter settings get cost_guard_enabled. Escalation.

5/day =

150/month. If this is sustained without commensurate revenue, kill the paid provider entirely (poindexter settings set <provider>_enabled false) and run on local Ollama only.

Content Quality Drop

Means. 7-day average quality_score is below 70. Pipeline is producing weaker content. Triage.

-- What's the current writer model?
docker exec poindexter-postgres-local psql -U poindexter -d poindexter_brain -c \
  "SELECT key, value, updated_at FROM app_settings WHERE key='pipeline_writer_model';"

-- Score by day
docker exec poindexter-postgres-local psql -U poindexter -d poindexter_brain -c \
  "SELECT DATE(updated_at) AS d, ROUND(AVG(quality_score)::numeric, 1) AS avg_q, COUNT(*) AS n
   FROM pipeline_tasks WHERE quality_score IS NOT NULL AND updated_at > NOW() - INTERVAL '14 days'
   GROUP BY d ORDER BY d DESC;"

Fix. Most common cause — pipeline_writer_model was flipped off the intended model. See troubleshooting.md “Approval rate drops to ~0%” entry.

-- Restore intended writer (gemma-4-31B-it-qat — bakeoff winner 2026-06-18, #1692)
UPDATE app_settings SET value = 'ollama/gemma-4-31B-it-qat:latest', updated_at = NOW()
WHERE key = 'pipeline_writer_model';

Traffic Anomaly

Means. Today’s page_views count is less than 50% of yesterday’s. Possible site issue. Triage.

# Is the site even up?
curl -sf https://www.gladlabs.io >/dev/null && echo "site OK" || echo "SITE DOWN"

# Did Vercel deploy something broken in the last 24h?
gh run list --repo Glad-Labs/poindexter --limit 5

Fix. If site is down → see § Site DOWN. If site is up but traffic still dropped:

Check Search Console for new manual penalties or indexing drops
Check Google Analytics for referrer changes
Check whether ViewTracker beacon is firing — open the site in a browser, watch the Network tab for the sendBeacon POST to the Cloudflare Worker (NEXT_PUBLIC_BEACON_URL). If it’s missing, the analytics beacon is broken (frontend issue, not a real traffic drop) — see § Page-View Capture Dead.

Page-View Capture Dead

Means. No fresh page_views rows — the own-analytics capture chain has stalled somewhere between the frontend beacon and the DB. Triage.

docker exec poindexter-postgres-local psql -U poindexter -d poindexter_brain -c \
  "SELECT MAX(created_at) AS newest, NOW() - MAX(created_at) AS age FROM page_views;"

Fix. Walk the chain front to back:

Beacon not firing (frontend): open a post in a browser, watch the Network tab for the sendBeacon POST to the Cloudflare Worker (NEXT_PUBLIC_BEACON_URL). If it’s missing, the frontend regressed — check web/public-site/components/ViewTracker.tsx and that NEXT_PUBLIC_BEACON_URL is set in Vercel.
Sync job stalled (backend): the worker imports Cloudflare Analytics Engine into page_views every 5 minutes via services/jobs/sync_cloudflare_analytics.py. Confirm the worker is up and that job is scheduled.
Real traffic drop, not a capture failure: cross-check Search Console / GA — if they dropped too, treat it as § Traffic Anomaly, not a capture bug.

Pipeline Stalled

Means. Zero new pipeline_tasks rows in the last 48 hours. Triage.

# Is the worker alive?
curl -s http://localhost:8002/api/health | python -m json.tool

# Is anything blocking new tasks?
docker exec poindexter-postgres-local psql -U poindexter -d poindexter_brain -c \
  "SELECT key, value FROM app_settings
   WHERE key IN ('pipeline_paused', 'max_approval_queue', 'pipeline_paused_for_gaming');"

# Approval queue full?
docker exec poindexter-postgres-local psql -U poindexter -d poindexter_brain -c \
  "SELECT COUNT(*) FROM pipeline_tasks WHERE status='awaiting_approval';"

Fix.

If pipeline_paused=true → poindexter settings set pipeline_paused false
If approval queue is at the cap → drain it (approve / reject pending posts)
If worker is healthy and queue is empty but no new tasks fire → topic discovery may be stuck. See troubleshooting.md “Topic discovery keeps generating the same rejected topic genre.”

Ollama Unresponsive

Means. No local LLM inference in the last 6 hours WHILE tasks are pending — i.e. no cost_logs rows with cost_type='inference', cost_usd = 0, and provider IN ('litellm','ollama','ollama_native'). Ollama is probably down.

Why the predicate looks the way it does. It keys on cost_usd = 0 (local inference is free — cost_guard stamps is_local calls at $0), not provider = 'ollama'. Since the 2026-05-16 LiteLLM router cutover, local inference is logged provider='litellm' (the real Ollama model is in the model column), so the old provider = 'ollama' literal matched zero rows and the alert false-fired on every active task. cost_usd = 0 also means a cloud fallback through LiteLLM (cost_usd > 0) is deliberately excluded, so a dead Ollama masked by a paid fallback still surfaces. Do not “simplify” it back to a provider literal.

Triage.

curl -s http://localhost:11434/api/tags
# Expected: a JSON list of models. If hung / refused, Ollama is dead.

ollama ps   # Currently loaded models
nvidia-smi  # Is Ollama using GPU?

Fix.

# Restart Ollama
# Windows: net stop ollama; net start ollama
# Or just: ollama serve   (foreground)

# Verify required models present
ollama list
# If missing:
ollama pull nomic-embed-text
ollama pull gemma3:27b                 # critic model
ollama pull gemma-4-31B-it-qat:latest  # writer model (bakeoff winner 2026-06-18)
ollama pull glm-4.7-5090:latest        # reviser model (qa.rewrite rescue cycle)

Zero Published Posts This Week

Means. No posts rows with status='published' and published_at > NOW() - 7d. Content publishing has stopped. Triage. Three possible causes:

Pipeline isn’t generating → see § Pipeline Stalled
Pipeline is generating but everything’s getting rejected → see “Approval rate drops to ~0%” in troubleshooting.md
Posts are reaching awaiting_approval but Matt hasn’t approved any → operator action needed

-- Distinguish the cases
docker exec poindexter-postgres-local psql -U poindexter -d poindexter_brain -c \
  "SELECT status, COUNT(*) FROM pipeline_tasks
   WHERE created_at > NOW() - INTERVAL '7 days'
   GROUP BY status ORDER BY status;"

Fix. Whichever case applies — drain the queue, fix the writer model, or unpause the pipeline.

GPU Metrics Stale

Means. No gpu_metrics rows in the last 30 minutes. The scraper crashed. Triage + fix.

# Is the exporter running?
curl -s http://localhost:9835/metrics | head -3
# If hung / refused, restart it.

# Windows
taskkill /IM python.exe /FI "WINDOWTITLE eq nvidia-smi-exporter" 2>NUL
pythonw scripts/nvidia-smi-exporter.py

If running via the GPU scraper service:

docker ps | grep gpu-scraper   # if containerized
# Or check Task Scheduler for the host script

Disk Space Low

Means. A host volume is running low on free disk space. This is now a Prometheus alert, defined in infrastructure/prometheus/alerts/infrastructure.yml. (It used to be a Grafana SQL rule, but that rule was mislabeled — it actually queried pg_database_size, duplicating § DB Size Warning — so it was removed 2026-06-03 and replaced with real host-disk monitoring.) Two thresholds fire:

PoindexterDiskSpaceLow (warning) — a drive-letter volume has under 20 GB free for 10 minutes.
PoindexterDiskSpaceCritical (critical) — under 10 GB free for 5 minutes; Postgres writes, Docker pulls, and image generation will start failing.

Both are sourced from windows_exporter’s windows_logical_disk_free_bytes (the windows scrape job on host.docker.internal:9182), filtered to drive-letter volumes ([A-Z]:). The {{ $labels.volume }} label in the alert names the affected drive. Triage.

# Free space per drive (Windows host)
Get-PSDrive -PSProvider FileSystem | Select-Object Name,Used,Free
docker system df

Fix. Free up headroom on the named volume:

powershell scripts/docker-prune.ps1   # gentle: prunes dangling images + build cache

# Heavier hammer (also drops unused volumes — read the warning):
docker system prune -af --volumes

Other space to reclaim: clear old local model files, or archive high-churn tables (cost_logs / page_views). For Postgres-specific growth, see § DB Size Warning.

Host Memory Pressure

Means. The worker host is running out of physical RAM. The box is triple-booked — the container stack (WSL2, memory-capped in ~/.wslconfig), the host-native Ollama/inference fleet, and the operator desktop all draw on the same physical RAM. When available RAM approaches zero the OS pages the working set to the pagefile; the desktop compositor (which shares the GPU and RAM) stalls and the UI freezes — occasionally hard enough to force a dirty reset (Windows Kernel-Power 41). Two Prometheus alerts (DB-rendered from app_settings.prometheus.*, sourced from windows_exporter — the windows scrape job on host.docker.internal:9182) cover it:

PoindexterHostMemoryLow (warning) — windows_memory_available_bytes under 4 GB for 10 minutes. A headroom heads-up: the next allocation spike will page.
PoindexterHostMemoryThrashing (critical) — rate(windows_memory_swap_pages_written_total[5m]) over 2000 pages/s for 2 minutes. Active memory-pressure eviction — the freeze-in-progress signal. Idle median is ~0.08 pages/s, so it fires only during a genuine episode.

Both thresholds are DB-tunable (prometheus.threshold.host_memory_available_warning_gb, prometheus.threshold.host_memory_paging_critical_pages_per_sec). Neither carries an absent() guard — a bare comparison yields no series when the exporter is down, so exporter death routes to WindowsExporterDown, not a false memory page. Visualize on the Hardware & Power board → “Host Memory — pressure” row (available RAM, commit-vs-RAM, page-out rate). Triage.

# What's holding RAM (Windows host)? vmmemWSL = the whole container VM.
Get-Process vmmemWSL,vmmem -EA SilentlyContinue |
  Select-Object Name,@{n='WS_GB';e={[math]::Round($_.WorkingSet64/1GB,1)}}
Get-Process | Sort-Object WorkingSet -Descending |
  Select-Object -First 12 @{n='GB';e={[math]::Round($_.WorkingSet/1GB,2)}},ProcessName
# Real pressure vs reclaimable cache (available includes reclaimable standby):
(Get-Counter '\Memory\Available MBytes').CounterSamples[0].CookedValue

# Which container is bloated?
docker stats --no-stream --format "table {{.Name}}\t{{.MemUsage}}"

Fix. In order of leverage:

Immediate: close idle browser/Electron apps (they hold both RAM and display-GPU memory), or pause heavy pipeline work (image/video generation).
Reclaim a leak: docker restart <container> on a bloated one — cAdvisor has leaked before (docker restart poindexter-cadvisor reclaims it instantly).
Unpin idle VRAM: a KEEP_ALIVE=-1 model (e.g. the vision instance on :11435) that WDDM has paged to system RAM is holding host RAM; unpin it if vision QA is idle.
Durable: the box is structurally oversubscribed — move the container tier onto separate hardware (bare-metal Linux) or add physical RAM.

Not the PSU. A recurring Kernel-Power 41 reset here reads like a power fault, but the HX1500i telemetry (sensor_samples) shows normal power (~380 W) and temperatures before each reset, and host telemetry goes dark 28–30 min ahead of the reset — the freeze signature, not an instantaneous power trip. Chase memory pressure first.

Site DOWN

Means. Brain daemon’s site probe failed for > 5 minutes. https://www.gladlabs.io is returning non-2xx. Triage.

curl -I https://www.gladlabs.io
# Note status code.

# Vercel deploy status
gh run list --repo Glad-Labs/poindexter --limit 5

Fix.

404 on routes that should exist → ISR cache issue, see troubleshooting.md “Post is Not Found” + “Static export writes succeed but the frontend still shows stale data”
500 → Vercel build broken; check the deploy logs in Vercel dashboard
Connection refused → DNS / Vercel domain issue; check Vercel project settings

# Force a redeploy of the latest build
gh run rerun <latest_run_id> --repo Glad-Labs/poindexter

Wan Server DEGRADED

Means. curl http://localhost:9840/health returned degraded:true. Video generation will silently fall back to image-gen/Pexels stills. Fix. See troubleshooting.md entries:

“Wan-server enters DEGRADED state — /generate returns 503 forever”
“poindexter-wan-server container restart-loops every ~30 seconds”

Quick recovery:

docker restart poindexter-wan-server
sleep 30
curl -s http://localhost:9840/health | python -m json.tool

TBD — needs operator confirmation: No Grafana alert for this exists yet (per troubleshooting.md). The brain daemon should probe wan-server /health and alert; for now, this section assumes you noticed manually.

Unknown alert — generic triage

If an alert fired with a name that’s not in the index above:

Step 1 — Find it in Grafana

# Open the alert rules dashboard (self-hosted Grafana)
xdg-open http://localhost:3000/alerting/list
# Or on Windows
start http://localhost:3000/alerting/list
# From phone via tailnet: http://<your-tailnet-ip>:3000/alerting/list

Find the firing rule. Read its summary and description annotations.

Step 2 — Open the alert’s source query

In Grafana → click the alert → “View rule” → copy the SQL. Run it directly:

docker exec -it poindexter-postgres-local psql -U poindexter -d poindexter_brain
# Paste the SQL, see the current value.

Step 3 — Decide if it’s real

Value just barely over threshold → likely flapping; consider raising the threshold (it’s in infrastructure/grafana/provisioning/alerting/alert-rules.yml)
Value way over threshold → real issue; investigate the table the SQL queries

Step 4 — Document it

Add a section above with the alert name, the meaning, and the fix you applied.

Planned maintenance — muting alerts

If you’re doing intentional work that will fire alerts (e.g., bringing the worker down for an upgrade):

# Pause the pipeline so it doesn't try to claim tasks during the downtime
poindexter settings set pipeline_paused true

# Mute Grafana alerts via the contact-point silence
# Open http://localhost:3000/alerting/silences/new (or via tailnet)
# Select all matchers (label=team / value=glad-labs), set duration, save.

After maintenance:

poindexter settings set pipeline_paused false
# Remove the silence in Grafana UI

Escalation

This is a single-operator system. There is no on-call rotation.

Critical bugs in product code (Poindexter): file a GitHub issue at https://github.com/Glad-Labs/poindexter/issues
Glad Labs business / operator issues: file in the private tracker at https://github.com/Glad-Labs/poindexter/issues
Hardware issues: consult the operator’s own hardware notes (~/.claude/projects/<your-project>/memory/ if using Claude Code memory, or wherever you keep your install spec)

Incident response

Incident Response Runbook

Scenarios this covers

Quick triage flowchart

Alert index — alert name to runbook section

Worker Offline

Brain Daemon Stale

GPU Temperature High

High Error Rate

Stale Tasks

Embedding Sync Lag

DB Size Warning

Daily Cost Spike

Content Quality Drop

Traffic Anomaly

Page-View Capture Dead

Pipeline Stalled

Ollama Unresponsive

Zero Published Posts This Week

GPU Metrics Stale

Disk Space Low

Host Memory Pressure

Site DOWN

Wan Server DEGRADED

Unknown alert — generic triage

Step 1 — Find it in Grafana

Step 2 — Open the alert’s source query

Step 3 — Decide if it’s real

Step 4 — Document it

Planned maintenance — muting alerts

Escalation

See also

​Incident Response Runbook

​Scenarios this covers

​Quick triage flowchart

​Alert index — alert name to runbook section

​Worker Offline

​Brain Daemon Stale

​GPU Temperature High

​High Error Rate

​Stale Tasks

​Embedding Sync Lag

​DB Size Warning

​Daily Cost Spike

​Content Quality Drop

​Traffic Anomaly

​Page-View Capture Dead

​Pipeline Stalled

​Ollama Unresponsive

​Zero Published Posts This Week

​GPU Metrics Stale

​Disk Space Low

​Host Memory Pressure

​Site DOWN

​Wan Server DEGRADED

​Unknown alert — generic triage

​Step 1 — Find it in Grafana

​Step 2 — Open the alert’s source query

​Step 3 — Decide if it’s real

​Step 4 — Document it

​Planned maintenance — muting alerts

​Escalation

​See also

Incident Response Runbook

Scenarios this covers

Quick triage flowchart

Alert index — alert name to runbook section

Worker Offline

Brain Daemon Stale

GPU Temperature High

High Error Rate

Stale Tasks

Embedding Sync Lag

DB Size Warning

Daily Cost Spike

Content Quality Drop

Traffic Anomaly

Page-View Capture Dead

Pipeline Stalled

Ollama Unresponsive

Zero Published Posts This Week

GPU Metrics Stale

Disk Space Low

Host Memory Pressure

Site DOWN

Wan Server DEGRADED

Unknown alert — generic triage

Step 1 — Find it in Grafana

Step 2 — Open the alert’s source query

Step 3 — Decide if it’s real

Step 4 — Document it

Planned maintenance — muting alerts

Escalation

See also