Skip to main content

​
OAuth tokens for Grafana alert webhooks

Grafana fires alerts at the worker via Alertmanager-shaped webhooks (POST /api/webhooks/alertmanager). This guide covers the OAuth-token path — option B from Glad-Labs/poindexter#247.

​
Why a pre-issued long-TTL JWT, not Grafana’s OAuth contact-point flow

Grafana’s contact-point UI exposes a oauth2 block on newer versions, but in practice it’s brittle:
  • The IdP fields drift across Grafana minor versions (the YAML keys in 10.x don’t match the UI labels in 11.x).
  • Grafana’s contact-point HTTP client has no way to refresh through a client_credentials grant — it expects an authorization-code style redirect URI, which a headless alerting system can’t satisfy.
  • The IdP-side error messages surface as opaque “401 Unauthorized” in the contact-point test page, with no log on the Grafana side.
Pre-issued long-TTL JWTs side-step all of that. The operator mints one JWT per quarter (90-day default), pastes it into the contact-point “Authorization Header” field, and rotates manually. The worker accepts both OAuth JWTs and the legacy static alertmanager_webhook_token during the migration window (Glad-Labs/poindexter#247).

​
Provision the token

The recommended path is --persist, which stashes the minted JWT encrypted in app_settings.grafana_webhook_oauth_jwt so scripts/start-stack.sh can decrypt and pass it through to Grafana via $GRAFANA_WEBHOOK_TOKEN. No manual paste-into-the-UI step — restart Grafana and the contact point picks up the new credential.
This:
  1. First call: registers a new OAuth client named grafana-alerts in oauth_clients, persists encrypted credentials to app_settings.grafana_oauth_client_id and app_settings.grafana_oauth_client_secret.
  2. Every call: mints a fresh JWT bound to that client with the default scopes api:read api:write and the requested TTL.
  3. With --persist: also stores the minted JWT encrypted in app_settings.grafana_webhook_oauth_jwt. scripts/start-stack.sh decrypts it on the host (via scripts/_grafana_webhook_token.py) and exports as $GRAFANA_WEBHOOK_TOKEN before bringing the Grafana container up. infrastructure/grafana/provisioning/alerting/contact-points.yml substitutes that env var into settings.authorization_credentials on the Poindexter Webhook receiver.

​
Why settings instead of secure_settings

Grafana 13.0.1’s file-provisioning loader does NOT run env-var substitution over secure_settings: — only settings:. Prior attempts to wire the JWT through secure_settings.authorization_credentials silently no-op’d (verified during the 2026-05-19 finding #2 investigation; secureFields stayed null after restart with both camelCase and snake_case keys, and with literal-token values too). Grafana auto-encrypts plain settings.authorization_credentials at rest on its side, redacts it on read, and (crucially) DOES run env-var substitution over it. Our source of truth stays encrypted at rest in app_settings.grafana_webhook_oauth_jwt via pgcrypto, so the secret isn’t duplicated unprotected anywhere on disk. Output looks like:
Capture the JWT NOW — it’s not stored anywhere, and re-running the command mints a different JWT (the old one keeps verifying until its own exp elapses).

​
Paste into Grafana (legacy — only if you ran mint-grafana-token WITHOUT --persist)

If you minted with --persist, skip this section — the JWT is already wired through provisioning. The manual paste workflow below is the fallback for operators who minted with the older flow OR who want to override the auto-provisioned credential.
Grafana’s UI labels move every few releases. The path below is for Grafana 11.x. If your Grafana looks different, search for “contact points” → your contact point → custom HTTP headers.
  1. Open Grafana → Alerting → Contact points.
  2. Click the contact point that points at the worker webhook (default name: Poindexter Webhook).
  3. Edit → Optional Webhook settings → Custom HTTP headers.
  4. Click + Add header.
  5. Header name: Authorization
  6. Header value: Bearer <paste the JWT here>
    • The literal word Bearer, then a single space, then the JWT.
  7. Save contact point.
  8. Test the contact point (button at top of the edit page). The worker should respond 200 OK with a JSON body counting persisted alerts. If you get 401 Unauthorized, double-check there’s a single space between Bearer and the token and no trailing whitespace.
We deliberately don’t push the token via the Grafana API for the manual workflow — the contact-point JSON contains the bearer in plaintext, and the API push ergonomics (token in argv, token in shell history, token in CI logs) aren’t worth the time saved.

​
Verify end-to-end

After saving the contact point:
If the test fires but the worker logs OAuth JWT rejected: token expired, re-run the mint command — your previous token’s exp already elapsed (unusual on a 90-day TTL, but possible if you provisioned a 60-min token by accident).

​
Rotation

Set a calendar reminder for ~80 days after each mint (the default 90-day TTL minus 10 days of buffer). When it fires:
Paste the new JWT into the same Grafana contact-point header, save. The previous JWT keeps verifying until its own exp elapses, so there’s no traffic gap. To invalidate ALL JWTs for the Grafana client immediately (e.g. after a suspected leak), revoke the underlying OAuth client:
The Grafana contact-point header continues using the old JWT until the operator pastes the new one in — same procedure as a normal rotation.

​
Troubleshooting

Symptom from GrafanaLikely cause
401 Unauthorized on testToken typo, missing Bearer prefix, the token’s exp elapsed, or the underlying OAuth client was revoked. Re-mint and re-paste.
503 Service Unavailableapp_settings.alertmanager_webhook_token is empty. Run poindexter settings set alertmanager_webhook_token "<value>" --secret.
Test succeeds, real alerts never fireCheck the Grafana alert rule’s Notifications block — the contact point must be selected.
Note: Glad-Labs/poindexter#249 dropped the static-Bearer fallback on the general worker-API middleware, but this webhook route is the exception. verify_alertmanager_token (routes/alertmanager_webhook_routes.py) still accepts the legacy static alertmanager_webhook_token alongside OAuth JWTs, so a correctly-set static token also passes. A header that is neither a valid JWT nor the configured static token gets 401 — or 503 if the static token is unset and the JWT path doesn’t accept (fail-closed).

​
See also