Niche Topic Discovery — Operator Guide

Day-to-day operator instructions for the niche-aware topic discovery + RAG writer modes shipped on 2026-04-30 (Glad-Labs/poindexter#278). For the architecture rationale (why this exists, table layout, decay math, provenance trail) see docs/architecture/niches-and-rag-modes.md. This page is the action surface — what to type, when to type it.

What this is, in 3 sentences

Topic discovery now runs per niche. You configure a niche once, then operate it as a stream of 5-candidate batches. The operator picks a winner from each batch; that handoff creates a content_task that flows through the regular pipeline.

Day-zero setup

The Glad Labs niche is pre-seeded (now folded into the baseline migration):

slug glad-labs, writer mode TWO_PASS, batch size 5, cadence floor 60 minutes
5 weighted goals — AUTHORITY 35, EDUCATION 25, BRAND 20, TRAFFIC 15, REVENUE 5
5 enabled sources — internal_rag 50, hackernews 20, devto 15, web_search 10, knowledge 5

Confirm it’s there:

poindexter topics niche list
poindexter topics niche show glad-labs

Adding a new niche today is a manual process: there is no poindexter niche create CLI yet (see “What’s not yet built” below). Until that lands, insert via SQL or call services.niche_service.NicheService from a Python shell:

psql "$DATABASE_URL" <<'SQL'
INSERT INTO niches (slug, name, target_audience_tags,
                    writer_rag_mode, batch_size,
                    discovery_cadence_minute_floor)
VALUES ('real-estate', 'Real Estate Investing',
        ARRAY['investors','agents'],
        'CITATION_BUDGET', 5, 120)
RETURNING id;
SQL
# Then INSERT rows into niche_goals (must sum to ~100)
# and niche_sources (enable + weight) for that niche_id.

The daily workflow

1. Trigger a sweep

Sweeps run one of four ways:

poindexter topics sweep --niche <slug> — fires an immediate sweep on demand. Calls the same TopicBatchService.run_sweep that the scheduler uses, so cadence-floor + open-batch checks still apply. Prints the new batch id + candidate count or the reason a sweep was skipped. This is the recommended interactive path (Glad-Labs/poindexter#349).
Scheduled run_niche_topic_sweep job — fires on the scheduler’s default cadence (every 30 minutes, gated per-niche by discovery_cadence_minute_floor). Default for hands-off operation.
Idle worker auto-trigger — the legacy IdleWorker background loop fires periodically (subject to the niche’s discovery_cadence_minute_floor and the topic_discovery_auto_enabled master switch — see kill-switch section).
Manual one-shot trigger — set topic_discovery_manual_trigger=true via poindexter settings set; the next idle worker tick sees it and fires regardless of the auto switch.

# Recommended interactive path:
poindexter topics sweep --niche glad-labs

# Manual one-shot via the settings switch (defers to next idle tick):
poindexter settings set topic_discovery_manual_trigger true \
  --category topic_discovery

2. View the open batch

poindexter topics show-batch --niche glad-labs

You’ll see candidate IDs, kinds (external / internal), effective scores, and titles.

3. Rank candidates

Pick your preferred order, best-first. The operator rank is the source of truth — it does not have to match the system rank.

poindexter topics rank-batch <batch_id> --order id1,id2,id3,id4,id5

4. (Optional) Edit the winner’s title or angle

poindexter topics edit-winner <batch_id> --topic "Sharper title"
poindexter topics edit-winner <batch_id> --angle "New framing"
poindexter topics edit-winner <batch_id> --topic "..." --angle "..."

5. Resolve — push the winner into the pipeline

poindexter topics resolve-batch <batch_id>

This marks the batch resolved, advances operator_rank=1 to a content_task, and unblocks the next sweep for that niche (subject to the cadence floor).

6. Or — reject everything and ask for a fresh sweep

poindexter topics reject-batch <batch_id> --reason "all stale"

Unpicked candidates carry forward into the next batch with their score multiplied by niche_carry_forward_decay_factor (default 0.7).

The 4 writer RAG modes

Set per-niche on niches.writer_rag_mode. Tasks without a writer_rag_mode fall back to the legacy generator, so pre-niche pipelines still work.

Mode	Snippet limit	What it does	Use when
`TOPIC_ONLY`	`writer_rag_topic_only_snippet_limit` = 8	Top 8 internal pgvector snippets dropped into the writer prompt. Single pass, no enforcement. Cheapest mode.	Broad niche, no hard citation contract, lowest cost per draft.
`CITATION_BUDGET`	`writer_rag_citation_budget_snippet_limit` = 12	Top 12 snippets fetched. Writer must cite ≥ N (default 3, key `writer_rag_citation_budget_min_citations`). Drafts under-budget are rejected pre-QA.	Authority/depth posts where unsupported claims must be cut.
`STORY_SPINE`	`writer_rag_story_spine_snippet_limit` = 15	Outline preprocessing pass (5-beat: hook / what_happened / why_it_matters / what_we_learned / close), then writer expands to prose.	Long-form narrative, postmortems, decision history, journey-style retros.
`TWO_PASS`	`writer_rag_two_pass_snippet_limit` = 20	Internal-only first draft → writer marks `[EXTERNAL_NEEDED: ...]` for missing facts → bounded external research → revise. LangGraph state machine. Glad Labs default.	First-person reporting on something nobody else has covered. Most expensive.

TWO_PASS revision loop is hard-capped at writer_rag_two_pass_max_revision_loops (default 3), and each external research call is capped at writer_rag_two_pass_research_max_sources (default 2 sources per [EXTERNAL_NEEDED] marker). All four modes use pipeline_writer_model for the LLM call (no per-mode model override yet — see “What’s not yet built”).

The kill-switch for legacy auto-discovery

The topic_discovery_auto_enabled setting (seeded on by default):

Default true — the legacy IdleWorker discovery loop keeps firing on its old signals (queue-low, stale-content, rejection-streak, 24h safety net). OSS installs without configured niches keep their existing behavior.
Set false — all auto-firing branches bail out early with an INFO log. Manual triggers still work. Use this for any install where you want to drive everything through poindexter topics.

Flip it:

poindexter settings set topic_discovery_auto_enabled false \
  --category topic_discovery

Glad Labs operators running niches should set this to false.

Tuning knobs you’ll actually touch

Most of these live in app_settings and are seeded with sane defaults. The two niches table columns at the top are per-niche, not app_setting.

Knob	Where	Default	What it controls
`niches.batch_size`	per-niche column	5 (Glad Labs)	Candidates per batch
`niches.discovery_cadence_minute_floor`	per-niche column	60 min (Glad Labs)	Minimum gap between sweeps for that niche
`niches.writer_rag_mode`	per-niche column	`TWO_PASS` (Glad Labs)	Switch the writer mode
`topic_discovery_auto_enabled`	`app_settings`	`true`	Legacy auto-discovery master kill-switch (see above)
`niche_carry_forward_decay_factor`	`app_settings`	`0.7`	How fast unpicked candidates fade across batches
`niche_top_n_per_pool`	`app_settings`	`5`	Top-N per pool fed to the LLM final-score stage
`niche_internal_rag_per_kind_limit`	`app_settings`	`4`	Items per source-kind fed into the internal RAG candidate pool
`niche_batch_expires_days`	`app_settings`	`7`	How long an open batch can sit before it expires
`niche_embedding_model`	`app_settings`	`nomic-embed-text`	Ollama embedding model (must match embeddings dim)
`niche_ollama_chat_timeout_seconds`	`app_settings`	`60`	Per-call HTTP timeout for the LLM scorer / distillation calls
`niche_goal_descriptions`	`app_settings` (JSON)	(7-key dict)	Prose anchors for each `goal_type` — drives goal-vector embedding
`writer_rag_topic_only_snippet_limit`	`app_settings`	`8`	Snippet count for `TOPIC_ONLY`
`writer_rag_citation_budget_snippet_limit`	`app_settings`	`12`	Snippet count for `CITATION_BUDGET`
`writer_rag_citation_budget_min_citations`	`app_settings`	`3`	Minimum internal citations the `CITATION_BUDGET` writer must hit
`writer_rag_story_spine_snippet_limit`	`app_settings`	`15`	Snippet count for `STORY_SPINE`
`writer_rag_two_pass_snippet_limit`	`app_settings`	`20`	Snippet count for `TWO_PASS` first draft
`writer_rag_two_pass_max_revision_loops`	`app_settings`	`3`	Hard cap on the `TWO_PASS` revise loop
`writer_rag_two_pass_research_max_sources`	`app_settings`	`2`	Sources fetched per `[EXTERNAL_NEEDED]` marker
`writer_rag_context_snippet_max_chars`	`app_settings`	`500`	Per-snippet character cap when building the writer prompt block

Set any app_setting via:

poindexter settings set <key> <value> --category <category>

Per-niche columns currently require a SQL UPDATE niches SET ... WHERE slug=... until the niche edit CLI lands — except the cadence target, which has its own command:

poindexter topics niche set-cadence <slug> <posts-per-day>

Sets niches.cadence_target_posts_per_day — a per-niche override that probe_cadence_slo (brain/health_probes.py) checks in addition to the site-wide cadence_slo_expected_posts_per_day target. A niche without this set isn’t checked individually; it’s only covered by the site-wide check.

Troubleshooting

Symptom	Likely cause	Fix
Tasks land in `awaiting_approval` you never queued	Legacy auto-discovery still firing	`poindexter settings set topic_discovery_auto_enabled false --category topic_discovery`
`topics show-batch` says “No open batch”	Either no sweep has run yet, or the last batch is still resolving	`poindexter topics sweep --niche <slug>` (see daily workflow §1)
Sweep runs but the batch is empty	`internal_rag` source disabled in `niche_sources`, or no recent embeddings to mine from	`poindexter topics niche show <slug>` and check sources; verify `embeddings` table has recent rows
Writer keeps citing the wrong source	`TOPIC_ONLY` has no enforcement; you may want `CITATION_BUDGET`	Switch the niche’s `writer_rag_mode`, or refine the niche’s prompt override (see follow-up table below)
`TWO_PASS` keeps hitting the revision cap and ships drafts with `[EXTERNAL_NEEDED]` markers still in them	Hard cap of 3 loops reached	Bump `writer_rag_two_pass_max_revision_loops` cautiously — high values risk runaway cost
Scoring LLM call times out	Local Ollama model is slow or wedged	Bump `niche_ollama_chat_timeout_seconds` or check Ollama health

What’s not yet built

Honest gap list as of 2026-04-30:

No poindexter niche create / edit / set-goal / enable-source CLI. New niches and goal/source edits go in via SQL or via the services.niche_service.NicheService Python API. The CLI subgroup exposes niche list, niche show, and niche set-cadence — everything else still needs SQL.
Writer-mode model selection is hardcoded to pipeline_writer_model. Every niche writer mode uses the one writer pin — there’s no per-mode “a smaller model for the STORY_SPINE outline, a heavier one for the TWO_PASS revise” knob yet. (The cost_tier.* tier ladder that might once have backed this was removed in PR #1907; a per-mode pin would be the replacement.)
niche_goal_descriptions is a single global JSON blob in app_settings. Per-niche goal prompt overrides need a niche_goal_prompts table keyed by (niche_id, goal_type). Filed as a follow-up; the JSON shape is the unblock-now compromise.
Some commands shown in docs/architecture/niches-and-rag-modes.md (poindexter topics discover, poindexter niche create, poindexter niche set-goal, poindexter niche enable-source) are forward-looking — they describe the intended end state, not the current CLI surface.

​Niche Topic Discovery — Operator Guide

​What this is, in 3 sentences

​Day-zero setup

​The daily workflow

​1. Trigger a sweep

​2. View the open batch

​3. Rank candidates

​4. (Optional) Edit the winner’s title or angle

​5. Resolve — push the winner into the pipeline

​6. Or — reject everything and ask for a fresh sweep

​The 4 writer RAG modes

​The kill-switch for legacy auto-discovery

​Tuning knobs you’ll actually touch

​Troubleshooting

​What’s not yet built