Skip to content

Add watchers: unified cron + event-triggered monitoring#67

Open
strix-tkellogg wants to merge 3 commits intomainfrom
watchers-system
Open

Add watchers: unified cron + event-triggered monitoring#67
strix-tkellogg wants to merge 3 commits intomainfrom
watchers-system

Conversation

@strix-tkellogg
Copy link
Copy Markdown
Collaborator

Summary

  • Watchers system extending the poller framework with event triggers (turn_complete, session_start, session_end)
  • Unified watchers.json format — each entry has name, command, and exactly one of cron or trigger
  • Event-triggered watchers receive {trigger, trace_id, events_path} on stdin, emit JSONL findings to stdout
  • Finding routing: log (default, written to events.jsonl), agent (re-enqueued as new event), operator (logged for external notification)
  • pollers.json fully backward compatible — watchers.json is additive
  • Builtin algedonic-signals skill: SKILL.md with VSM context, anti-pattern catalog, hook authoring guide, and two example watchers (silent_session, codex_bypass)
  • 26 new tests (discovery, subprocess execution, fire_watchers integration)
  • 207/207 passing

Design Decisions

Per conversation Mar 26:

  1. Naming: watchers (covers both cron and event triggers without implying either mechanism)
  2. Input contract: Minimal — absolute path to events.jsonl + trace_id. Hook reads what it needs.
  3. Targets: Self-monitoring + operator + logging. Cross-agent deliberately left possible but not built-in.
  4. Latency: turn_complete fires after response sent = zero user-visible latency

Design doc: state/research/algedonic-hooks-design.md in discord-letta-bot

Test plan

  • watchers.json discovery works alongside existing pollers.json
  • Event-triggered watchers fire after agent turns
  • Cron-based watchers schedule correctly (same as pollers)
  • Watcher findings route correctly (log, agent, operator)
  • Invalid configs (both cron+trigger, unknown trigger) rejected with logging
  • Existing poller tests still pass

🤖 Generated with Claude Code

Extends the poller framework with event-triggered hooks under a new
watchers.json format. Watchers have either a cron schedule or an event
trigger (turn_complete, session_start, session_end) — same subprocess
and JSONL contract as pollers.

- WatcherConfig dataclass with cron/trigger fields
- _discover_watchers() scans skills/*/watchers.json
- _run_watcher_subprocess() sends {trigger, trace_id, events_path} on stdin
- fire_watchers() dispatches by trigger type, collects findings
- _run_post_turn_watchers() in app.py fires after each agent turn
- Finding routing: log (default), agent (re-enqueue), operator
- pollers.json backward compatible — watchers.json is additive
- Builtin algedonic-signals skill with SKILL.md, anti-pattern catalog,
  and two example watchers (silent_session, codex_bypass)
- 26 new tests covering discovery, subprocess execution, and fire_watchers

Co-Authored-By: Strix <strix@timkellogg.me>
Copy link
Copy Markdown
Owner

@tkellogg tkellogg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll have more feedback, just spitting this out since I have time now



@dataclass
class WatcherConfig:
Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

issue: I see that PollerConfig is still there above. Can you merge them so they're the same? Also have a clean upgrade path (I'm thinking pollers.jsonl remains there, if it's there already and continues to be recognized but stops being documented, in favor of watchers.json)

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 1b27f8b. Merged PollerConfig into WatcherConfig — single class, pollers.json still discovered (backward compat) but undocumented in favor of watchers.json. The _on_watcher_cron_fire wrapper simplified since it no longer needs to construct a temporary PollerConfig.

@@ -0,0 +1,172 @@
---
name: algedonic-signals
description: Understand and implement watchers — event-triggered monitoring hooks that detect behavioral drift, agreement violations, and operational anomalies. Use when the user wants to monitor agent behavior, detect drift patterns, set up turn_complete hooks, or understand the watchers.json system.
Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question: should this really be its own skill? my concern is that most of the time the agent doesn't need to be aware of this, but it's always there in the context. Maybe it should be a sub-section of the introspection skill? idk, curious what you think

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good call — moved it into introspection as a companion guide (debugging-watchers.md), same pattern as debugging-jobs/communication/drift. Example watchers live in introspection/watcher-examples/. The agent only sees 'watchers and algedonic signals?' as a one-line pointer in the introspection SKILL.md now, reads the full guide on demand.

…trospection

1. Unified PollerConfig into WatcherConfig — single config class for both
   cron-triggered and event-triggered monitors. pollers.json still discovered
   (backward compat) but watchers.json is the documented format going forward.

2. Moved algedonic-signals from standalone skill into introspection companion
   guide (debugging-watchers.md). Reduces always-in-context prompt load —
   agents read the guide only when investigating behavioral monitoring.
   Example watchers moved to introspection/watcher-examples/.

205/205 tests passing.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Copy link
Copy Markdown
Owner

@tkellogg tkellogg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

whelp, sorry, I just threw a wrench into the machine. We can discuss

Watchers extend the existing poller framework with event triggers — a watcher can
fire on a cron schedule (like a poller) **or** on an agent event (like `turn_complete`).

### watchers.json
Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion: separate watchers.json documentation from algedonic signals docs, then for the latter go even deeper into when and why you'd do it. For both, include more suggested patterns

Hey! Another thought -- that script we have for constructing a DAG over files is generally useful. We have that skill builder skill. We should use the script to check that all files in a skill are referenced, starting from SKILL.md and make sure there's no unreferenced sub-trees. And i'm saying this because you should run said skill script on this skill addition to make sure it works. I'm thinking we phrase it as start with a dir + a root file and the script spits out maybe mermaid, or json, idk. Some graph representation. The LLM can reason easily about the graph output. Mermaid is useful if humans want to peek. A reusable script like that could be used for linting skills, but also when we're linting the entire agents memory you can start from the memory blocks file(s) (oh, and jobs, those should have file references too).

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 95ab34d. Three changes:

  1. Doc separation: debugging-watchers.md is now pure mechanics (format, triggers, contracts, debugging checklist, 7 suggested patterns). New debugging-algedonic.md covers when/why — VSM context, 8 anti-pattern categories (agreement violations, silent sessions, behavioral shift, metric gaming, performative compliance, scope creep, context hoarding, premature escalation), detection shapes, in-process vs out-of-process, what to do with findings.

  2. DAG lint script: builtin_skills/scripts/dag_lint.py — takes a directory + root file, walks all file references (markdown links, backtick paths, bare relative paths, YAML strings), builds a reachability graph, outputs mermaid or JSON. --strict exits 1 on orphans. Ran it on introspection: 8/8 reachable, 0 unreferenced.

  3. 7 watcher patterns in debugging-watchers.md: cron review, multi-signal, stateful, threshold escalation, baseline comparison, frequency anomaly, cross-session consistency. Each with config example + code snippet.

4 new tests for DAG lint (211/211 passing).


## Writing a Watcher

See `watcher-examples/` in this skill directory for complete working examples.
Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion: link to individual files

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 95ab34d. SKILL.md now links to individual files for each companion guide entry — the watchers entry specifically links to silent_session.py, codex_bypass.py, and dag_lint.py individually, plus anchor links to specific sections in debugging-watchers.md.

Address Tim's PR #67 review feedback:

1. Split debugging-watchers.md (mechanics) from new
   debugging-algedonic.md (when/why + 8 anti-pattern categories)
2. Add dag_lint.py — reusable script that walks file references from
   a root file, builds dependency graph, flags unreferenced files.
   Outputs mermaid or JSON. --strict exits 1 on orphans.
3. Updated SKILL.md companion guide entries with individual file links
4. Added 7 suggested watcher patterns to debugging-watchers.md
5. 4 new tests for DAG lint (unreferenced detection, strict mode,
   full coverage, mermaid output)

DAG lint validates introspection skill: 8/8 files reachable, 0 orphans.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants