feat: autodream — background memory consolidation

open_strix/builtin_skills/autodream/SKILL.md

@@ -0,0 +1,159 @@
+---
+name: autodream
+description: Background memory consolidation that runs between sessions. Merges redundant blocks, prunes stale content, resolves contradictions, and surfaces patterns from session transcripts. Fires via turn_complete watcher with gate conditions.
+---
+
+# autodream — Background Memory Consolidation
+
+You are a stateful agent whose memory blocks accumulate noise over time. New
+information gets appended, old facts go stale, contradictions creep in, and
+blocks grow until they dilute everything else in your prompt. Left unchecked,
+your memory degrades your own cognition.
+
+autodream is the maintenance cycle that prevents this. It runs in the background
+after enough activity has accumulated, consolidating what you've learned into
+denser, more accurate memory.
+
+## When It Fires
+
+The `autodream-gate` watcher runs on every `turn_complete`. It checks gate
+conditions and only produces a finding when ALL conditions are met:
+
+1. **Enough sessions** — at least 5 distinct session IDs in `logs/events.jsonl`
+   since the last dream
+2. **Enough time** — at least 24 hours since the last dream
+3. **Not currently dreaming** — a lock file prevents concurrent dreams
+
+The gate script is lightweight (reads a timestamp file + counts session IDs).
+Most turns, it exits silently.
+
+When the gate opens, the watcher emits a finding routed to the agent. The agent
+then executes the dream using this skill's instructions.
+
+## Lock & State
+
+- **Lock file:** `state/autodream.lock` — contains PID and start timestamp.
+  Stale if mtime > 1 hour (the dream process died).
+- **Last dream timestamp:** `state/autodream-last.txt` — ISO 8601 UTC timestamp
+  of last successful dream completion.
+- **Dream log:** `logs/autodream.jsonl` — append-only record of every dream:
+  what was merged, pruned, or flagged.
+
+## The Four Phases
+
+When triggered, execute these phases in order. Each phase should be a distinct
+step in your reasoning.
+
+### Phase 1: Orient
+
+Read your current state to understand what you're working with.
+
+1. `list_memory_blocks` — get all blocks with sizes
+2. Read `state/autodream-last.txt` for when you last dreamed
+3. Count sessions since last dream (from `logs/events.jsonl`)
+4. Read `logs/journal.jsonl` — last 50 entries for recent context
+
+**Output:** A mental model of what's changed since last dream.
+
+### Phase 2: Gather Signal
+
+Scan session transcripts for information that should be in memory but isn't,
+or that contradicts what's currently stored.
+
+1. List session directories in `logs/sessions/` newer than last dream
+2. For each session, read the turn files (JSON with message history)
+3. Look for:
+   - **New facts** — people, channels, IDs, schedules, preferences
+   - **Corrections** — "actually it's X not Y", "that changed", explicit updates
+   - **Patterns** — recurring topics, repeated tool failures, behavioral shifts
+   - **Stale references** — files/paths/IDs mentioned in blocks that no longer exist
+
+**Important:** You're gathering signal, not acting yet. Note what you find.
+
+### Phase 3: Consolidate
+
+Apply changes to memory blocks and state files. Each change should be one of:
+
+#### Merge
+Two blocks cover overlapping territory. Combine into one denser block.
+- Keep the more specific block ID
+- Delete the redundant block
+- Log: `{"action": "merge", "from": "block-a", "into": "block-b", "reason": "..."}`
+
+#### Prune
+A block contains information that is:
+- **Derivable** — grep/git can answer it (file paths, code patterns, git history)
+- **Stale** — references things that no longer exist
+- **Verbose** — prose where facts would suffice
+
+Rewrite the block to be denser. Don't delete useful information — compress it.
+- Log: `{"action": "prune", "block": "block-id", "removed_chars": N, "reason": "..."}`
+
+#### Update
+A block contains outdated information contradicted by recent sessions.
+- Update the specific lines
+- Log: `{"action": "update", "block": "block-id", "field": "...", "reason": "..."}`
+
+#### Surface
+Session transcripts revealed something that should be in a block but isn't.
+- Create a new block or add to an existing one
+- Log: `{"action": "surface", "block": "block-id", "source": "session-id", "reason": "..."}`
+
+#### Flag
+Something needs human attention — a contradiction you can't resolve, a decision
+that requires operator input, or a pattern that might indicate drift.
+- Don't resolve it yourself. Note it in the dream log with `"action": "flag"`.
+- If the agent has a channel for operator communication, mention it there.
+
+### Phase 4: Record
+
+1. Append all actions to `logs/autodream.jsonl` as one entry:
+   ```json
+   {
+     "timestamp": "2026-03-31T12:00:00Z",
+     "sessions_processed": 7,
+     "actions": [...],
+     "blocks_before": 12,
+     "blocks_after": 11,
+     "total_block_chars_before": 15000,
+     "total_block_chars_after": 12000
+   }
+   ```
+2. Write current UTC timestamp to `state/autodream-last.txt`
+3. Remove `state/autodream.lock`
+4. Journal the dream: what changed, what was flagged, net compression
+
+## What NOT to Consolidate
+
+- **Journal entries** — these are temporal narrative. Never edit or delete them.
+- **events.jsonl** — immutable append-only log. Never touch it.
+- **Session transcripts** — read-only during dreams.
+- **Active task state** — today.md, inbox.md, commitments.md are operator-managed.
+  Flag stale content but don't edit these without operator intent.
+
+## What to Prioritize
+
+The most valuable dream actions, in order:
+
+1. **Resolve contradictions** — two blocks saying different things is worse than
+   either being slightly stale
+2. **Prune derivable content** — if `grep` or `git log` can answer it, it
+   doesn't belong in a memory block
+3. **Surface missing context** — things the agent keeps re-discovering every
+   session because they're not in memory
+4. **Compress verbose blocks** — prose → facts, paragraphs → bullet points
+5. **Merge overlapping blocks** — fewer, denser blocks > many sparse ones
+
+## Behavioral Guidelines
+
+- **Conservative by default.** When unsure whether to prune something, don't.
+  It's cheaper to carry slight redundancy than to lose something useful.
+- **One dream at a time.** The lock file prevents concurrent dreams. If you find
+  a stale lock (mtime > 1 hour), remove it and proceed.
+- **Don't dream during active conversation.** If the operator is actively
+  messaging, defer the dream to the next quiet period.
+- **Log everything.** Every change gets a reason. The dream log is how operators
+  audit what happened to their agent's memory.
+- **Respect operator blocks.** Some blocks are operator-written (often the init
+  block or identity blocks). Be extra conservative with these — compress
+  your own additions, but don't rewrite the operator's words.

open_strix/builtin_skills/autodream/gate_check.py

@@ -0,0 +1,157 @@
+#!/usr/bin/env python3
+"""autodream gate check — lightweight watcher that decides whether to trigger a dream.
+
+Reads minimal state (timestamp file + session count) and exits silently most
+turns. Only emits a JSONL finding when all gate conditions are met.
+
+Receives on stdin (watcher contract):
+    {"trigger": "turn_complete", "trace_id": "...", "events_path": "/..."}
+
+Emits on stdout (when gate opens):
+    {"route": "agent", "message": "...", "severity": "info"}
+"""
+
+from __future__ import annotations
+
+import json
+import os
+import sys
+import time
+from datetime import datetime, timezone
+from pathlib import Path
+
+
+# Gate thresholds — tune these per deployment
+MIN_SESSIONS = int(os.environ.get("AUTODREAM_MIN_SESSIONS", "5"))
+MIN_HOURS = float(os.environ.get("AUTODREAM_MIN_HOURS", "24"))
+LOCK_STALE_SECONDS = 3600  # 1 hour
+
+
+def _home_dir() -> Path:
+    """Resolve the agent home directory from STATE_DIR or watcher env."""
+    state_dir = os.environ.get("STATE_DIR", "")
+    if state_dir:
+        return Path(state_dir).parent
+    return Path.cwd()
+
+
+def _read_last_dream(home: Path) -> datetime | None:
+    """Read the last dream timestamp, or None if never dreamed."""
+    ts_file = home / "state" / "autodream-last.txt"
+    if not ts_file.exists():
+        return None
+    try:
+        text = ts_file.read_text().strip()
+        return datetime.fromisoformat(text.replace("Z", "+00:00")).astimezone(
+            timezone.utc
+        )
+    except (ValueError, OSError):
+        return None
+
+
+def _check_lock(home: Path) -> bool:
+    """Return True if a dream is currently running (lock is fresh)."""
+    lock = home / "state" / "autodream.lock"
+    if not lock.exists():
+        return False
+    try:
+        age = time.time() - lock.stat().st_mtime
+        if age > LOCK_STALE_SECONDS:
+            # Stale lock — previous dream died. Remove it.
+            lock.unlink(missing_ok=True)
+            return False
+        return True  # Fresh lock — dream in progress
+    except OSError:
+        return False
+
+
+def _count_sessions_since(events_path: Path, since: datetime | None) -> int:
+    """Count distinct session IDs in events.jsonl since a given timestamp.
+
+    Reads the file backwards (tail) for efficiency — recent events are at the
+    end. Stops when it hits an event older than `since`.
+    """
+    if not events_path.exists():
+        return 0
+
+    session_ids: set[str] = set()
+    since_ts = since.isoformat() if since else ""
+
+    try:
+        with open(events_path, "r", encoding="utf-8") as f:
+            for line in f:
+                line = line.strip()
+                if not line:
+                    continue
+                try:
+                    event = json.loads(line)
+                except json.JSONDecodeError:
+                    continue
+
+                ts = event.get("timestamp", "")
+                if since_ts and ts < since_ts:
+                    continue
+
+                sid = event.get("session_id", "")
+                if sid:
+                    session_ids.add(sid)
+    except OSError:
+        return 0
+
+    return len(session_ids)
+
+
+def main() -> None:
+    # Read watcher stdin contract
+    try:
+        stdin_data = json.loads(sys.stdin.read())
+    except (json.JSONDecodeError, EOFError):
+        stdin_data = {}
+
+    events_path_str = stdin_data.get("events_path", "")
+    events_path = Path(events_path_str) if events_path_str else None
+
+    home = _home_dir()
+
+    # Gate 1: Not currently dreaming
+    if _check_lock(home):
+        return  # Silent exit — dream in progress
+
+    # Gate 2: Enough time since last dream
+    last_dream = _read_last_dream(home)
+    now = datetime.now(tz=timezone.utc)
+
+    if last_dream is not None:
+        hours_since = (now - last_dream).total_seconds() / 3600
+        if hours_since < MIN_HOURS:
+            return  # Silent exit — too soon
+
+    # Gate 3: Enough sessions
+    if events_path and events_path.exists():
+        session_count = _count_sessions_since(events_path, last_dream)
+    else:
+        # Fallback: check home events.jsonl
+        fallback = home / "logs" / "events.jsonl"
+        session_count = _count_sessions_since(fallback, last_dream)
+
+    if session_count < MIN_SESSIONS:
+        return  # Silent exit — not enough activity
+
+    # All gates passed — emit finding
+    finding = {
+        "route": "agent",
+        "severity": "info",
+        "message": (
+            f"autodream gate open: {session_count} sessions since last dream"
+            f" ({hours_since:.0f}h ago). "
+            f"Use the autodream skill to run memory consolidation."
+            if last_dream
+            else f"autodream gate open: {session_count} sessions, first dream. "
+            f"Use the autodream skill to run memory consolidation."
+        ),
+    }
+    print(json.dumps(finding))
+
+
+if __name__ == "__main__":
+    main()

open_strix/builtin_skills/autodream/watchers.json

@@ -0,0 +1,9 @@
+{
+  "watchers": [
+    {
+      "name": "autodream-gate",
+      "command": "uv run python .open_strix_builtin_skills/autodream/gate_check.py",
+      "trigger": "turn_complete"
+    }
+  ]
+}