diff --git a/.claude/commands/file-jiras.md b/.claude/commands/file-jiras.md new file mode 100644 index 000000000..5d35be0bd --- /dev/null +++ b/.claude/commands/file-jiras.md @@ -0,0 +1,26 @@ +Parse proposed JIRAs from a spike doc and file them via the Jira API + +You are filing JIRA sub-tickets for a Lightspeed Core feature. + +The user will provide either a spike doc path or tell you which feature's +JIRAs to file. They will also provide the parent JIRA ticket number. + +## Credentials + +Jira credentials must be in `~/.config/jira/credentials.json`. If this file +doesn't exist, tell the user to create it (see `dev-tools/file-jiras.sh` for +the format and instructions). + +## Process + +1. Run `dev-tools/file-jiras.sh ` with + `echo "quit"` piped in, so it parses and exits without filing. + +2. Read every file in `/tmp/jiras/`. For each, verify: + - Content matches the corresponding section in the spike doc (no truncation, + no extra content swallowed from subsequent sections). + +3. Report any issues to the user. If all files look correct, tell the user + to run the script interactively — provide the full command including `cd` + to the repository root: + `cd && sh dev-tools/file-jiras.sh ` diff --git a/.claude/commands/spec-doc.md b/.claude/commands/spec-doc.md new file mode 100644 index 000000000..00ad441ee --- /dev/null +++ b/.claude/commands/spec-doc.md @@ -0,0 +1,12 @@ +Create a feature spec doc with requirements, architecture, and implementation guide + +You are creating a feature spec doc for the Lightspeed Core project. + +Follow the guidance in `docs/contributing/howto-write-a-spec-doc.md`. Use +`docs/contributing/templates/spec-doc-template.md` as the starting point. + +The user will provide context about the feature — a spike doc, a description, +or a conversation. Read what is provided and ask for anything missing. + +Place the spec doc at `docs/design//.md`. Confirm the +feature name and path with the user. diff --git a/.claude/commands/spike.md b/.claude/commands/spike.md new file mode 100644 index 000000000..547af95dd --- /dev/null +++ b/.claude/commands/spike.md @@ -0,0 +1,12 @@ +Run a design spike: research, PoC, decisions, and proposed JIRAs + +You are starting a spike for a feature in the Lightspeed Core project. + +Follow the process in `docs/contributing/howto-run-a-spike.md`. Use the +templates it references. + +The user will provide context about the feature — a JIRA ticket, a description, +or a conversation. Read what is provided and ask for anything missing. + +At decision points, present what you've found and ask the user before proceeding. +The user makes the decisions — you assist with the research and documenting. diff --git a/dev-tools/file-jiras.sh b/dev-tools/file-jiras.sh new file mode 100755 index 000000000..a6ec59a6a --- /dev/null +++ b/dev-tools/file-jiras.sh @@ -0,0 +1,462 @@ +#!/usr/bin/env bash +# File JIRA sub-tickets from a spike doc. +# +# Usage: +# file-jiras.sh +# +# Example: +# file-jiras.sh docs/design/conversation-compaction/conversation-compaction-spike.md LCORE-1311 +# +# Prerequisites: +# ~/.config/jira/credentials.json with email, token, instance. +# +# The script: +# 1. Parses JIRA sections from the spike doc (### LCORE-???? headings) +# 2. Writes each to /tmp/jiras/NN-short-name.md +# 3. Opens an interactive menu: view, edit, drop, file +# 4. Files selected tickets via Jira REST API + +set -euo pipefail + +CREDS="$HOME/.config/jira/credentials.json" +JIRA_DIR="/tmp/jiras" + +# --- Argument parsing --- + +if [ $# -lt 2 ]; then + echo "Usage: file-jiras.sh " + echo "Example: file-jiras.sh docs/design/.../spike.md LCORE-1311" + exit 1 +fi + +SPIKE_DOC="$1" +PARENT_TICKET="$2" + +if [ ! -f "$SPIKE_DOC" ]; then + echo "Error: spike doc not found: $SPIKE_DOC" + exit 1 +fi + +# --- Credentials --- + +if [ ! -f "$CREDS" ]; then + echo "Error: Jira credentials not found at $CREDS" + echo "Create it with:" + echo ' {"email": "you@redhat.com", "token": "...", "instance": "https://redhat.atlassian.net"}' + echo "Get a token at: https://id.atlassian.com/manage-profile/security/api-tokens" + exit 1 +fi + +JIRA_EMAIL=$(python3 -c "import json; print(json.load(open('$CREDS'))['email'])") +JIRA_TOKEN=$(python3 -c "import json; print(json.load(open('$CREDS'))['token'])") +JIRA_INSTANCE=$(python3 -c "import json; print(json.load(open('$CREDS'))['instance'])") + +# --- Parse spike doc --- + +rm -rf "$JIRA_DIR" +mkdir -p "$JIRA_DIR" + +python3 - "$SPIKE_DOC" "$JIRA_DIR" << 'PYEOF' +import re +import sys +from pathlib import Path + +spike_doc = Path(sys.argv[1]).read_text() +out_dir = Path(sys.argv[2]) + +# Split on ### LCORE-???? headings +pattern = r'^### (LCORE-\?{4}.*?)$' +sections = re.split(pattern, spike_doc, flags=re.MULTILINE) + +# sections[0] is everything before the first JIRA heading +# Then alternating: heading, body, heading, body, ... +count = 0 +for i in range(1, len(sections), 2): + heading = sections[i].strip() + body = sections[i + 1].strip() if i + 1 < len(sections) else "" + + # Stop if we hit a non-JIRA heading (e.g., "# PoC results") + if not heading.startswith("LCORE-"): + break + + count += 1 + + # Truncate body at the first # or ## heading (end of JIRAs section) + end_match = re.search(r'^#{1,2}\s', body, flags=re.MULTILINE) + if end_match: + body = body[:end_match.start()].strip() + + # Strip "LCORE-????: " prefix to get clean title + clean_title = re.sub(r'^LCORE-\?+:?\s*', '', heading).strip() + + # Extract short name for filename + short_name = re.sub(r'[^a-z0-9]+', '-', clean_title.lower()).strip('-') + if not short_name: + short_name = f"ticket-{count}" + + filename = f"{count:02d}-{short_name}.md" + content = f"### {clean_title}\n\n{body}\n" + (out_dir / filename).write_text(content) + +print(f"Parsed {count} JIRAs from {sys.argv[1]}") +PYEOF + +# --- Count tickets --- + +TICKET_COUNT=$(find "$JIRA_DIR" -maxdepth 1 -name '*.md' | wc -l) +if [ "$TICKET_COUNT" -eq 0 ]; then + echo "No JIRA sections found in $SPIKE_DOC" + echo "Expected headings like: ### LCORE-???? Title" + exit 1 +fi + +# --- Helper functions --- + +show_summary() { + echo "" + echo " # File Title" + echo " -- -------------------------------------------- ----------------------------------------" + i=1 + for f in "$JIRA_DIR"/*.md; do + title=$(head -1 "$f" | sed 's/^### //') + fname=$(basename "$f") + printf " %-2d %-44s %s\n" "$i" "$fname" "$title" + i=$((i + 1)) + done + echo "" + echo "Parent: $PARENT_TICKET" + echo "Total: $TICKET_COUNT tickets" + echo "" +} + +get_file_by_number() { + find "$JIRA_DIR" -maxdepth 1 -name '*.md' | sort | sed -n "${1}p" +} + +file_ticket() { + local ticket_file="$1" + local title + title=$(head -1 "$ticket_file" | sed 's/^### //') + + # Check for duplicates + local project_key="${PARENT_TICKET%%-*}" + local url_title + url_title=$(python3 -c "import urllib.parse,sys; print(urllib.parse.quote(sys.argv[1]))" "$title") + local dup_check + dup_check=$(curl -sS --connect-timeout 10 --max-time 30 \ + -u "$JIRA_EMAIL:$JIRA_TOKEN" \ + "$JIRA_INSTANCE/rest/api/3/search/jql?jql=project%3D${project_key}%20AND%20summary~%22${url_title}%22&fields=key,summary&maxResults=5" 2>/dev/null || echo "{}") + + local dup_count_file + dup_count_file=$(mktemp) + python3 -c " +import json, sys +title = sys.argv[1] +instance = sys.argv[2] +count_file = sys.argv[4] +try: + data = json.loads(sys.argv[3]) + issues = data.get('issues', []) + exact = [i for i in issues if i['fields']['summary'].strip().lower() == title.strip().lower()] + for i in exact: + print(f' Existing JIRA with same summary: {i[\"key\"]} — {i[\"fields\"][\"summary\"]}') + print(f' {instance}/browse/{i[\"key\"]}') + with open(count_file, 'w') as f: + f.write(str(len(exact))) +except Exception as e: + print(f' Duplicate check failed: {e}') + with open(count_file, 'w') as f: + f.write('-1') +" "$title" "$JIRA_INSTANCE" "$dup_check" "$dup_count_file" >&2 + local dup_count + dup_count=$(cat "$dup_count_file") + rm -f "$dup_count_file" + + if [ "$dup_count" -lt 0 ] 2>/dev/null; then + echo " Duplicate check failed; skipping ticket for safety." >&2 + return 1 + fi + if [ "$dup_count" -gt 0 ] 2>/dev/null; then + printf " File anyway? (y/n): " >&2 + read -r confirm < /dev/tty + if [ "$confirm" != "y" ] && [ "$confirm" != "Y" ]; then + echo " Skipped: $title" >&2 + return 1 + fi + fi + + # Extract description body (everything after the heading) + local body + body=$(tail -n +3 "$ticket_file") + + # Build ADF description from the body text + local adf_desc + adf_desc=$(python3 - "$body" << 'ADFEOF' +import json +import re +import sys + + +def parse_inline(text): + """Convert markdown inline formatting to ADF text nodes with marks.""" + nodes = [] + # Match: **bold**, *italic*, `code`, [text](url), plain text + pattern = r'(\*\*.*?\*\*|\*[^*]+\*|`[^`]+`|\[[^\]]+\]\([^)]+\))' + parts = re.split(pattern, text) + for part in parts: + if not part: + continue + if part.startswith("**") and part.endswith("**"): + nodes.append({ + "type": "text", + "text": part[2:-2], + "marks": [{"type": "strong"}] + }) + elif part.startswith("*") and part.endswith("*") and not part.startswith("**"): + nodes.append({ + "type": "text", + "text": part[1:-1], + "marks": [{"type": "em"}] + }) + elif part.startswith("`") and part.endswith("`"): + nodes.append({ + "type": "text", + "text": part[1:-1], + "marks": [{"type": "code"}] + }) + elif part.startswith("["): + m = re.match(r'\[([^\]]+)\]\(([^)]+)\)', part) + if m: + nodes.append({ + "type": "text", + "text": m.group(1), + "marks": [{"type": "link", "attrs": {"href": m.group(2)}}] + }) + else: + nodes.append({"type": "text", "text": part}) + else: + nodes.append({"type": "text", "text": part}) + return nodes + + +def make_paragraph(text): + return {"type": "paragraph", "content": parse_inline(text)} + + +def parse_block(para): + """Convert a markdown block (paragraph, list, heading, code) to ADF node(s).""" + # Heading + m = re.match(r'^(#{1,6})\s+(.*)', para) + if m: + level = len(m.group(1)) + return {"type": "heading", "attrs": {"level": level}, "content": parse_inline(m.group(2))} + + # Bullet list + if para.startswith("- "): + items = [line.lstrip("- ").strip() for line in para.split("\n") if line.strip().startswith("- ")] + list_items = [{"type": "listItem", "content": [make_paragraph(item)]} for item in items] + if list_items: + return {"type": "bulletList", "content": list_items} + + # Numbered list + if re.match(r'^\d+[\.\)]\s', para): + items = [re.sub(r'^\d+[\.\)]\s*', '', line).strip() for line in para.split("\n") if re.match(r'^\s*\d+[\.\)]\s', line)] + list_items = [{"type": "listItem", "content": [make_paragraph(item)]} for item in items] + if list_items: + return {"type": "orderedList", "content": list_items} + + # Code block + if para.startswith("```"): + code = para.strip("`").strip() + return {"type": "codeBlock", "content": [{"type": "text", "text": code}]} + + # Plain paragraph + return make_paragraph(para) + + +text = sys.argv[1] + +# Strip the redundant "**Description**:" line — Jira already has a Description field +text = re.sub(r'^\*\*Description\*\*:\s*', '', text).strip() + +paragraphs = [p.strip() for p in text.split("\n\n") if p.strip()] + +content = [] +for para in paragraphs: + node = parse_block(para) + if node: + content.append(node) + +doc = {"version": 1, "type": "doc", "content": content} +print(json.dumps(doc)) +ADFEOF + ) + + # Create the issue + local payload + payload=$(python3 - "${PARENT_TICKET%%-*}" "$title" "$adf_desc" "$PARENT_TICKET" << 'PAYEOF' +import json +import sys + +project_key, summary, adf_desc_json, parent_ticket = sys.argv[1:5] +print(json.dumps({ + "fields": { + "project": {"key": project_key}, + "issuetype": {"name": "Task"}, + "summary": summary, + "description": json.loads(adf_desc_json), + }, + "update": { + "issuelinks": [{ + "add": { + "type": {"name": "Blocks"}, + "outwardIssue": {"key": parent_ticket}, + } + }] + } +})) +PAYEOF +) + + local response + response=$(curl -sS --connect-timeout 10 --max-time 30 -w "\n%{http_code}" \ + -u "$JIRA_EMAIL:$JIRA_TOKEN" \ + -H "Content-Type: application/json" \ + -X POST "$JIRA_INSTANCE/rest/api/3/issue" \ + -d "$payload") + + local http_code + http_code=$(echo "$response" | tail -1) + local body_resp + body_resp=$(echo "$response" | sed '$d') + + if [ "$http_code" = "201" ]; then + local key + key=$(echo "$body_resp" | python3 -c "import sys,json; print(json.load(sys.stdin)['key'])") + echo " Created: $key — $title" >&2 + echo " $JIRA_INSTANCE/browse/$key" >&2 + echo "$key" + return 0 + else + echo " FAILED ($http_code): $title" >&2 + echo " $body_resp" >&2 + return 1 + fi +} + +# --- Interactive loop --- + +show_summary + +while true; do + printf "Command (view|v, edit|e, drop|d, file|f, quit|q): " + read -r cmd args || exit 0 + args="${args:-}" + + case "$cmd" in + view|v) + if [ "$args" = "all" ]; then + for f in "$JIRA_DIR"/*.md; do + echo "" + echo "════════════════════════════════════════════════════════════" + echo " $(basename "$f")" + echo "════════════════════════════════════════════════════════════" + echo "" + cat "$f" + echo "" + done + elif [ -n "$args" ]; then + for n in $(echo "$args" | tr ',' ' '); do + f=$(get_file_by_number "$n") + if [ -n "$f" ]; then + echo "" + echo "════════════════════════════════════════════════════════════" + echo " $(basename "$f")" + echo "════════════════════════════════════════════════════════════" + echo "" + cat "$f" + echo "" + else + echo " No ticket #$n" + fi + done + else + echo " Usage: view N or view N,M or view all" + fi + show_summary + ;; + edit|e) + editor="${EDITOR:-vi}" + if [ "$args" = "all" ]; then + $editor "$JIRA_DIR"/*.md + elif [ -n "$args" ]; then + files="" + for n in $(echo "$args" | tr ',' ' '); do + f=$(get_file_by_number "$n") + if [ -n "$f" ]; then + files="$files $f" + else + echo " No ticket #$n" + fi + done + if [ -n "$files" ]; then + # shellcheck disable=SC2086 + $editor $files + fi + else + echo " Usage: edit N or edit N,M or edit all" + fi + show_summary + ;; + drop|d) + if [ -n "$args" ]; then + for n in $(echo "$args" | tr ',' ' '); do + f=$(get_file_by_number "$n") + if [ -n "$f" ]; then + echo " Dropped: $(basename "$f")" + rm "$f" + TICKET_COUNT=$((TICKET_COUNT - 1)) + else + echo " No ticket #$n" + fi + done + show_summary + else + echo " Usage: drop N or drop N,M" + fi + ;; + file|f) + created_keys="" + if [ "$args" = "all" ]; then + for f in "$JIRA_DIR"/*.md; do + key=$(file_ticket "$f") && created_keys="$created_keys $key" + done + elif [ -n "$args" ]; then + for n in $(echo "$args" | tr ',' ' '); do + f=$(get_file_by_number "$n") + if [ -n "$f" ]; then + key=$(file_ticket "$f") && created_keys="$created_keys $key" + else + echo " No ticket #$n" + fi + done + else + echo " Usage: file N or file N,M or file all" + fi + if [ -n "$created_keys" ]; then + echo "" + echo "Created:$created_keys" + fi + show_summary + ;; + quit|q) + echo "Exiting. Ticket files remain in $JIRA_DIR/" + exit 0 + ;; + "") + ;; + *) + echo " Commands: view(v), edit(e), drop(d), file(f), quit(q)" + ;; + esac +done diff --git a/docs/contributing/howto-organize-poc-output.md b/docs/contributing/howto-organize-poc-output.md new file mode 100644 index 000000000..b22fe2dfd --- /dev/null +++ b/docs/contributing/howto-organize-poc-output.md @@ -0,0 +1,53 @@ +# How to organize PoC output + +When a spike includes a proof-of-concept, the validation results should be +structured so that reviewers can quickly understand what was tested and what was +found. + +## Directory structure + +Place results in `docs/design//poc-results/`. + +Name files with a numeric prefix that reflects reading order. Order them by +usefulness for the human reviewer: + +```text +poc-results/ +├── 01-poc-report.txt — findings, methodology, implications +├── 02-conversation-log.txt — human-readable record of the PoC +├── 03-token-usage.txt — quantitative data +├── 04-events.json — structured event data +├── 05-summaries.txt — extracted outputs +├── ... +└── NN-raw-data.json — full machine-readable data +``` + +Not all files apply to every PoC, use ones that make sense. + +## What a good report file contains + +The report file (`01-poc-report.txt`) is the most important output. A +reviewer who reads only this file should understand everything significant. + +Include: + +- **Glossary**: Define terms specific to the PoC. +- **PoC design**: What was tested, how, what parameters. +- **Results**: What happened, with numbers. +- **Findings**: What the results mean for the production design — what was + proved, disproved, or surprising. +- **Implications**: How the findings influence the design decisions in the + spike doc and spec doc. + +## What NOT to include in the merge + +PoC results are removed before merging the spike PR (see +[howto-run-a-spike.md](howto-run-a-spike.md), step 10). They serve their +purpose during review and are preserved in git history. + +## Naming conventions + +- Use plain English filenames, not timestamps or hashes. +- Prefer `.txt` for human-readable content, `.json` for structured data. +- If there are multiple PoC runs, use separate directories or name them + descriptively: `poc-results-5-query/`, `poc-results-50-query/`. diff --git a/docs/contributing/howto-run-a-spike.md b/docs/contributing/howto-run-a-spike.md new file mode 100644 index 000000000..fe586926b --- /dev/null +++ b/docs/contributing/howto-run-a-spike.md @@ -0,0 +1,189 @@ +# How to run a spike + +A spike is a time-boxed research task that produces a design recommendation and +proposed JIRAs. This document describes how to run one in the Lightspeed Core +project. + +**Claude Code shortcut**: `/spike` runs this process interactively. + +## Outputs + +A spike produces: + +1. **Spike doc** — decisions with recommendations, design alternatives with + pros/cons, proposed JIRAs. Use + [spike-template.md](templates/spike-template.md). +2. **Spec doc** — permanent in-repo feature spec (requirements, use cases, + architecture, implementation suggestions). Use + [spec-doc-template.md](templates/spec-doc-template.md). See + [howto-write-a-spec-doc.md](howto-write-a-spec-doc.md) for details. +3. **PoC** (optional but recommended) — working prototype that validates the + core mechanism. Not production code. +4. **PoC validation results** (if PoC was done) — structured evidence. See + [howto-organize-poc-output.md](howto-organize-poc-output.md). + +## Process + +### 1. Set up + +- Create a feature branch: `lcore-XXXX-spike-short-description` off + `upstream/main`. + +### 2. Research + +- **Current state**: Document how the relevant part of the system works today. + Include code references (file:line). +- **Existing approaches**: How do other APIs, tools, or frameworks solve the + same problem? Focus on the most relevant ones, not an exhaustive survey. +- **Gaps**: What capabilities are missing in the codebase for this feature + (e.g., no token estimation, no schema for summaries)? + +> **Example (LCORE-1311 conversation compaction spike):** Researched OpenAI, +> Anthropic, and Bedrock APIs for compaction approaches. Identified that +> lightspeed-stack has no token estimation capability. + +### 3. Design alternatives + +Identify the viable design alternatives. For each, document: + +- What it does +- Implementation sketch +- Pros/cons table +- Verdict (recommended / possible for later / too complex) + +Don't include alternatives that are obviously bad. Only include alternatives +that are genuinely worth considering. + +### 4. Build a PoC (recommended) + +A PoC validates that the core mechanism works. It is explicitly not production +code — cut corners on error handling, config, scope, and edge cases. + +What to include: +- The minimum code to prove the mechanism works. +- Unit tests for the core logic. +- Pass `uv run make format && uv run make verify`. + +What to skip: +- Production config integration. +- Error handling beyond the happy path. + +After building, run the PoC against a real stack to verify it works end-to-end. +Document the results in a structured evidence directory (see +[howto-organize-poc-output.md](howto-organize-poc-output.md)). + +> **Example (LCORE-1311 conversation compaction spike):** Built a recursive +> summarization PoC, ran a 50-query experiment with probe questions at +> intervals to test context fidelity. + +### 5. Write the spike doc + +Use [spike-template.md](templates/spike-template.md). + +Key principles: +- **Decisions up front, background below.** The first sections should be the + decisions that need confirmation. Background (current architecture, API + research, etc.) goes in later sections and is linked from the decisions. +- **Split decisions by audience.** Strategic decisions (approach, model, + threshold strategy) go in a section for the decision-makers and relevant + stakeholders. Technical decisions (storage schema, field naming, buffer + calculation) go in a section for the tech lead and relevant team members. +- **Proposed JIRAs** follow the decisions. Each JIRA should have: Description, + Scope, Acceptance Criteria, and an Agentic tool instruction pointing to the + spec doc. Use [jira-ticket-template.md](templates/jira-ticket-template.md). + +### 6. Write the spec doc + +Use [spec-doc-template.md](templates/spec-doc-template.md) and see +[howto-write-a-spec-doc.md](howto-write-a-spec-doc.md). + +The spec doc assumes all recommendations are accepted. It is the permanent +in-repo reference for implementation. If a decision is overridden during +review, update the spec doc accordingly. + +### 7. Open the PR + +Use [spike-pr-template.md](templates/spike-pr-template.md). + +The PR should contain: +- The spike doc and spec doc (in `docs/design//`). +- PoC code and tests (will be removed before merge). +- PoC validation results (will be removed before merge). + +In the PR description: +- List the decisions that need confirmation, with links to the specific lines + in the spike doc. +- Point reviewers to the "Proposed JIRAs" section for JIRA review. +- Note which sections need reviewer input and which are background reference. + +**Constructing review links**: Use the full commit hash with `?plain=1` for +line references in markdown files on GitHub. Format: +``` +https://github.com/ORG/REPO/blob/FULL_COMMIT_HASH/path/to/file.md?plain=1#L10-L25 +``` +Without `?plain=1`, GitHub renders the markdown and line anchors don't work. + +> **Example (LCORE-1311 conversation compaction spike):** PR grouped reviewer +> asks into strategic decisions (5 items), technical decisions (4 items), and +> proposed JIRAs — each with links to the specific sections. + +### 8. Incorporate reviewer feedback + +When reviewers comment or an external review comes in: + +1. Update both the spike doc and spec doc to reflect adopted changes. +2. Post a re-review request in the PR tagging the decision-makers. Group + by action needed: + - **New decisions** to confirm (link to each) + - **Changed decisions** to re-confirm (link to each) + - **Updated JIRAs** to review (link to each) + +> **Example (LCORE-1311 conversation compaction spike):** Reviewer suggested +> marker-based conversation handling instead of bypassing the `conversation` +> parameter. Adopted the suggestion, updated +> [Decision 6](../design/conversation-compaction/conversation-compaction-spike.md) +> in the spike doc and R10 in the spec doc. + +### 9. File JIRAs + +Once all decisions are confirmed: + +1. Update the parent feature ticket description to point to the spec doc. +2. File sub-JIRAs under the parent ticket using + [jira-ticket-template.md](templates/jira-ticket-template.md). + Use `dev-tools/file-jiras.sh` to parse and file them from the spike doc + (Claude Code shortcut: `/file-jiras`). +3. Each sub-JIRA's agentic tool instruction should point to the **spec doc** + (not the spike doc), since the spec doc is the permanent reference. + +### 10. Prepare for merge + +Before merging: + +- **Keep**: spec doc, spike doc. +- **Remove**: PoC code, PoC validation results, test config files, experiment + scripts. +- File the JIRA tickets under the parent ticket (step 9). +- Communicate the merge plan in the PR (what stays, what goes) and get + acknowledgement before merging. + +The spike doc stays in the repo because it records decision rationale, PoC +evidence, and the design space explored — context that the spec doc doesn't +capture. + +## Checklist + +``` +[ ] Branch created off upstream/main +[ ] Current state documented +[ ] Existing approaches researched +[ ] Design alternatives documented with pros/cons +[ ] PoC built and validated (if applicable) +[ ] Spike doc written (decisions up front, background below) +[ ] Spec doc written (with accepted recommendations) +[ ] PR opened with structured reviewer asks +[ ] Reviewer feedback incorporated +[ ] JIRAs filed under parent ticket +[ ] PoC code and experiment data removed before merge +[ ] Spike doc and spec doc remain in merge +``` diff --git a/docs/contributing/howto-write-a-spec-doc.md b/docs/contributing/howto-write-a-spec-doc.md new file mode 100644 index 000000000..f73a534ba --- /dev/null +++ b/docs/contributing/howto-write-a-spec-doc.md @@ -0,0 +1,62 @@ +# How to write a spec doc + +A spec doc is the permanent in-repo feature specification. It is the single +source of truth for what the feature does and how it works. All implementation +JIRAs reference it. Agentic coding tools read it for guidance. + +**Claude Code shortcut**: `/spec-doc` creates one interactively. + +## When to write one + +- As part of a spike (see [howto-run-a-spike.md](howto-run-a-spike.md), step 6). +- When a feature is well-understood but not yet documented. +- When an existing feature needs a retroactive spec. + +## How to write one + +Use [spec-doc-template.md](templates/spec-doc-template.md). + +### Location + +Place the spec doc at `docs/design//.md`. + +### Filling in the template + +**What**: Describes the feature. + +**Why**: The problem it solves. + +**Requirements (Rx)**: Numbered requirements. For each requirement it should be +easy to provide clear acceptance criteria. + +**Use Cases (Ux)**: "As a [role], I want [X], so that [Y]." + +**Architecture**: Flow diagram, then subsections for each component. Include +where things live (file paths), function signatures, schemas, configuration. + +**Implementation Suggestions**: File paths, insertion points, code patterns, +test patterns. Be specific — this section is read by both humans and agentic +coding tools. + +**Latency and Cost**: How the feature affects performance. Include if +applicable to the feature. + +**Open Questions**: Things explicitly deferred, and why. + +**Changelog**: Record significant changes after initial creation. Date, what +changed, why. + +**Appendices**: PoC evidence, API comparisons, reference sources. + +### Relationship to the spike doc + +The spike doc records everything that was considered. + +The spec doc records the approved decisions. + +### Keeping it up to date + +The spec doc is a living document. Update it when: +- A decision is changed. +- Implementation reveals something the spec didn't anticipate. +- A reviewer raises a point that changes the design. diff --git a/docs/contributing/templates/jira-ticket-template.md b/docs/contributing/templates/jira-ticket-template.md new file mode 100644 index 000000000..88b8dcca0 --- /dev/null +++ b/docs/contributing/templates/jira-ticket-template.md @@ -0,0 +1,21 @@ +### LCORE-???? TODO fill in title + +**User story** (if Story, delete if Task): As a TODO [persona], I want to TODO [action], so that TODO [reason/impact]. Use Story for user-facing changes, Task for feature work that does not represent a distinct user story. + +**Description**: TODO + +**Scope**: TODO + +- TODO + +**Acceptance criteria**: TODO + +- TODO + +**Agentic tool instruction**: TODO + +```text +Read the "[section]" section in docs/design//.md. +Key files to create or modify: [list]. +To verify: [how to manually test that the change works]. +``` diff --git a/docs/contributing/templates/spec-doc-template.md b/docs/contributing/templates/spec-doc-template.md new file mode 100644 index 000000000..4da0b0b5c --- /dev/null +++ b/docs/contributing/templates/spec-doc-template.md @@ -0,0 +1,133 @@ +TODO (example, delete): [conversation-compaction.md](../../design/conversation-compaction/conversation-compaction.md) (LCORE-1311) + +# Feature design for TODO: feature name + +| | | +|--------------------|-------------------------------------------| +| **Date** | TODO | +| **Component** | TODO | +| **Authors** | TODO | +| **Feature** | TODO: [LCORE-XXXX](https://redhat.atlassian.net/browse/LCORE-XXXX) | +| **Spike** | TODO: [LCORE-XXXX](https://redhat.atlassian.net/browse/LCORE-XXXX) | +| **Links** | TODO | + +## What + +TODO: What does this feature do? + +## Why + +TODO: What problem does this solve? What happens today without it? + +## Requirements + +TODO: Numbered, testable requirements. For each, it should be easy to provide clear acceptance criteria. + +- **R1:** +- **R2:** + +## Use Cases + +TODO: User stories in "As a [role], I want [X], so that [Y]" format. + +- **U1:** +- **U2:** + +## Architecture + +### Overview + +TODO: Flow diagram showing the request/response path with the new feature. + +```text +TODO: flow diagram +``` + +TODO: Add subsections below for each relevant component. Delete any that don't apply, add feature-specific ones. + +### Trigger mechanism + +TODO: When and how the feature activates. + +### Storage / data model changes + +TODO: Schema changes, which backends need updates. + +### Configuration + +TODO: YAML config example and configuration class. + +``` yaml +TODO: config example +``` + +``` python +TODO: configuration class +``` + +### API changes + +TODO: New or changed fields in request/response models. + +### Error handling + +TODO: How errors are surfaced — new error types, HTTP status codes, recovery behavior. + +### Security considerations + +TODO: Auth, access control, data sensitivity implications. Remove if not applicable. + +### Migration / backwards compatibility + +TODO: Schema migrations, API versioning, feature flags for gradual rollout. Remove if not applicable. + +## Implementation Suggestions + +### Key files and insertion points + +TODO: Table of files to create or modify. + +| File | What to do | +|------|------------| +| TODO | TODO | + +### Insertion point detail + +TODO: Where the feature hooks into existing code — function name, what's available at that point, what the code should do. + +### Config pattern + +All config classes extend `ConfigurationBase` which sets `extra="forbid"`. +Use `Field()` with defaults, title, and description. Add +`@model_validator(mode="after")` for cross-field validation if needed. + +Example config files go in `examples/`. + +### Test patterns + +- Framework: pytest + pytest-asyncio + pytest-mock. unittest is banned by ruff. +- Mock Llama Stack client: `mocker.AsyncMock(spec=AsyncLlamaStackClient)`. +- Patch at module level: `mocker.patch("utils.module.function_name", ...)`. +- Async mocking pattern: see `tests/unit/utils/test_shields.py`. +- Config validation tests: see `tests/unit/models/config/`. + +TODO: Describe any feature-specific test considerations (e.g., tests that need a running service, special fixtures, concurrency testing). + +## Open Questions for Future Work + +TODO: Things explicitly deferred and why. + +- TODO +- TODO + +## Changelog + +TODO: Record significant changes after initial creation. + +| Date | Change | Reason | +|------|--------|--------| +| | Initial version | | + +## Appendix A + +TODO: Supporting material — PoC evidence, API comparisons, reference sources. Add appendices as needed. diff --git a/docs/contributing/templates/spike-pr-template.md b/docs/contributing/templates/spike-pr-template.md new file mode 100644 index 000000000..798fc39d0 --- /dev/null +++ b/docs/contributing/templates/spike-pr-template.md @@ -0,0 +1,29 @@ +TODO (example, delete): [LCORE-1314 PR](https://github.com/lightspeed-core/lightspeed-stack/pull/1328) + +## LCORE-XXXX: TODO spike title + +Spike deliverable for TODO: LCORE-XXXX (parent feature). + +### What's in this PR + +**Design docs** (`docs/design/TODO/`): +- [TODO: `...-spike.md`](TODO: link to file on branch) _(use the "Outline" button)_ — the spike: research, design alternatives, PoC results, proposed JIRAs +- [TODO: `....md`](TODO: link to file on branch) _(use the "Outline" button)_ — feature spec — _changeable based on final decisions, will be kept in the repo long-term_ + +**PoC code** _(if applicable)_: +- TODO + +### Main findings + +1. TODO +2. TODO +3. TODO + +### For reviewers + +TODO: Tag the decision-makers. List strategic decisions with links to specific lines (use branch URL, not commit URL, so links stay current: `?plain=1#Lstart-Lend`). List technical decisions separately if different audience. Point to the "Proposed JIRAs" section for JIRA review. + +_**Doc structure note:** The decision and JIRA sections of the spike doc are where your input is needed. They link to background sections later in the doc — read those if you need more context on a specific point, but it is optional._ + +Closes LCORE-XXXX +Related: LCORE-XXXX diff --git a/docs/contributing/templates/spike-template.md b/docs/contributing/templates/spike-template.md new file mode 100644 index 000000000..aace4dac9 --- /dev/null +++ b/docs/contributing/templates/spike-template.md @@ -0,0 +1,83 @@ +TODO (example, delete): [conversation-compaction-spike.md](../../design/conversation-compaction/conversation-compaction-spike.md) (LCORE-1314) + +# Spike for TODO: feature name + +## Overview + +**The problem**: TODO + +**The recommendation**: TODO + +**PoC validation**: TODO + +## Decisions for TODO specify reviewer(s) + +These are the high-level decisions that determine scope, approach, and cost. +Each has a recommendation — please confirm or override. + +### Decision 1: TODO title + +TODO: Context, options table, recommendation. Link to the relevant background section(s) below. + +| Option | Description | +|--------|-------------| +| A | | +| B | | + +**Recommendation**: TODO + +## Technical decisions for TODO specify reviewer(s) + +Architecture-level and implementation-level decisions. + +### Decision N: TODO title + +TODO: Context, options table, recommendation. Link to the relevant background section(s) below. + +**Recommendation**: TODO + +## Proposed JIRAs + +TODO: One subsection per JIRA. Each JIRA's agentic tool instruction should point to the spec doc (the permanent reference), not this spike doc. + +### LCORE-???? TODO fill in title + +TODO: Use the format from `docs/contributing/templates/jira-ticket-template.md`. + +**Description**: TODO + +**Scope**: TODO + +- TODO + +**Acceptance criteria**: TODO + +- TODO + +**Agentic tool instruction**: TODO + +```text +Read the "[section]" section in docs/design//.md. +Key files: [files]. +``` + +## PoC results + +TODO: If a PoC was built, document what it does, what it proved, and how it diverges from the production design. + +### What the PoC does + +**Important**: The PoC diverges from the production design in these ways: +- TODO + +### Results + +TODO + +## Background sections + +TODO: Research and analysis that supports the decisions above. These sections are linked from the decisions, not read front-to-back. Common topics: current architecture, existing approaches, design alternatives. Add or remove as needed. + +## Appendix A + +TODO: Supporting material — external references, responses to suggestions from team members. Add appendices as needed. diff --git a/docs/contributing_guide.md b/docs/contributing_guide.md index eacfc308d..ddd40aa19 100644 --- a/docs/contributing_guide.md +++ b/docs/contributing_guide.md @@ -10,6 +10,7 @@ * [PR description](#pr-description) * [Definition of Done](#definition-of-done) * [A deliverable is to be considered “done” when](#a-deliverable-is-to-be-considered-done-when) +* [Feature design process](#feature-design-process) * [AI assistants](#ai-assistants) * [“Mark” code with substantial AI-generated portions.](#mark-code-with-substantial-ai-generated-portions) * [Copyright and licence notices](#copyright-and-licence-notices) @@ -118,6 +119,41 @@ Happy hacking! * Code changes reviewed by at least one peer * Code changes acked by at least one project owner +## Feature design process + +When implementing a new feature or significant change, we take the proposal, +run a spike for it, present the findings for review, make decisions about the +scope and design, create a permanent document with the feature specification, +and file ready-for-implementation JIRA tickets: + +1. **Run a spike** — research the problem, evaluate design alternatives, build + a PoC if needed, document decisions and recommendations. The spike produces + two documents and a set of proposed JIRAs. + → [How to run a spike](contributing/howto-run-a-spike.md) + +2. **Write a spec doc** — the permanent in-repo feature spec. Records the + approved design: requirements, architecture, implementation suggestions. + All implementation JIRAs reference it. + → [How to write a spec doc](contributing/howto-write-a-spec-doc.md) + +3. **Get decisions confirmed** — open a PR with the spike doc and spec doc. + Reviewers confirm or override the design decisions. + +4. **File implementation tickets** — once decisions are confirmed, file the + JIRA sub-tickets. Each ticket references the spec doc. + +If the feature is well-understood and doesn't need research, skip step 1 and +start at step 2. + +Templates for all of the above are in +[docs/contributing/templates/](contributing/templates/). If a PoC is part of +the spike, see +[how to organize PoC output](contributing/howto-organize-poc-output.md). + +**Claude Code shortcuts**: `/spike`, `/spec-doc`, and `/file-jiras` automate +parts of this process. Use `dev-tools/file-jiras.sh` directly for JIRA filing +without Claude Code. + ## AI assistants ### “Mark” code with substantial AI-generated portions.