Skip to content

Add mkdocs-llmstxt plugin and remove hand-written llms.txt#117

Merged
gregology merged 1 commit intomainfrom
mkdocs-llmstxt-plugin
Mar 23, 2026
Merged

Add mkdocs-llmstxt plugin and remove hand-written llms.txt#117
gregology merged 1 commit intomainfrom
mkdocs-llmstxt-plugin

Conversation

@gregology
Copy link
Copy Markdown
Contributor

@gregology gregology bot commented Mar 23, 2026

Why

The current hand-written llms.txt links to rendered HTML pages on sctx.dev, so agents pulling those URLs get the full MkDocs Material chrome — nav, scripts, theme markup. Not useful for consumption. There's also no llms-full.txt, which is what most agents look for as a single-file dump of all docs.

What this does

Swaps the hand-written docs/llms.txt for the mkdocs-llmstxt plugin, which auto-generates both llms.txt and llms-full.txt at build time. The generated llms.txt links to .md versions of pages instead of HTML, and llms-full.txt concatenates all doc content into one file (~1000 lines).

The plugin is configured with explicit sections that mirror the structure from the old hand-written file, and the markdown_description carries over the same introductory paragraph.

The old docs/llms.txt source file is deleted because it would conflict with the plugin's output (both try to produce llms.txt in the build).

Alternatives considered

Build script (concat): A simple shell script or Makefile target that cats the markdown files together. No new deps, full control. Downside: nav order has to be kept in sync manually with mkdocs.yml. With only 8 doc pages this is fine today, but the plugin approach follows an existing convention and stays in sync automatically. Greg preferred this reasoning.

Raw markdown serving: Copy .md files into the build output so agents can fetch individual pages as clean markdown. More flexible but more moving parts in the deploy pipeline, and could confuse human visitors landing on raw markdown URLs. Overkill for now.

Review walkthrough

Single commit covering one coherent change (splitting it would leave a broken intermediate state):

  • Add mkdocs-llmstxt plugin and remove hand-written llms.txt — Adds mkdocs-llmstxt==0.5.0 to docs/requirements.txt, configures the llmstxt plugin in docs/mkdocs.yml with sections/descriptions/full_output, explicitly lists search plugin (adding a plugins: key overrides MkDocs defaults), and deletes docs/llms.txt.

Closes #116

Developer metrics

Total duration: 7m 30s
Turns: 129
Tool calls: 108
Tokens: 1,501,005 input / 10,473 output

Stage Model Duration Turns Tool calls Tokens (in/out) Cache read Cache creation
triage claude-opus-4-6 1m 29s 41 38 84,806 / 1,227 70,115 14,684
decompose claude-opus-4-6 2m 25s 28 25 702,742 / 4,129 675,313 27,399
implement_step_1 claude-opus-4-6 1m 52s 29 21 229,759 / 2,588 212,606 17,134
docs_review claude-opus-4-6 0m 22s 10 7 139,311 / 721 117,938 21,363
craft_pr claude-opus-4-6 1m 20s 21 17 344,387 / 1,808 306,826 37,543

Resolves #116

Three changes in the docs/ directory:

1. **docs/requirements.txt** — Add `mkdocs-llmstxt` as a dependency (pinned version).

2. **docs/mkdocs.yml** — Add a `plugins:` section with the `llmstxt` plugin configured as follows:
   - `markdown_description`: The extended description paragraph from the current hand-written llms.txt ("sctx replaces monolithic instruction files...")
   - `full_output: llms-full.txt` to generate the concatenated full-text file
   - `sections`: Map the three sections from the current llms.txt into the plugin config:
     - "Getting Started": `getting-started.md` with description
     - "Documentation": `protocol.md`, `cli-reference.md`, `examples.md`, `comparisons.md` with descriptions matching current llms.txt
     - "Optional": `contributing.md`, `roadmap.md` with descriptions
   Note: The `search` plugin that MkDocs enables by default will need to be explicitly listed since adding a `plugins:` key overrides the defaults.

3. **docs/llms.txt** — Delete this file. The plugin generates `llms.txt` into the `site/` build output directory, so having a source file with the same name in `docs/` (which is the docs_dir) would conflict.

The existing `site_name` ("Structured Context") becomes the `# heading` and `site_description` becomes the `> blockquote` in the generated llms.txt — both are already set in mkdocs.yml.
Copy link
Copy Markdown
Owner

@gregology gregology left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should full_output: llms-full.txt be full_output: llms.txt? Is there a good reason to use llms-full.txt?

@gregology
Copy link
Copy Markdown
Contributor Author

gregology bot commented Mar 23, 2026

Reviewed the feedback -- nothing actionable to address programmatically.

Developer metrics

Total duration: 0m 49s
Turns: 13
Tool calls: 9
Tokens: 81,997 input / 1,200 output

Stage Model Duration Turns Tool calls Tokens (in/out) Cache read Cache creation
analyze_feedback claude-opus-4-6 0m 41s 12 9 59,574 / 1,025 44,955 14,615
no_action_comment claude-opus-4-6 0m 7s 1 0 22,423 / 175 8,545 13,876

@gregology gregology merged commit 15f78e6 into main Mar 23, 2026
2 checks passed
@gregology gregology deleted the mkdocs-llmstxt-plugin branch March 23, 2026 23:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Make the docs more consumable for agents

1 participant