Skip to content

edimuj/tokenlean

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

161 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Tokenlean mascot - a squirrel collecting tokens

tokenlean

57 CLI tools + MCP server that let AI agents understand codebases without burning tokens

npm version npm downloads license

WhyInstallQuick ReferenceAll ToolsAI IntegrationWorkflowsLanguages


Zero confignpm i -g tokenlean and you're done  ·  Token-conscious — every tool outputs only what's needed  ·  Fast — ripgrep-powered with disk caching  ·  Multi-language — JS/TS, Python, Go, Rust, Ruby  ·  MCP native — structured tool access, no shell overhead  ·  Minimal deps — installs in seconds


The Problem

If you've ever checked your usage quota at 2 PM and decided whether to debug now or wait for the reset — this is for you. Enterprise teams with unlimited API keys don't feel the burn. Solo devs, small teams, anyone on a Pro subscription does. Every cat of a 2000-line file when you needed one function, every 300-line test run when only 3 lines failed, every full directory read to find a signature — that's your working day getting shorter.

tokenlean fixes this:

Instead of... Use Savings
Reading a 500-line file for signatures tl-symbols ~90% fewer tokens
Reading all files to find exports tl-exports ~95% fewer tokens
Guessing what a change might break tl-impact Know for sure
Reading a file to extract one function tl-snippet ~85% fewer tokens
Running npm test and reading all output tl-run "npm test" Errors only
Scanning long logs for real failures tl-tail app.log Errors/warns + dedupe

How much are you wasting?

Find out in one command, no install needed:

npx tokenlean audit --all --savings --claudecode
Summary (348 Claude Code sessions)
  Opportunities:
  Category                Count  Actual     Saveable   Suggestion
  ----------------------------------------------------------------------------
  read-large-file          75x    253.4k      202.7k   -> tl-symbols + tl-snippet
  build-test-output        34x     28.2k       18.3k   -> tl-run
  grep-command            115x     59.6k       11.9k   -> Grep tool
  tail-command              7x      4.8k        3.3k   -> tl-tail
  find-command             25x      8.6k        2.6k   -> Glob tool
  curl-command             13x      3.2k        2.3k   -> tl-browse
  cat-large-file            1x      1.1k         902   -> tl-symbols + tl-snippet
  webfetch                  4x      1.2k         823   -> tl-browse
  head-command             11x      3.8k         759   -> Read tool (with limit)

  Still saveable:     243.6k of 363.9k (67%)

  Already saved by tokenlean:
  Tool              Count  Compressed   Raw estimate   Saved
  ------------------------------------------------------------------
  tl-snippet          233x      215.7k            2.2M   1.9M
  tl-symbols           93x       59.0k          295.0k   236.0k
  tl-run               98x       28.7k           82.0k   53.3k

  Tokens saved:       2.2M (424 uses)
  Capture rate:       90% of potential savings realized

tokenlean demo — tl-structure, tl-symbols, and tl-exports in action

Install

npm install -g tokenlean

Requires Node.js >= 18, ripgrep for search tools, and git for history tools.

Use tl as the global entry point for discovery and health checks:

tl          # List available tokenlean commands and tools
tl doctor   # Verify Node.js, ripgrep, git, hooks, and config
tl update   # Update the global tokenlean install
Development setup
git clone https://github.com/edimuj/tokenlean.git
cd tokenlean
npm link

Quick Reference

# What's in this file?           tl-symbols src/auth.ts
# Functions only (dir/multi-file) tl-symbols src/ --filter function
# Extract just one function      tl-snippet handleSubmit
# What does this module export?  tl-exports src/lib/
# How many tokens will this cost? tl-context src/api/
# What's the project shape?      tl-structure

# What depends on this file?     tl-impact src/auth.ts
# How complex is this code?      tl-complexity src/auth.ts
# Where are the tests?           tl-related src/Button.tsx

# What changed recently?         tl-diff --staged
# Is it safe to commit?          tl-guard
# Find real usage examples       tl-example useAuth
# Summarize noisy logs            tl-tail logs/app.log
# What's the tech stack?         tl-stack

Every tool supports -l N (limit lines), -t N (limit tokens), -j (JSON output), -q (quiet), and -h (help).

AI Agent Integration

Add tokenlean instructions to your AI tool's config with a single command:

AI Tool Command
Claude Code tl-prompt >> CLAUDE.md
Codex tl-prompt --codex >> AGENTS.md
Cursor tl-prompt --minimal >> .cursorrules
GitHub Copilot tl-prompt >> .github/copilot-instructions.md
Windsurf tl-prompt --minimal >> .windsurfrules

MCP Server — expose tokenlean as native MCP tools for structured, faster access with no CLI overhead:

# Add to any project's .mcp.json (Claude Code, Codex, etc.)
{
  "mcpServers": {
    "tokenlean": { "command": "tl-mcp" }
  }
}

# Or pick just the tools you need
{ "command": "tl-mcp", "args": ["--tools", "symbols,snippet,run"] }

8 core tools available via MCP: tl_symbols, tl_snippet, tl_run, tl_impact, tl_browse, tl_tail, tl_guard, tl_diff. Structured JSON in/out — no bash command construction, no argument parsing, no stdout scraping.

Hooks — automatically nudge agents toward token-efficient tool usage:

tl-hook install claude-code    # Gentle reminders when agents waste tokens
tl-audit --all --savings       # Measure actual savings across sessions

Agent Skills

Ready-made workflows following the Agent Skills open format, organized for both Claude Code and Codex runtimes.

skills/
  claude/   # Claude Code skill variants
  codex/    # Codex skill variants

Claude Code skills

Skill What it does
code-review Review PRs efficiently — scope, blast radius, complexity, then targeted code reading
explore-codebase Understand an unfamiliar project in minutes without reading everything
safe-refactor Rename, move, or extract code with impact analysis and verification at each step
add-feature Add functionality by studying existing patterns first — locate, learn conventions, implement, verify
debug-bug Systematic bug investigation — reproduce, localize with blame/history, trace call path, verify fix
debug-performance Measure before optimizing — establish baselines, identify bottlenecks, confirm improvements with numbers
write-tests Write tests by studying existing patterns and code under test before writing assertions
upgrade-deps Upgrade dependencies safely — audit usage, research breaking changes, scale effort to version jump
migrate-framework Incremental framework/API migration with verification at each step, batched by dependency order

Codex skills

Skill What it does
code-review Risk-first code review workflow for Codex using git diff + targeted validation
explore-codebase Build a fast architecture map in Codex with targeted reads and dependency tracing
safe-refactor Refactor safely in Codex using blast-radius checks and incremental verification
add-feature Add features in Codex by mapping precedent first, then implementing minimal safe changes
debug-bug Repro-first bug fixing workflow in Codex with root-cause tracing and regression checks
debug-performance Performance debugging in Codex with baseline metrics and before/after proof
write-tests Write behavior-focused tests in Codex that match project conventions
upgrade-deps Dependency upgrade workflow in Codex with changelog-driven risk control
migrate-framework Incremental framework/API migrations in Codex with batch-level verification
# Claude Code — copy a skill
cp -r node_modules/tokenlean/skills/claude/code-review ~/.claude/skills/

# Claude Code — copy all skills
cp -r node_modules/tokenlean/skills/claude/* ~/.claude/skills/

# Codex — copy a skill
cp -r node_modules/tokenlean/skills/codex/code-review ~/.codex/skills/

# Codex — copy all skills
cp -r node_modules/tokenlean/skills/codex/* ~/.codex/skills/

# Or clone and pick what you need
git clone https://github.com/edimuj/tokenlean.git
cp -r tokenlean/skills/claude/code-review ~/.claude/skills/
cp -r tokenlean/skills/codex/code-review ~/.codex/skills/

All Tools

Essential — the tools agents use 90% of the time
Tool Description Example
tl-symbols Function/class signatures without bodies tl-symbols src/utils.ts or src/
tl-snippet Extract one function/class by name tl-snippet handleSubmit
tl-impact Blast radius — what depends on this file tl-impact src/auth.ts
tl-run Token-efficient command output (tests, builds) tl-run "npm test"
tl-tail Token-efficient log tailing and summarization tl-tail logs/app.log
tl-guard Pre-commit check (secrets, TODOs, unused, circular) tl-guard
tl-structure Project overview with token estimates tl-structure src/
tl-browse Fetch any URL as clean markdown tl-browse https://docs.example.com
tl-context7 Look up library docs via Context7 API tl-context7 react "hooks"
tl-component React component analyzer (props, hooks, state) tl-component Button.tsx
tl-analyze Composite file profile (5 tools in 1) tl-analyze src/auth.ts
Understanding Code — structure and signatures without reading implementations
Tool Description Example
tl-context Estimate token usage for files/dirs tl-context src/api/
tl-types Full TypeScript type definitions tl-types src/types/
tl-exports Public API surface of a module tl-exports src/lib/
tl-docs Extract JSDoc/TSDoc documentation tl-docs src/utils/
tl-entry Find entry points and main files tl-entry src/
tl-scope Show what symbols are in scope at a line tl-scope src/cache.mjs:52
tl-schema Extract DB schema from ORMs/migrations tl-schema
tl-stack Auto-detect project technology stack tl-stack
Before Modifying Files — understand dependencies and impact
Tool Description Example
tl-deps Show what a file imports (with tree mode) tl-deps src/api.ts --tree
tl-related Find tests, types, and importers tl-related src/Button.tsx
tl-flow Call graph — what calls this, what it calls tl-flow src/utils.ts
tl-coverage Test coverage info for files tl-coverage src/
tl-complexity Code complexity metrics tl-complexity src/ --threshold 10
tl-errors Map error types and throw points tl-errors src/
tl-test-map Map source files to their test files tl-test-map src/cache.mjs
tl-style Detect coding conventions from code tl-style src/
Understanding History — track changes and authorship
Tool Description Example
tl-diff Token-efficient git diff summary tl-diff --staged
tl-history Recent commits for a file tl-history src/api.ts
tl-blame Compact per-line authorship tl-blame src/api.ts
tl-hotspots Frequently changed files (churn) tl-hotspots --days 30
tl-pr Summarize PR/branch for review tl-pr feature-branch
tl-changelog Generate changelog from commits tl-changelog --from v1
Finding Things — search and discover code patterns
Tool Description Example
tl-example Find diverse usage examples tl-example useAuth
tl-search Run pre-defined search patterns tl-search hooks
tl-secrets Find hardcoded secrets & API keys tl-secrets --staged
tl-todo Find TODOs/FIXMEs in codebase tl-todo src/
tl-env Find environment variables used tl-env --required-only
tl-unused Find unused exports/files tl-unused src/
tl-api Extract REST/GraphQL endpoints tl-api src/routes/
tl-routes Extract routes from web frameworks tl-routes app/
tl-npm Quick npm package lookup/compare tl-npm express fastify
GitHub Workflows — multi-step gh operations in single commands
Tool Description Example
tl-gh issue view View issue + all sub-issues in one call tl-gh issue view -R owner/repo 434 --no-body
tl-gh issue create-batch Create issues in bulk from JSON/JSONL echo '[...]' | tl-gh issue create-batch -R owner/repo
tl-gh issue create-tree Create parent + children with sub-issue links echo '{...}' | tl-gh issue create-tree -R owner/repo
tl-gh issue add-sub Link existing issues as sub-issues tl-gh issue add-sub -R owner/repo --parent 10 42 43
tl-gh issue close-batch Close multiple issues with optional comment tl-gh issue close-batch -R owner/repo 1 2 3 -c "Done"
tl-gh issue label-batch Add/remove labels across multiple issues tl-gh issue label-batch -R owner/repo --add "P0" 1 2 3
tl-gh pr digest Full PR status: CI, reviews, comments, merge readiness tl-gh pr digest -R owner/repo 123
tl-gh pr comments Review comments grouped by file with resolution status tl-gh pr comments -R owner/repo 123 --unresolved
tl-gh pr land Check CI, merge, close linked issues, delete branch tl-gh pr land -R owner/repo 123
tl-gh release notes Auto-changelog from commits/PRs, create release tl-gh release notes -R owner/repo --tag v1.2.0

All issue-creating commands support --project owner/N to auto-add to a GitHub project board.

Utilities
Tool Description Example
tl-audit Analyze Claude/Codex sessions and estimate token savings tl-audit --all --savings
tl-quota Check AI subscription quota (Claude, Codex) tl-quota
tl-hook Install token-saving agent hooks tl-hook install claude-code
tl-reddit Read Reddit threads as clean text tl-reddit <url> -c 20
tl-cache Manage tokenlean caches tl-cache stats
tl-config Show/manage configuration tl-config --init
tl-name Check name availability (npm/GH/domains) tl-name myproject -s
tl-playwright Headless browser content extraction tl-playwright example.com
tl-prompt Generate AI agent instructions tl-prompt --minimal

Language Support

Code analysis tools are JS/TS-first, but many work across languages. Git-based and search tools work with any language.

JS/TS Python Go Rust Ruby Elixir/Lua Other
tl-symbols
tl-snippet
tl-exports
tl-deps
tl-impact - - - -
tl-complexity - - - -
tl-flow - - - -
tl-docs - - - - -
tl-types - - - - - -
tl-component - - - - - -
tl-style - - - - - -
tl-routes - - - - -

full support   partial (regex-based patterns, may miss language-specific constructs)   - not supported

Tools not listed (tl-structure, tl-search, tl-diff, tl-todo, tl-secrets, tl-guard, tl-blame, tl-history, tl-hotspots, tl-example, tl-env, tl-run, tl-tail, etc.) are language-agnostic and work with any codebase.

Configuration

Create .tokenleanrc.json in your project root or ~/.tokenleanrc.json globally
{
  "output": {
    "maxLines": 100,
    "maxTokens": null
  },
  "skipDirs": [
    "generated",
    "vendor"
  ],
  "skipExtensions": [
    ".gen.ts"
  ],
  "importantDirs": [
    "domain",
    "core"
  ],
  "importantFiles": [
    "ARCHITECTURE.md"
  ],
  "searchPatterns": {
    "hooks": {
      "description": "Find React hooks",
      "pattern": "use[A-Z]\\w+",
      "glob": "**/*.{ts,tsx}"
    }
  },
  "hotspots": {
    "days": 90
  },
  "structure": {
    "depth": 3
  },
  "cache": {
    "enabled": true,
    "ttl": 300,
    "maxSize": "100MB",
    "location": null
  }
}

Config values extend built-in defaults (they don't replace them).

Caching

tokenlean caches expensive operations with git-based invalidation — including ripgrep-backed searches, cached JS/TS semantic facts for tl-symbols and tl-snippet, and the JS/TS dependency graph used by tl-deps and tl-impact. Cache entries invalidate automatically on commits or file changes.

tl-cache stats      # View cache statistics
tl-cache clear      # Clear cache for current project
tl-cache clear-all  # Clear all cached data

Disable with TOKENLEAN_CACHE=0 or in config: {"cache":{"enabled":false}}

Workflows

Starting on an unfamiliar codebase
tl-structure                    # Get the lay of the land
tl-entry                        # Find entry points
tl-exports src/lib/             # Understand the public API
tl-docs src/utils/              # Read documentation, not code
tl-types src/types/             # Understand data shapes
tl-schema                       # Understand the database
Before refactoring a file
tl-impact src/core/auth.ts      # What would break?
tl-deps src/core/auth.ts        # What does it depend on?
tl-related src/core/auth.ts     # Find the tests
tl-coverage src/core/auth.ts    # Is it well tested?
tl-complexity src/core/auth.ts  # How complex is it?
Understanding a component
tl-component src/Button.tsx     # Props, hooks, dependencies
tl-symbols src/Button.tsx       # Function signatures (or src/ for all)
tl-symbols src/ --filter function # Functions only across directory
tl-history src/Button.tsx       # Recent changes
tl-blame src/Button.tsx         # Who wrote what
Finding technical debt
tl-complexity src/ --threshold 15  # Complex functions
tl-unused src/                     # Dead code
tl-todo                            # Outstanding TODOs
tl-hotspots                        # Frequently changed (unstable?)
Security check before committing
tl-secrets                         # Scan for hardcoded secrets
tl-secrets --staged                # Only check staged files
tl-secrets --min-severity high     # Only high severity issues
Reviewing a PR
tl-pr feature-branch               # Summary of branch changes
tl-pr 123                          # GitHub PR #123 (needs gh CLI)
tl-pr --full                       # Include files, stats, commits
tl-gh pr digest -R owner/repo 123  # Full status: CI, reviews, comments, readiness
tl-gh pr comments -R owner/repo 123 --unresolved  # Unresolved review threads
Preparing a release
tl-changelog --unreleased          # What's new since last tag
tl-changelog v0.1.0..v0.2.0       # Between versions
tl-changelog --format compact      # Quick summary
tl-gh release notes -R owner/repo --tag v1.2.0 --dry-run  # Preview auto-changelog
tl-gh release notes -R owner/repo --tag v1.2.0             # Create GitHub release
Starting a new project
tl-name coolproject awesomelib     # Check npm, GitHub, domains
tl-name myapp -s                   # Suggest variations if taken
tl-npm express fastify koa         # Compare framework options
Running commands efficiently
tl-run "npm test"                  # Summarize test results
tl-run "npm run build"             # Extract build errors only
tl-run "eslint src/"               # Summarize lint violations
tl-run "npm test" --raw            # Full output with stdout/stderr preserved
tl-run "npm test" --raw -j         # Raw JSON includes separate stdout/stderr fields
tl-run "npm test" -j               # Structured JSON output
tl-tail logs/app.log               # Collapse repeats + surface errors/warnings
tl-tail logs/app.log -f            # Follow file updates with compact summaries
npm test 2>&1 | tl-tail            # Summarize piped logs
Looking up documentation
tl-browse https://docs.example.com/api  # Fetch docs as markdown
tl-context7 react "useEffect"           # Look up React docs via Context7
tl-context7 nextjs "app router"         # Next.js docs
tl-npm lodash --deps                    # Check package dependencies
tl-npm chalk --versions                 # Version history
Measuring token savings
# run without install
npx --package tokenlean tl-audit --latest                  # Direct tool invocation via npx
npx tokenlean audit --all --savings                        # Package entrypoint via npx

# provider-aware sessions
tl-audit --provider claude --latest                       # Claude Code only
tl-audit --codex --latest                                 # Codex only
tl-audit --latest --savings                               # Auto-detect provider; combined summary

# output detail levels
tl-audit --all --savings                                  # Summary only, across all matching sessions
tl-audit -n 5 --verbose --savings                         # Add per-session detail
tl-audit session.jsonl                                    # Analyze a specific session file

tl-audit analyzes both Codex and Claude Code sessions:

  • Opportunities — tokens wasted on large file reads, verbose build output, raw grep/cat/tail
  • Savings (with --savings) — tokens already saved by tokenlean usage, with capture rate
  • Add --verbose for per-session breakdown
Summary (270 sessions: 180 Claude Code, 90 Codex)
  Still saveable:     496k of 661k (75%)
  By provider:
  Claude Code       180 sessions     341k saveable     1.5M saved
  Codex              90 sessions     155k saveable     820k saved
  Already saved:      2.3M (531 tokenlean uses)
  Capture rate:       82% of potential savings realized

Install hooks to automatically nudge agents toward token-efficient alternatives:

# Claude Code — PreToolUse hooks (nudge toward better tools)
tl-hook install claude-code        # Install hooks (auto-detects claude-rig)
tl-hook install claude-code --global   # Force install to ~/.claude/
tl-hook install claude-code --rig dev  # Install to a specific rig
tl-hook status claude-code         # Check what's active
tl-hook uninstall claude-code      # Remove hooks

# Open Code — plugin (auto-wraps commands with tl-run/tl-browse)
tl-hook install opencode           # Install plugin to ~/.config/opencode/plugins/
tl-hook status opencode            # Check if installed
tl-hook uninstall opencode         # Remove plugin
GitHub batch operations
# View an issue with all its sub-issues (one API call)
tl-gh issue view -R owner/repo 434              # Bodies truncated to 5 lines
tl-gh issue view -R owner/repo 434 --no-body    # Compact: titles + labels only
tl-gh issue view -R owner/repo 434 --full       # Complete bodies

# Create issues in bulk from JSON
echo '[{"title":"Bug A","labels":["bug"]},{"title":"Bug B"}]' | \
  tl-gh issue create-batch -R owner/repo --project edimuj/1

# Create an epic with sub-issues
echo '{"title":"Epic","children":[{"title":"Task 1"},{"title":"Task 2"}]}' | \
  tl-gh issue create-tree -R owner/repo --project edimuj/1

# Sprint cleanup — close a batch with comment
tl-gh issue close-batch -R owner/repo 10 11 12 -c "Sprint complete"

# Label triage
tl-gh issue label-batch -R owner/repo --add "P1" --remove "triage" 5 6 7

# Land a PR (wait for CI, merge, close linked issues, delete branch)
tl-gh pr land -R owner/repo 123 --method squash
Extracting web content
tl-browse https://example.com/docs        # Fast: native markdown or HTML conversion
tl-browse https://example.com -t 2000     # Limit to ~2000 tokens
tl-playwright example.com                 # Full: headless browser (JS-rendered pages)
tl-playwright example.com -s "h1,h2,h3"  # Extract headings only
tl-playwright example.com --screenshot p  # Save screenshot

Design Principles

  1. Single purpose — one tool, one job
  2. Minimal output — show what's needed, nothing more
  3. Composable — every tool supports -j for JSON piping
  4. Fast — no heavy parsing, no external services, aggressive caching
  5. Multi-language — JS/TS first, expanding to Python, Go, Rust, Ruby

Also by Exelerus

Project Description
claude-rig Run multiple Claude Code profiles side-by-side — isolate or share settings, plugins, MCPs per rig
agent-awareness Modular awareness plugins for AI coding agents
claude-mneme Persistent memory for Claude Code — context across sessions
claude-simple-status Minimal statusline — model, context usage, quota at a glance
vexscan-claude-code Security scanner for untrusted plugins, skills, MCPs, and hooks

Changelog

Use tl-changelog to generate changelogs from git history on demand:

tl-changelog                      # Since last tag
tl-changelog v0.19.0..v0.21.0    # Between versions
tl-changelog --unreleased         # What's new since last release

Contributing

Contributions are welcome! See CONTRIBUTING.md for guidelines on adding tools, code style, and submitting PRs.

License

MIT © Edin Mujkanovic

About

Lean CLI tools for AI agents and developers - reduce context, save tokens

Topics

Resources

License

Contributing

Stars

Watchers

Forks

Contributors