Conversation
Full rewrite of the LimaCharlie Python SDK and CLI (v2.0.0) with AI/LLM-first discoverability. All commands follow a consistent noun-verb pattern with rich help, --explain flags, and multiple output formats (json/yaml/csv/table/jsonl). Core infrastructure: - Click-based CLI with auto-discovered command modules - HTTP client with JWT auth, retry with backoff, rate limiting - Config system supporting ~/.limacharlie, env vars, named environments - Structured error hierarchy with suggestion messages - Output formatting with jmespath filtering SDK (31 modules in limacharlie/sdk/): - Organization, Sensor, D&R Rules, FP Rules, Hive, Outputs - Artifacts, Payloads, Search, Extensions, Installation/Ingestion Keys - Users, Groups, API Keys, Billing, Spout, Replay - Integrity, Exfil, Logging Rules, Configs (sync), AI generation - Investigations, USP, Jobs, YARA, ARL CLI (49 command modules in limacharlie/commands/): - Full CRUD for all resource types - Hive shortcuts: secret, lookup, playbook, adapter, cloud-sensor, sop, note - Help system: discover, help topics, cheatsheets, schema - Streaming, sync, search with LCQL support Tests: - 481 unit tests (v2) covering all SDK and core modules - 25 integration test files (63 tests) for API validation - Removed 3 obsolete v1 unit tests replaced by v2 equivalents Packaging: - Entry point updated to limacharlie.cli:main - Added click>=8.0 and jmespath dependencies Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- organization.py: service_request now base64-encodes request_data,
get_urls unwraps data["url"], get_detections/get_audit_logs use
correct response keys (detects/events) with next_cursor and
gzip decompression, get_available_services unwraps replicants key,
get_jobs decompresses compressed response
- sensor.py: get_info extracts data["info"], is_online checks
data["online"] dict, get_tags handles v1 {sid: {tag: null}} format,
is_isolated/is_sealed use should_isolate/should_seal fields,
get_events uses next_cursor with decompression, get_children_events
decompresses, get_overview extracts overview key
- search.py: adds https:// prefix and /v1 suffix to search URL,
includes oid in all request bodies, sends times as strings
- extensions.py: rewritten to use correct extension/request/{name}
endpoint with gzip+base64 encoded gzdata (was incorrectly going
through service_request)
- configs.py: uses Extensions class directly instead of nonexistent
org.extension_request method
- artifacts.py: rewritten to use ingestion URL with Basic auth
headers matching v1 Logs.py upload pattern
- insight.py: search_ioc info param now configurable (was hardcoded),
batch_search uses form-encoded params matching v1
- replay.py: passes is_async=True matching v1 behavior
- client.py: added unwrap() for gzip+base64 decompression
- cli.py: import errors logged when LC_DEBUG is set
- artifact.py CLI: --type/--start/--end/--limit no longer silently ignored
- sensor.py CLI: fixed --offset+--limit pagination
- _hive_shortcut.py: YAML-first parsing, supports usr_mtd/etag,
better --confirm error message
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Bug fixes: - Fix HiveRecord crash from invalid kwargs in _hive_shortcut.py - Fix org rename() sending wrong query param (newName -> name) - Fix subscribe/unsubscribe extension not splitting res_cat/res_name - Fix replay double-serialization of detect/respond via service_request - Fix on_refresh_auth callback inconsistency (now always passes client) - Fix sensor set-version --version incorrectly required - Fix org delete --confirm-token incorrectly required - Remove sensor upgrade --selector that was silently ignored - Fix ioc hosts using invalid IOC type instead of find_sensors_by_hostname - Add defensive .get() for bare resp["key"] in org and sensor SDK - Fix register_explain using spaces instead of dots in hive shortcuts New features: - Add OAuth browser-based login: limacharlie auth login --oauth - Support Google and Microsoft providers with --provider flag - Support headless OAuth with --no-browser flag - MFA/2FA handled automatically via existing infrastructure Documentation: - Rewrite README with comprehensive CLI reference for all command groups - Add accurate Python SDK examples matching actual v2 API patterns - Document both API key and OAuth authentication flows - Add SDK class reference table - Move legacy v1 docs to bottom, v2 content first Tests: - Add tests for rename, subscribe/unsubscribe param splitting - Add tests for get_all_tags defensive .get() and find_sensors_by_hostname - Add tests for on_refresh_auth callback consistency - Add tests for HiveRecord raw dict construction - Add test for replay non-double-serialization Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
SDK fixes discovered by running integration tests against the live API: - organization.py: fix add_api_key param (permissions→perms, json→join) - organization.py: add default start/end time for get_jobs() - artifacts.py: add required start/end params to list() with defaults - search.py: add required startTime/endTime defaults to validate() - integrity.py: replace non-existent get_rule action with list+filter - logging_rules.py: same get_rule fix as integrity - exfil.py: fix all action names, split delete into delete_event/delete_watch CLI fixes: - commands/exfil.py: add --type option to delete for event vs watch rules - commands/integrity.py: handle None return from get() with error message - commands/logging_cmd.py: same not-found handling as integrity - commands/artifact.py: pass start/end to SDK, remove redundant filtering Tests: 65 CLI integration tests covering all command groups (63 pass, 2 skip due to missing hive permissions on test API key). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove deprecated REST endpoints (GET/POST/DELETE /rules/{oid} and
/fp/{oid}) from the SDK and rewrite CLI commands to use the Hive API.
Rename `limacharlie rule` to `limacharlie dr` with namespace-to-hive
mapping (general→dr-general, managed→dr-managed, service→dr-service).
Replace FP commands with a 3-line hive shortcut backed by the fp hive.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix Click command naming: add explicit @group.command("set") in dr.py
and hive.py to prevent Click deriving "set-cmd"/"set-record" names
- Fix HiveRecord construction crash in extension.py config-set (use
raw= pattern instead of invalid kwargs)
- Fix payloads SDK: implement two-step signed URL pattern for upload
and download matching v1 behavior
- Fix OAuth token persistence: save refreshed tokens to config file
- Fix exception swallowing in dr_rules.py/fp_rules.py: only catch
ApiError with 404 status, re-raise other errors
- Fix insight.py: decompress response when is_compressed=true is set
- Fix hive.py rename: remove double URL encoding of new_name
- Update payload CLI download command to handle raw bytes from SDK
- Add test for non-404 errors propagating in DR rules get
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
New `limacharlie auth signup` command that lets users create a brand new LimaCharlie account and organization directly from the CLI. The flow: OAuth authentication -> user profile creation via the signUp Cloud Function -> new organization creation -> credentials saved. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
New `limacharlie download` command group with sensor, adapter, and list subcommands. Downloads pre-built binaries from downloads.limacharlie.io for all supported platforms (Windows, Linux, macOS, Chrome, AIX, FreeBSD, OpenBSD, NetBSD, Solaris) with sensible default filenames, auto-executable permissions, and stdout piping support. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Modernize the CLI v2 codebase with three major improvements: - Add `from __future__ import annotations` and full type annotations to all 80+ Python files (core, SDK, and command modules) - Convert LimaCharlieContext and HiveRecord to @DataClass, add Credentials TypedDict to config.py - Migrate packaging from setup.py/setup.cfg/requirements.txt to pyproject.toml with setuptools backend, add PEP 561 py.typed marker - Update Dockerfile and CI (cloudbuild) for new packaging - Add tests for dataclass conversions, type infrastructure, and packaging Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add consistent type annotations to _make_explain_callback inner functions across all 15 command files that were missing them - Remove unused imports (signal, sys, os) from stream.py, artifact.py, output_cmd.py - Fix spotcheck.py register_explain key from space to dot separator - Fix test_packaging.py tomllib import for Python 3.9/3.10 compat - Add tomli backport to pyproject.toml dev dependencies Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Current PyPI latest is 4.11.3, so 2.0.0 would be a downgrade. Version 5.0.0 signals the major rewrite and sorts above 4.x. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix AI methods: remove spurious {oid} from paths, use "query" not "description"
- Fix org param names: "permission"->"perm", "email"->"member_email"
- Fix groups set_permissions: "permissions" dict -> "perm" list
- Rewrite investigations SDK to use Hive for CRUD operations
- Remove non-existent endpoints: artifacts.get, billing.get_sku_definitions,
ingestion_keys.configure_usp, insight.get_object_timeline,
insight.get_host_count_per_platform, org.find_sensors_by_ip,
org.configure_usp_key, org.convert_extension_rules
- Remove CLI commands for removed endpoints (artifact get, billing skus,
extension convert-rules)
- Update investigation CLI commands to use --name with Hive-based SDK
- Update tests for new investigation Hive interface
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Update README.md investigation examples and all _EXPLAIN_* texts in investigation.py to use --name instead of --id, matching the Hive-based CLI commands. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Previously only ~8% of SDK methods had tests verifying actual HTTP contracts (URL paths, methods, param names, body fields). This led to bugs shipping undetected (wrong param names, wrong paths, wrong body fields). Now every SDK method has a unit test verifying its full HTTP contract. - 16 new test files for ai, artifacts, billing, exfil, extensions, firehose, fp_rules, insight, integrity, investigations, jobs, logging_rules, payloads, replay, wrappers, yara - Expanded organization (17→107 tests), sensor (13→27), spout, dr_rules, hive tests with full contract verification - Slimmed test_sdk_misc.py to only ARL/USP/downloads (moved rest to dedicated files) - Total: 814 passing tests (up from ~564) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
SDK fixes: - config.py: Clear stale uid from config when explicit api_key provided - Manager.py: Only inherit GLOBAL_UID when api_key also from globals - organization.py: get_all_tags handles null, unwrap api_keys/installation_keys - exfil.py: Split string path into list for service request - usp.py: Wrap single json_input dict in list Integration test fixes: - Fix exfil/hive/artifacts/outputs/search/AI/groups/replay tests - Handle user-only auth endpoints gracefully - Use valid FP rule data for hive CRUD test - Fix CLI command names for search tests Unit test updates: - Update exfil path and USP json_input assertions to match SDK fixes Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Table output now auto-truncates large dicts ({N keys}) and long lists
([N items]) to fit the terminal. --wide/-W disables truncation, and
--filter applies a JMESPath expression to transform output data.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds ~40 new tests across two files: - test_output.py: truncation, max_value_width, _table_value, wide mode, module-level filter expr, and detect_output_format tests - test_cli_commands.py: CliRunner tests for global --wide/--filter flags, auth (whoami, test, list-envs), org (info, urls, errors), sensor (list, get, delete without confirm), DR (list), and --explain flag Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove the infrastructure-service code path and use_extension toggle from Configs — all sync now goes through ext-infrastructure. Drop the legacy --rules / --fps CLI flags (and sync_rules / sync_fps SDK params) in favor of hive flags (--hive-dr-general, --hive-fp, etc.) matching the old v1 CLI. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…unknown api key" When users login with OAuth but don't set a default org, org-scoped commands would fail with the cryptic JWT endpoint error "unknown api key". Now we catch the missing OID before hitting the network and suggest use-org, --oid flag placement, or LC_OID env var. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Two issues caused OAuth login to fail with "unknown api key": 1. All _get_org helpers double-resolved credentials — they called resolve_credentials() then re-passed api_key explicitly to Client(), which cleared OAuth creds inside Client's own resolve_credentials(). Fix: let Client resolve credentials once from oid + environment. 2. OAuth login preserved any stale api_key already in the config file. Fix: clear api_key from config on successful OAuth login. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ut help text - adapter.py now uses "external adapter" noun to distinguish from cloud adapters - Hive shortcut factory uses proper a/an article for nouns starting with vowels - Subcommand help text uses the noun instead of generic "records" - Group help drops the internal hive name for cleaner output Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Mock resolve_credentials so the test doesn't pick up api_key from the developer's ~/.limacharlie config file. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ud adapters The bare 'adapter' name was ambiguous — cloud sensors are often called "cloud adapters". Now 'ext-adapter' and 'cloud-sensor' are clearly distinct. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Rename 'ext-adapter' to 'external-adapter' for clarity, and rename 'cloud-sensor' to 'cloud-adapter' so both adapter types use consistent naming. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Reflect the cloud-sensor → cloud-adapter and adapter → external-adapter renames in the hive shortcuts section. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Strip top-level wrapper keys (payloads, users, user_permissions, groups,
errors) so CLI output shows useful data directly instead of a single
wrapper object. Follows the existing resp.get("key", resp) pattern
already used by get_api_keys, get_installation_keys, get_outputs, etc.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Implement _is_list_of_dicts() to detect dict-of-dicts data (e.g. payloads keyed by name) and flatten it into a list-of-dicts for proper columnar table display. Adds a "name" column from the dict key when values don't already contain one. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The seal/unseal subcommands have nothing to do with network policy; 'endpoint-policy' better describes the full scope of the command group (network isolation + configuration sealing). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add a unified --ai-help flag that generates compact markdown help at three levels (top-level overview, command group, individual command). Remove the per-command --explain boilerplate from all 40 command files (-1,407 lines) while keeping the explain text registry that feeds --ai-help context. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
#242) The SDK was reading nextToken from the top-level SearchResponse, but the backend places it inside each SearchResult item. This meant only the first page of results was ever returned. Also use the backend's nextPollInMs hint for poll delays instead of a hardcoded 1 second. Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* fix: Update ext-ticketing references to ext-cases The extension has been renamed from ext-ticketing to ext-cases. Update API root URL, extension name, and action name to match. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: Update default API root to cases.limacharlie.io Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
…#244) AI agents were confusing the "key" and "json_key" fields returned by the installation key API, leading to incorrect sensor install instructions. Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* feat: Add AI session lifecycle management and usage tracking to CLI Add commands to manage AI sessions after creation and monitor usage: - `ai session list` - List org sessions with status filter and pagination - `ai session get` - Get session details (status, cost, tokens, model) - `ai session terminate` - Stop a running session - `ai session history` - View session conversation history - `ai usage list` - List API key identities with usage data - `ai usage get` - Get hourly token/cost breakdown per identity SDK methods added to AI class: list_sessions, get_session, terminate_session, get_session_history, list_usage_identities, get_usage. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: Improve session list/get/history output readability - session list: truncate initial_prompt to 120 chars (was dumping full multi-KB prompts in listing output) - session get: truncate prompt by default, add --full-prompt flag - session history: filter out internal system init messages (credential_diagnostics, mcp_config_debug, etc.) by default, add --raw flag to include everything Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Matches ext-cases 18620a2 which added an "info" severity below "low" for non-actionable cases. Updates CLI choices, help text, SLA example config, and SDK docstrings. Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* feat: add custom token expiry for search queries and get-token command Add support for generating JWT tokens with custom expiry times to prevent token expiration during long-running search queries. Changes: - Add Client.get_jwt(expiry_hours) method for generating tokens with custom validity duration - Add --token-expiry option to 'search run' and 'search saved-run' CLI commands (e.g. --token-expiry 8 for 8-hour token) - Add 'auth get-token' CLI command with --hours and --format options (port of v1 get-token command) - Security warning when generating tokens > 24 hours - Comprehensive tests for SDK method and CLI commands This addresses the issue where long-running searches (e.g. 1 month of WEL data) fail because the user's JWT expires mid-query. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: remove dead test helper and fix expiry timestamp skew - Remove unused _make_client helper from test file - Compute expiry_ts before calling get_jwt so the displayed value in JSON output closely matches the actual JWT expiry Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat: default search token expiry to 4 hours with config override Search queries now automatically use a 4-hour JWT token instead of the default ~1 hour, preventing mid-query token expiry on long-running searches. The expiry is resolved with three-tier priority: 1. --token-expiry CLI flag (highest) 2. search_token_expiry_hours in ~/.limacharlie config file 3. DEFAULT_SEARCH_TOKEN_EXPIRY_HOURS constant (4.0 hours) Adds get_config_value() to config module for reading arbitrary config keys with environment-aware lookup. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs: add changelog entry for 5.0.x release Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* feat: include query_id, region, oid in search error messages Customer reported a search timeout with no way to identify which query failed. Search errors now include query_id, region, and oid context for faster troubleshooting. Changes: - Add SearchError exception class to errors.py with query_id, region, and oid fields that are appended to the error message - Update sdk/search.py execute() to raise SearchError with full context on initiation failures, poll errors, and unexpected exceptions - Extract region from search URL for error context Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * test: add comprehensive tests for SearchError and search error handling Add TestSearchError class covering all context field combinations, bracket formatting, inheritance, empty strings, and custom suggestions. Expand TestSearchExecuteErrors with tests for initiation errors, poll errors, cleanup behavior, exception wrapping, and region extraction. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: wrap initiation transport exceptions in SearchError with context Transport-level exceptions (ConnectionError, AuthenticationError, etc.) during search initiation were propagating unwrapped, losing the region/oid context. Now all initiation failures produce a SearchError with available troubleshooting context. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat: add query string to SearchError, fix region URL pattern, add changelog - SearchError now stores and displays the original query string (truncated to 120 chars in message, full string in .query attribute) - Updated region extraction regex to match real search URLs (e.g. 9157798c50af372c.replay-search.limacharlie.io) - Added changelog entry for 5.0.x - Region extraction gracefully returns None for non-standard URLs Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat: add smart suggestion for token expiry errors in SearchError When SearchError detects keywords like "401", "unauthorized", or "token expired" in the error message, it now automatically suggests using --token-expiry or search_token_expiry_hours config instead of the generic "contact support" suggestion. Custom suggestions still override the automatic detection. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: avoid duplicate suggestions and suppress noise for self-explanatory errors Use raw_message when wrapping inner exceptions in SearchError to prevent suggestion text from being duplicated. Skip suggestions entirely for syntax, validation, and quota errors that are self-explanatory. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Search run/saved-run commands now display flattened event rows instead of raw SearchResult wrapper objects in table mode. Events from all pages are merged into a single table with smart column selection. Machine-readable formats (json, yaml, csv, jsonl) pass through the raw API response unchanged. Changes: - Unwrap events: flatten mtd + nested data using dot notation - Smart column limiting: max 15 columns, priority fields first, noisy routing metadata dropped. --wide bypasses the limit. - Progress feedback to stderr in interactive mode: query_id on start, page/events/elapsed during pagination, waiting status during polls - Cumulative stats summary using server-aggregated cumulativeStats - Stats from events results only (not facets/timeline) - Clean Ctrl+C: sends DELETE to cancel query on server, prints status - Handle null rows/facets/timeseries in API response (JSON null) - Flush stderr for immediate progress display - --raw flag to get old behavior in table mode - 79 tests covering all new functionality Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
The PATCH endpoint now accepts severity. Updated docstrings to reflect that detection fields are on CaseDetection, not Case. Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: Add on-disk JWT caching for CLI Each CLI invocation previously created a new JWT via HTTP roundtrip to jwt.limacharlie.io, even when running many commands within the same hour. This caches JWTs to disk so subsequent invocations reuse the same token until it nears expiry (10-minute buffer). Cache design: - File: ~/.limacharlie_jwt_cache (respects LC_CREDS_FILE) - Key: SHA-256 of oid + auth_method_tag + credential identity - Atomic writes (tempfile + os.replace), 0o600 permissions - Symlink rejection on read/write (O_NOFOLLOW on Unix) - Last-write-wins concurrency, no file locking - 10-minute expiry buffer before reusing cached JWT - 401 recovery: invalidate stale cache, fetch fresh, re-cache Search command optimization: - get_jwt() reuses cached JWT if remaining TTL >= requested expiry - Long-lived search JWTs (4h default) are cached to disk Also: - Refactor config.py save_config() to use cross-platform atomic_write (fixes existing Windows bug with Unix-only os.chown) - Add debug logging for all cache operations (--debug flag) - Add microbenchmarks (pytest-benchmark) and CI step - Disable via LC_NO_JWT_CACHE env var or no_jwt_cache config option Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: Close Response object in Spout.shutdown() to prevent GC warning On Python 3.14, when a streaming Response is garbage-collected without being explicitly closed, the iter_lines/iter_content generator finalization triggers urllib3's _error_catcher which tries to flush an already force-closed socket, causing: ValueError: I/O operation on closed file. The fix: after force-closing the raw socket (to unblock iter_lines), also call self._conn.close() to cleanly tear down the Response object. This prevents the GC finalizer from encountering the closed socket. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…249) * feat: fix --debug for all CLI commands and add enhanced debug output Previously --debug was silently ignored by every command except sync. This wires the debug callback through all 39 command files and enhances debug output with full request/response logging similar to Apache libcloud. New debug modes: - --debug: verbose request/response (headers, body truncated to 2048 chars) - --debug-full: like --debug but no body truncation - --debug-curl: print reproducible curl commands (uses shlex.quote for safe shell escaping). Can be combined with --debug for both outputs. All debug output goes to stderr. Sensitive headers (Authorization, X-Api-Key, Cookie) are masked in verbose output. Curl commands include real token values for direct copy-paste reproducibility. Includes 66 new tests covering header masking, shell escaping, curl command generation, mode interactions, body truncation, and shell injection safety. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * test: add integration tests for --debug flag across all CLI commands Add 119 tests verifying every CLI command correctly propagates --debug, --debug-full, and --debug-curl flags to the Client constructor. Tests include source-level AST inspection to catch future regressions, CLI integration via CliRunner for representative commands, flag combination tests, position flexibility tests, and hive shortcut coverage. Also fix test_curl_output_is_parseable_by_shlex which broke after the JWT cache feature added init-time debug messages to the log. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
…e support (#253) SDK layer: add _is_transient_poll_error() classification and _poll_with_retry() method to Search. Transient errors (5xx, ConnectionError, TimeoutError, SSLError) are retried with exponential backoff (2^attempt, capped at 30s). Permanent errors (401, 403, 404, 422, 429) and search-engine body errors are not retried. Add start_token/start_page parameters to execute() for server-side resume. Resume passes the stored pagination token to the server which re-runs the query from the cursor position embedded in the token. Works even after long delays between sessions. CLI layer: add --checkpoint, --resume, and --force flags to 'search run'. Add 'search checkpoints' to list checkpoints and 'search checkpoint-show' to display results from a checkpoint file through the same output pipeline as live searches. Context-aware error messages when checkpoint file exists. On Ctrl+C prints session stats and exact resume command. New limacharlie/search_checkpoint.py module (named to clarify it is search-specific, leaving room for other checkpoint mechanisms in the future). Performance: CheckpointReader.iter_results() streams JSONL lazily via generator. checkpoint-show uses streaming for JSONL output. Resume loop does not accumulate results in memory. Table/expand/JSON formats require the full result set in memory for column width computation; for large checkpoints use --output jsonl which streams. Security: checkpoint directories 0o700, files 0o600. Data files use O_EXCL|O_NOFOLLOW. Metadata reads use safe_open_read(). Path validation rejects non-absolute paths. Shell-escapes via shlex.quote. Robustness: corrupt mid-file JSONL lines raise ValueError (only last line tolerated via lookahead parsing). secure_makedirs tightens existing dirs. Also fixes pre-existing bug in client.py where KeyboardInterrupt during u.read() left 'data' variable unbound. Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The Cases SDK class and case CLI commands were implemented but not referenced in the documentation index files. Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…#254) * feat: streaming output, orjson, and CLI help improvements Add streaming output to avoid OOM on large searches. Previously all search results were buffered in a list before output, causing OOM on constrained VMs (e.g. 4GB RAM with 500K+ events). Now results stream one at a time for JSONL, JSON, expand, and table formats. Streaming behavior by format: - JSONL: one result per line, constant memory (all paths) - JSON: streaming array ([, item, item, ]), constant memory (all paths) - expand: one event block at a time, constant memory (all paths) - table (live search): sample first N pages for column widths, then stream remaining rows. O(sample + columns) memory. - table (checkpoint): two-pass over file - pass 1 computes exact column widths O(columns), pass 2 streams rows. Perfectly accurate layout. - CSV/YAML: still buffered (inherent to format, rarely used for large data) Key changes: - _stream_search_output(): core streaming function for JSONL, JSON, expand, and table from any iterable. Returns False for CSV/YAML. - _stream_table_events(): sample-based streaming table for live searches (configurable via _TABLE_SAMPLE_PAGES constant). - _stream_table_from_file(): two-pass streaming table for checkpoint files. - _run_normal and saved_run: try streaming first, fall back to list() only for CSV/YAML. - _run_with_checkpoint: search loop does not accumulate results in memory. Add orjson as dependency for ~3-10x faster JSON serialization: - New limacharlie/json_compat.py module: unified API (dumps, dumps_pretty, loads, backend_name) with graceful fallback to stdlib json. - output.py: format_json, format_jsonl, _table_value, _csv_value all use json_compat. Benefits ALL CLI commands, not just search. - Debug log (--debug) shows which JSON backend is active. CLI improvements: - Add -h as alias for --help on all commands (context_settings). - Add help strings to all search subcommands (run, validate, estimate, saved-list, saved-get, saved-create, saved-delete, saved-run). - Fix checkpoint-show --checkpoint error to show "--checkpoint" not "checkpoint_path" in missing parameter message. - Warn on large time range searches (>7 days) without --checkpoint when using buffered output formats (table/CSV/YAML). Suggests --checkpoint or --output jsonl. Threshold configurable via _LARGE_TIME_RANGE_WARN_SECONDS. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add Python version classifiers and packaging tests Add PyPI classifiers for Python 3.9-3.14, development status, topic, and audience. CI already tests on Python 3.14 via cloudbuild_pr.yaml. Add packaging tests: classifiers present, current Python version included, requires-python minimum, production/stable status. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: update PyPI metadata - URLs, description, and changelog Add project URLs (Documentation, Repository, Issues, Changelog, REST API Docs) so links render on the PyPI page. Update description to better reflect the package scope. Add packaging tests for URL presence and HTTPS validation. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: split orjson markers for Python 3.9 compat, add multi-version CI dist checks orjson 3.11+ requires Python 3.10+, but we support 3.9. Split the dependency into two environment markers: - Python <3.10: orjson >=3.10.0,<3.11 (last series with 3.9 support) - Python >=3.10: orjson >=3.10.0 (latest) Add distribution install checks for Python 3.9-3.13 in CI (3.14 already covered by existing steps). All run in parallel. Python 3.9 step also verifies orjson 3.10.x is installed (not 3.11+) and runs the full unit test suite to catch syntax/compat issues. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * ci: run unit tests on all supported Python versions (3.9-3.14) in parallel Consolidate the separate "Unit Tests" and "Dist Check" steps into unified per-version steps that build, install, verify orjson, and run the full unit test suite. All 6 versions run in parallel. Use E2_HIGHCPU_8 machine type (8 vCPUs) to handle ~10 concurrent steps efficiently. Previously used the default E2_MEDIUM (2 vCPUs). Integration tests and benchmarks remain on Python 3.14 only since they test API behavior, not Python version compatibility. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: add future annotations import for Python 3.9 compat test_jwt_cache.py used `float | int` union syntax which requires Python 3.10+. Adding `from __future__ import annotations` makes it work on 3.9. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: user agent test compat with Python 3.9 platform.freedesktop_os_release was added in Python 3.10. Use create=True on mock.patch so the test works on 3.9 where the attribute doesn't exist. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * ci: separate dist and unit test steps, add echo banners for readability Split each Python version into separate "Dist" and "Unit Tests" steps for clearer CI output and easier debugging. Each step: Dist steps: build wheel in /tmp/build-<ver>, install, verify pip show, limacharlie --version, orjson backend. Clean isolation per step. Unit test steps: install from source with dev deps in /tmp/test-<ver>, run full pytest suite. Clean isolation per step. All steps use unique /tmp dirs to avoid cross-step interference. Added echo banners (======) and phase markers (--- phase ---) so CI logs are easy to scan. Also added sdist check as separate parallel step. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: add summary parameter to case create Mirror the ext-cases backend change that supports setting a summary at case creation time. The summary is included in the 'created' audit event metadata for D&R rules and webhooks. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: update CLI create tests for summary parameter, add CLI test The existing CLI tests asserted create_case was called without the summary kwarg, but the CLI now always passes summary=. Fix all 4 existing assertions and add a test for --summary. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Help topics were registered with plural names (e.g. "outputs", "sensors")
but the CLI commands use singular names ("output", "sensor"), so
`limacharlie help output` failed while `limacharlie help outputs` worked.
Renamed topic keys to match their CLI commands and added singular/plural
fallback in get_help_topic() and get_cheatsheet() so both forms work.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: consolidate config files into ~/.limacharlie.d/ directory
Unify all CLI config files (credentials, JWT cache, search checkpoints)
under a single directory with platform-appropriate defaults. CLI v2
(5.x) is the right time for this - establishing consistent conventions
before the public release rather than accumulating tech debt.
New layout:
- Unix: ~/.limacharlie.d/config.yaml, jwt_cache.json, search_checkpoints/
- Windows: %APPDATA%/limacharlie/config.yaml, jwt_cache.json, search_checkpoints/
Backward compatible: reads from legacy ~/.limacharlie with deprecation
warning. New env vars LC_CONFIG_DIR, LC_LEGACY_CONFIG. Existing
LC_CREDS_FILE continues to work. Migration via `limacharlie config migrate`.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* test: add microbenchmarks for path resolution and config loading overhead
Benchmarks cover cached (hot) and uncached (cold) path resolution,
config loading at various sizes, credential resolution, config writing,
migration overhead, and simulated CLI startup cost.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: address review feedback for config directory consolidation
- Fix priority inconsistency: get_config_dir() now checks LC_LEGACY_CONFIG
before LC_CONFIG_DIR, matching the priority order used by get_config_path()
and all other path functions. Previously, setting both env vars would cause
get_config_dir() and get_config_path() to disagree on where config lives.
- Fix JWT cache directory permissions: _save_cache() now uses
secure_makedirs() (0o700) instead of os.makedirs() (default 0o755) when
creating the parent directory. JWT cache contains auth tokens and should
have restricted permissions matching other config directories.
- Fix redundant exception catch: _safe_content_match() now catches only
OSError instead of (OSError, Exception), since OSError is already a
subclass of Exception.
- Extract duplicated _output helper: config_cmd.py and auth.py now import
from shared _output_helpers.py instead of duplicating the same 4-line
function.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* test: add tests for review feedback fixes, remove stdlib tests
- Add TestPriorityConsistency: verifies get_config_dir() and
get_config_path() agree on priority when both LC_LEGACY_CONFIG and
LC_CONFIG_DIR are set.
- Add TestSaveCacheDirectoryPermissions: verifies _save_cache() creates
parent directories with 0o700 permissions via secure_makedirs.
- Add TestSafeContentMatch: verifies _safe_content_match() handles
matching files, different files, missing files, and permission errors.
- Update test_multiple_env_vars_set_simultaneously to expect the fixed
priority behavior (legacy mode wins over LC_CONFIG_DIR in
get_config_dir).
- Remove 8 tests that exercised Python stdlib behavior rather than our
path resolution logic: spaces in paths, unicode paths, null bytes,
root path, very long paths, trailing slashes.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* test: add comprehensive security tests for permissions, symlinks, and races
New test classes in test_file_utils.py covering gaps identified in review:
Symlink rejection (_reject_symlink, safe_open_read, atomic_write):
- Relative symlinks (os.symlink("target", link))
- Chained symlinks (link -> link -> file)
- Circular/self-referencing symlinks
- Dangling symlinks on read path
- Symlinks in secure_makedirs path components
- No temp file leak on symlink rejection
Permission model (secure_makedirs, atomic_write):
- secure_makedirs tightens existing permissive (0o755, 0o777) dirs to 0o700
- All intermediate dirs created with 0o700 (not just leaf)
- atomic_write overwriting world-readable (0o644) or world-writable (0o666)
file results in 0o600
- safe_open_read does not alter file permissions
Race conditions / concurrency (atomic_write):
- Concurrent writers produce no partial reads (atomicity via os.replace)
- Final file after concurrent writes is valid (not a mix of two writers)
- Permissions preserved at 0o600 after concurrent writes
- No temp files left after concurrent writes complete
- os.replace replaces symlink entry itself, does not follow (TOCTOU defense)
Integration (config.save_config, config.load_config, jwt_cache.clear):
- save_config refuses to write through symlinked config.yaml
- load_config raises OSError on symlinked config.yaml (does not silently
return attacker-controlled data)
- clear_jwt_cache removes symlink itself, not the symlink target
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: skip permission test when running as root
Root bypasses file permission checks (0o000 is still readable), so the
test_unreadable_file_returns_false test fails in CI which runs as root.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* test: remove stdlib symlink tests, keep only our security primitive tests
Removed 14 tests that tested stdlib/OS behavior rather than our code:
- test_o_nofollow_rejects_symlink_at_kernel_level (tested os.open directly)
- test_os_replace_does_not_follow_symlinks (tested os.replace directly)
- _reject_symlink variant tests for relative/chained/circular/self-ref
symlinks (all exercise the same os.path.islink call)
- Duplicate symlink variant tests on atomic_write and safe_open_read
(relative, chained, dangling - same _reject_symlink code path)
- test_rejects_symlink_in_path_components (secure_makedirs delegates to
os.mkdir which follows symlinks - stdlib behavior)
- test_read_does_not_change_file_permissions (verifying absence of code
that doesn't exist)
Kept: all tests that exercise our security logic - _reject_symlink core
paths (file, parent dir, regular file accept, dangling), atomic_write
symlink rejection + temp cleanup, safe_open_read fd leak prevention,
permission tightening, concurrency, and integration tests.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs: add shell completion setup instructions to README Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: use one-time file-based completion install instead of eval Write completion scripts to standard shell-specific directories (~/.local/share/bash-completion/completions/, ~/.zfunc/, ~/.config/fish/completions/) so they are lazy-loaded by the shell rather than eval'd on every session start. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: document both eval and static file completion setup Default to eval approach (always up to date), with a subsection for static file-based install as an alternative for faster shell startup. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: lazy command loading for faster CLI startup Replace eager auto-discovery with lazy command loading using a static command-to-module map. Command modules are only imported when the specific command is invoked, not at CLI import time. Key changes: - _GlobalOptionsGroup replaced with _LazyCommandGroup that combines lazy loading with global option hoisting - Static _COMMAND_MODULE_MAP provides O(1) command name to module resolution without importing any command modules - list_commands() returns names from the static map (no imports) - get_command() imports only the requested module on first access - --ai-help injection deferred to per-command first access - Import __version__ from _version directly instead of client.py to avoid pulling in ssl, yaml, urllib at import time Performance (end-to-end subprocess, cold start): - limacharlie --version: ~55ms (was ~280ms) - limacharlie sensor list --help: ~67ms (was ~265ms) - limacharlie completion bash: ~58ms (was ~280ms) - limacharlie --help (all commands): ~134ms (loads all modules) Tests: - 816 regression tests covering full CLI surface: command registration, subcommand structure, module mapping, global options, --help for every command, --ai-help injection and output, explain registry, discovery profiles, completion, and context propagation - End-to-end subprocess benchmarks and regression tests - In-process microbenchmarks for import, help, completion, resolution Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: --ai-help shows all command groups with lazy loading _top_level_help() iterated cli.commands directly which is empty with lazy loading. Use list_commands() + get_command() instead, matching how Click itself resolves commands for --help. Also: remove dead pkgutil/field imports from cli.py, always emit stderr warning for broken command modules (not just with LC_DEBUG), fix weak test assertion, and add regression tests + microbenchmarks for --ai-help. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * perf: defer limacharlie.output import to cli callback limacharlie.output pulls in jmespath, tabulate, yaml, and csv which adds significant import overhead. Deferring the import from module level into the cli() callback avoids this cost on fast paths like --help, --version, and --ai-help that never render command output. Benchmark results (e2e subprocess, cold process): - cli import: 2.2ms -> 1.2ms (47% faster) - --version: 114ms -> 55ms (52% faster) - --help: 537ms -> 176ms (67% faster) - --ai-help: 481ms -> 205ms (57% faster) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…265) - Add config_cmd to _COMMAND_MODULE_MAP in cli.py (missing after #257 and #261 were merged independently) - Add config to EXPECTED_TOP_LEVEL_COMMANDS, EXPECTED_MODULE_MAP, and EXPECTED_SUBCOMMANDS in regression tests - Fix test_cli_import_does_not_load_output to handle third-party deps already loaded by other tests in the same pytest process - Add ci.yml GHA workflow that runs unit tests and dist checks on every push and PR Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The note shortcut command was passing 'note' as the hive name to the SDK's Hive class, which hits the REST API at /hive/note/... — but the backend hive is named 'org_notes'. This caused UNKNOWN_HIVE errors when using the SDK directly (not through the CLI gateway which may translate names). Also fixes the _KNOWN_HIVE_TYPES list and explain text. Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of the change
Temporary PR to trigger the test validations.
Type of change