caching: export vlog mmap telemetry via expvar#830
caching: export vlog mmap telemetry via expvar#830snissn wants to merge 6 commits intopr/vlog-mmap-sealed-byte-budgetfrom
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6339068dba
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
There was a problem hiding this comment.
Pull request overview
This PR adds an expvar telemetry bridge for TreeDB’s caching DB so live mmap residency/budget counters (and selected process memory stats) can be inspected via /debug/vars for runtime forensics.
Changes:
- Publish a top-level
expvarvariable namedtreedbthat exports a filtered subset ofdb.Stats()keys (treedb.cache.vlog_mmap.*andtreedb.process.memory.*). - Track the “current” open caching
DBfor expvar export and clear it onDB.Close(). - Add unit tests for key filtering and string→typed-value coercion.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 5 comments.
| File | Description |
|---|---|
| TreeDB/caching/expvar_stats.go | Implements expvar publishing, current-DB tracking, filtering, and type coercion. |
| TreeDB/caching/expvar_stats_test.go | Adds tests for filtering and coercion behavior. |
| TreeDB/caching/db.go | Hooks registration into Open() and unregistration into Close(). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
You can also share your feedback on Copilot code review. Take the survey.
|
Follow-up full validation on Runs
ImplicationThe expvar bridge works and is now directly observable during runs. In these two runs, file-backed RSS is not being driven by live valuelog mmaps ( |
There was a problem hiding this comment.
Pull request overview
Adds an expvar bridge in TreeDB/caching to publish a selected subset of caching.DB runtime stats (focused on value-log mmap and process memory) under the top-level treedb variable, enabling live scraping via /debug/vars.
Changes:
- Introduces
TreeDB/caching/expvar_stats.goto publish a filtered/coerced stats payload viaexpvar. - Registers/unregisters the “current DB” for expvar export from
caching.Open/(*DB).Close. - Adds unit tests for the stat filtering/coercion helper.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| TreeDB/caching/expvar_stats.go | Publishes treedb expvar var and filters/coerces selected db.Stats() keys. |
| TreeDB/caching/expvar_stats_test.go | Adds tests for expvar stat selection/coercion behavior. |
| TreeDB/caching/db.go | Hooks expvar registration into Open and cleanup into Close. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
You can also share your feedback on Copilot code review. Take the survey.
There was a problem hiding this comment.
Pull request overview
Adds an expvar telemetry bridge for TreeDB caching so selected runtime stats (vlog mmap + process memory) are available live under /debug/vars as treedb, tracking the currently-open caching DB and clearing on close.
Changes:
- Introduce
expvarpublisher that exports a filtered/coerced subset ofDB.Stats()undertreedb. - Wire DB lifecycle hooks to register/unregister the currently active DB for export.
- Add unit tests for filtering and type coercion behavior.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 5 comments.
| File | Description |
|---|---|
| TreeDB/caching/expvar_stats.go | Implements expvar publishing, DB registration, filtering, and value coercion. |
| TreeDB/caching/db.go | Registers the opened DB for expvar export and unregisters on close. |
| TreeDB/caching/expvar_stats_test.go | Adds tests validating filtering and coercion for selected stats. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
You can also share your feedback on Copilot code review. Take the survey.
… into pr/vlog-mmap-expvar-telemetry
There was a problem hiding this comment.
Pull request overview
This PR adds an expvar bridge in the caching layer to expose a focused subset of TreeDB runtime stats (notably value-log mmap telemetry and process memory stats) under the top-level treedb key, enabling live forensics via /debug/vars.
Changes:
- Add
TreeDB/caching/expvar_stats.goto publishexpvarvartreedb, select onlytreedb.cache.vlog_mmap.*andtreedb.process.memory.*, and coerce stat values into numeric/bool types where possible. - Register the active open caching
*DBfor expvar export onOpen, and clear it onClose. - Add unit tests for stat selection/filtering and basic type coercion.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| TreeDB/caching/expvar_stats.go | Implements expvar publishing, current-DB tracking, stat key filtering, and string→typed coercion. |
| TreeDB/caching/expvar_stats_test.go | Tests filtering and coercion behavior for the expvar stat selection helper. |
| TreeDB/caching/db.go | Hooks expvar DB registration into Open() and unregister into Close(). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
You can also share your feedback on Copilot code review. Take the survey.
Summary
expvarundertreedbso forensics can read mmap telemetry live from/debug/varstreedb.cache.vlog_mmap.*andtreedb.process.memory.*Why
PR #829 added sealed mmap byte budgeting and deny-reason counters, but celestia forensics could not directly collect those counters during a run. This adds a low-risk telemetry bridge so we can correlate:
treedb.cache.vlog_mmap.active_bytes/dead_bytes/...smaps_rollupPss_Filewithout changing storage semantics.
Validation
go test ./TreeDB/caching -run 'Expvar|SelectTreeDBExpvarStats' -count=1go test ./TreeDB/internal/valuelog -run 'SealedLazyMmap|MmapResidencyStatsAggregatesCounters' -count=1go test ./TreeDB/db -run TestDoesNotExist -count=1go vet ./TreeDB/caching ./TreeDB/internal/valuelog ./TreeDB/dbfast, interrupted after startup):curl http://localhost:6062/debug/varsnow includes top-leveltreedbjq '.treedb|keys[]'includes:treedb.cache.vlog_mmap.active_bytestreedb.cache.vlog_mmap.dead_bytestreedb.cache.vlog_mmap.sealed_map_denied.count_captreedb.cache.vlog_mmap.sealed_map_denied.bytes_cap