Skip to content

feat: adding additional fine grained spans#142

Open
chatton wants to merge 10 commits intomainfrom
cian/add-tier1-tracing-spans
Open

feat: adding additional fine grained spans#142
chatton wants to merge 10 commits intomainfrom
cian/add-tier1-tracing-spans

Conversation

@chatton
Copy link
Contributor

@chatton chatton commented Feb 25, 2026

Description

Follow up to #137

Adding additional spans.

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update
  • Performance improvement
  • Refactoring

Related Issues

Fixes #(issue)

Checklist

  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • Any dependent changes have been merged and published

Testing

Additional Notes

Summary by CodeRabbit

  • Chores

    • Enhanced observability with span-based timing and metadata capture across payload building, validation, and parsing flows.
    • Added automatic duration tracking that reliably records elapsed time even on early returns.
    • Standardized informational logging for payload and validation operations.
  • Tests

    • Added tests verifying spans emit expected fields (ids, counts, durations) and that instrumentation produces the expected span metadata.

@coderabbitai
Copy link

coderabbitai bot commented Feb 25, 2026

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

Add a crate-local drop-based duration recorder and instrument multiple node components with tracing spans that record timing and metadata (duration_ms, tx counts, payload_id, block/parent hashes); update logging and unit tests to assert span fields.

Changes

Cohort / File(s) Summary
Tracing utility
crates/node/src/tracing_ext.rs, crates/node/src/lib.rs
Introduce pub(crate) RecordDurationOnDrop::new() and Drop impl; register tracing_ext module. Records duration_ms on span drop.
Payload attributes & builder
crates/node/src/attributes.rs, crates/node/src/builder.rs, crates/node/src/payload_service.rs
Add spans/instrumentation to payload builder paths; replace manual timing with RecordDurationOnDrop; record fields like parent_hash, raw_tx_count, gas_limit, payload_id; update logging and add span-field tests.
Transaction pool validation
crates/node/src/txpool.rs
Instrument validate_evnode and validate_transaction with spans and RecordDurationOnDrop; record tx_hash, is_evnode, duration_ms; add span assertions in tests.
Payload parsing & validation
crates/node/src/validator.rs
Add #[instrument] and duration guard to ensure_well_formed_payload and parse_evolve_payload; record block_hash, tx_count, block_number, duration_ms; update tests to assert span fields.
Tests & assertions
crates/node/src/... (tests across modified modules)
Extend/add unit tests using span collection to assert presence and values of fields including duration_ms across payload builder, payload service, txpool, and validator tests.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Poem

"I nibble code with whiskers bright,
I time each hop from left to right,
I plant a span on every trail,
duration_ms tells the tale,
A rabbit's hop makes metrics light." 🐇

🚥 Pre-merge checks | ✅ 3
✅ Passed checks (3 passed)
Check name Status Explanation
Title check ✅ Passed The title accurately describes the main change: adding fine-grained tracing spans throughout the codebase for improved observability.
Description check ✅ Passed The description is minimal but structurally complete. It identifies the PR as a new feature with a reference to related PR #137, though testing details and additional notes sections are empty.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch cian/add-tier1-tracing-spans

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@claude
Copy link

claude bot commented Feb 25, 2026

Claude finished @chatton's task —— View job


Code Review: Fine-Grained Tracing Spans

I've completed a comprehensive review of this PR that adds observability improvements through tracing spans. Here's my detailed feedback:

Todo List:

  • Read and analyze all changed files
  • Review the RecordDurationOnDrop utility design
  • Evaluate instrumentation patterns across components
  • Check test coverage and quality
  • Assess performance and security implications
  • Provide comprehensive feedback

✅ Overall Assessment

This is a well-structured observability enhancement that adds valuable tracing spans across the payload building, validation, and transaction pool flows. The implementation is consistent and follows good patterns.

🎯 Code Quality & Best Practices

Excellent Design Patterns

  • Consistent instrumentation: All instrumented functions follow the same pattern with #[instrument(...)] and RecordDurationOnDrop
  • Structured fields: Good use of structured logging with relevant fields like tx_hash, parent_hash, payload_id
  • Drop guard pattern: The RecordDurationOnDrop utility ensures timing is captured even on early returns

🔧 Areas for Improvement

1. Test Assertion Quality (crates/node/src/attributes.rs:227)

// Current - only checks field existence
assert!(span.has_field("duration_ms"), "span missing duration_ms field");

// Better - verify actual value was recorded
let duration = span.field("duration_ms").expect("duration_ms field should exist");
assert!(duration.is_some(), "duration_ms should have a value");

2. Duration Recording on Error Paths (crates/node/src/builder.rs:77)
The current pattern records duration only at function end. If functions return early via ? operators, duration_ms remains Empty. Consider the pattern already used in txpool.rs:

// Good example from txpool.rs - records duration on all paths
fn validate_evnode_inner(&self, ...) -> Result<_, _> {
    // actual implementation
}

#[instrument(...)]
fn validate_evnode(&self, ...) -> Result<_, _> {
    let _duration = RecordDurationOnDrop::new();
    self.validate_evnode_inner(...)
}

3. Test Code Duplication
The test setup in payload_service.rs is duplicated across multiple tests. Consider extracting to a helper function.

🚀 Performance Considerations

Positive:

  • Tracing overhead is minimal with proper subscriber configuration
  • RecordDurationOnDrop uses Instant which is efficient
  • Structured fields avoid string formatting in hot paths

Neutral:

  • The span creation overhead is acceptable for these operations
  • Duration calculation happens only once per span via drop guard

🔒 Security Assessment

✅ Secure: No security concerns identified

  • No sensitive data in span fields
  • Tracing fields are properly structured
  • No injection risks in logged values

🧪 Test Coverage

Strong coverage:

  • Tests verify span creation and field presence across all components
  • Good use of SpanCollector for test assertions
  • Covers both success and error scenarios

Improvement needed:

  • Tests should verify duration values are actually recorded, not just field existence
  • Consider testing span hierarchy and relationships

📊 Specific File Analysis

crates/node/src/tracing_ext.rs

Elegant and simple design. The drop guard pattern ensures reliability.

crates/node/src/builder.rs

Good instrumentation of payload building with relevant context fields.

crates/node/src/payload_service.rs

Proper span instrumentation for both try_build and build_empty_payload methods.

crates/node/src/txpool.rs

Excellent example of comprehensive validation instrumentation with proper error handling.

crates/node/src/validator.rs

Good coverage of validation flows with structured context.

crates/node/src/attributes.rs

Clean attribute processing instrumentation.

🎯 Recommendations

  1. Enhance test assertions to verify duration values are populated
  2. Consider the inner/outer function pattern for functions with early returns to ensure duration is always recorded
  3. Extract test setup helpers to reduce code duplication
  4. Consider adding span relationships for better tracing visualization

✨ Positive Highlights

  • Consistent pattern application across all components
  • Thoughtful field selection providing valuable operational insights
  • Comprehensive test coverage ensuring instrumentation works
  • Clean implementation that doesn't clutter the business logic
  • Good documentation in span field names and structure

This PR significantly improves the observability of ev-reth's payload building pipeline and will be valuable for debugging and performance analysis in production.


@chatton chatton changed the title [FEATURE]: adding additional fine grained spans feat: adding additional fine grained spans Feb 25, 2026
@chatton chatton mentioned this pull request Feb 25, 2026
8 tasks
@chatton chatton marked this pull request as ready for review February 25, 2026 09:09
@chatton chatton requested a review from a team as a code owner February 25, 2026 09:09
Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (2)
crates/node/src/payload_service.rs (1)

396-476: Consider extracting shared test setup into a helper.

The genesis/provider/config/builder setup (lines 401–446) is nearly identical to the try_build_span_has_expected_fields test above (lines 312–357), and also mirrors setup in builder.rs and validator.rs tests. This boilerplate is copy-pasted across at least 6 test functions in this PR.

A shared test helper (e.g., in test_utils.rs) returning (EvolveEnginePayloadBuilder<MockEthProvider>, Header, B256) would reduce duplication and make future test additions easier.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@crates/node/src/payload_service.rs` around lines 396 - 476, Extract the
repeated genesis/provider/config/builder setup into a shared test helper (e.g.,
in a new test_utils.rs) that returns the needed items like
(EvolveEnginePayloadBuilder<MockEthProvider>, Header, B256, ChainSpec or config)
and update tests such as build_empty_payload_span_has_expected_fields and
try_build_span_has_expected_fields to call that helper instead of duplicating
lines ~401–446; locate the setup by looking for MockEthProvider,
EvolveEnginePayloadBuilder, RpcPayloadAttributes, SealedHeader and PayloadConfig
usage, move the creation of
genesis_hash/genesis_header/provider/config/evm_config/evolve_builder into the
helper, and have tests use the returned tuple to construct their specific Attrs
or PayloadConfig before calling build_empty_payload or other builder methods.
crates/node/src/builder.rs (1)

65-76: duration_ms won't be recorded on error paths.

If build_payload returns early via any of the ? operators (lines 78–184), duration_ms stays Empty on the span. This means error cases won't have timing data, which can be useful for diagnosing slow failures. Consider recording duration_ms unconditionally, e.g., with a drop guard or by restructuring to always record before returning.

This pattern is repeated across all files in this PR, so noting it here as the primary location.

Example: use a scope guard to always record duration

One approach is a small helper or a defer-style pattern:

     pub async fn build_payload(
         &self,
         attributes: EvolvePayloadAttributes,
     ) -> Result<SealedBlock<ev_primitives::Block>, PayloadBuilderError> {
         let _start = std::time::Instant::now();
+        // Record duration on all exit paths (success and error).
+        let _record_duration = scopeguard::guard((), |_| {
+            Span::current().record("duration_ms", _start.elapsed().as_millis() as u64);
+        });

Alternatively, the inner/outer pattern already used in validate_evnode / validate_evnode_inner in txpool.rs would work here too.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@crates/node/src/builder.rs` around lines 65 - 76, The span field duration_ms
(declared in the #[instrument(...)] on build_payload) is left Empty on
early-return/error paths because duration is only computed at function end; add
an unconditional recorder (e.g., a small Drop guard created right after let
_start = Instant::now() or restructure into inner/outer functions) that computes
Instant::elapsed() and calls tracing::Span::current().record("duration_ms",
&elapsed.as_millis()) so duration_ms is always set even when build_payload (or
any early-return using ?) exits early; apply the same pattern wherever this
#[instrument(..., duration_ms = tracing::field::Empty, ...)] pattern is used.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@crates/node/src/builder.rs`:
- Around line 65-76: The span field duration_ms (declared in the
#[instrument(...)] on build_payload) is left Empty on early-return/error paths
because duration is only computed at function end; add an unconditional recorder
(e.g., a small Drop guard created right after let _start = Instant::now() or
restructure into inner/outer functions) that computes Instant::elapsed() and
calls tracing::Span::current().record("duration_ms", &elapsed.as_millis()) so
duration_ms is always set even when build_payload (or any early-return using ?)
exits early; apply the same pattern wherever this #[instrument(..., duration_ms
= tracing::field::Empty, ...)] pattern is used.

In `@crates/node/src/payload_service.rs`:
- Around line 396-476: Extract the repeated genesis/provider/config/builder
setup into a shared test helper (e.g., in a new test_utils.rs) that returns the
needed items like (EvolveEnginePayloadBuilder<MockEthProvider>, Header, B256,
ChainSpec or config) and update tests such as
build_empty_payload_span_has_expected_fields and
try_build_span_has_expected_fields to call that helper instead of duplicating
lines ~401–446; locate the setup by looking for MockEthProvider,
EvolveEnginePayloadBuilder, RpcPayloadAttributes, SealedHeader and PayloadConfig
usage, move the creation of
genesis_hash/genesis_header/provider/config/evm_config/evolve_builder into the
helper, and have tests use the returned tuple to construct their specific Attrs
or PayloadConfig before calling build_empty_payload or other builder methods.

ℹ️ Review info

Configuration used: defaults

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between ecfefef and c840315.

📒 Files selected for processing (5)
  • crates/node/src/attributes.rs
  • crates/node/src/builder.rs
  • crates/node/src/payload_service.rs
  • crates/node/src/txpool.rs
  • crates/node/src/validator.rs

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 4

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@crates/node/src/attributes.rs`:
- Around line 215-227: The test currently only checks
span.has_field("duration_ms") which only verifies presence, not that the drop
guard populated a value; update the assertion to fetch the "duration_ms" field
from span (e.g., via span.field("duration_ms") or the test helper used to read
fields) and assert the returned value is Some and contains a numeric duration
(for example not null and > 0 or at least >= 0) so the test guarantees the
duration was actually recorded by the drop guard.

In `@crates/node/src/payload_service.rs`:
- Around line 23-24: The file crates/node/src/payload_service.rs is failing
formatting; run rustfmt by executing `cargo fmt --all` (or `rustfmt` on that
file) to normalize spacing, import ordering and line breaks — specifically
reformat the use statements such as `use
crate::tracing_ext::RecordDurationOnDrop;` and `use tracing::{info,
instrument};` and any surrounding function/struct definitions so the file passes
cargo fmt formatting checks.

In `@crates/node/src/txpool.rs`:
- Around line 414-418: The file fails rustfmt formatting; run rustfmt/cargo fmt
to reformat the affected code block containing the #[instrument(...)] attribute
(the attribute applied to the txpool method with fields
tx_hash/is_evnode/duration_ms) and re-run cargo fmt --all --check, then commit
the formatted changes so the formatting diff in crates/node/src/txpool.rs is
resolved.

In `@crates/node/src/validator.rs`:
- Around line 23-25: The file fails rustfmt; run the Rust formatter and commit
the formatted changes for crates/node/src/validator.rs (e.g., run cargo fmt
--all or rustfmt on that file) so imports like
crate::tracing_ext::RecordDurationOnDrop and tracing::{debug, info, instrument,
Span} and the rest of validator.rs conform to the project's formatting rules and
the CI formatting check passes.

ℹ️ Review info

Configuration used: defaults

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between ec5bcb0 and 6ea5e78.

📒 Files selected for processing (7)
  • crates/node/src/attributes.rs
  • crates/node/src/builder.rs
  • crates/node/src/lib.rs
  • crates/node/src/payload_service.rs
  • crates/node/src/tracing_ext.rs
  • crates/node/src/txpool.rs
  • crates/node/src/validator.rs

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
crates/node/src/payload_service.rs (1)

394-474: Consider extracting shared test setup into a helper function.

The test correctly validates the span fields for build_empty_payload. However, lines 399-444 duplicate nearly identical setup code from try_build_span_has_expected_fields (lines 310-355). Extracting this into a helper would improve maintainability.

♻️ Suggested helper extraction
// Add a helper at the top of the tests module:
fn create_test_engine_builder() -> (
    SpanCollector,
    tracing::subscriber::DefaultGuard,
    EvolveEnginePayloadBuilder<MockEthProvider>,
    B256,
    Header,
) {
    let collector = SpanCollector::new();
    let guard = collector.as_default();

    let genesis: alloy_genesis::Genesis =
        serde_json::from_str(include_str!("../../tests/assets/genesis.json"))
            .expect("valid genesis");
    let chain_spec = Arc::new(
        ChainSpecBuilder::default()
            .chain(reth_chainspec::Chain::from_id(1234))
            .genesis(genesis)
            .cancun_activated()
            .build(),
    );

    let provider = MockEthProvider::default();
    let genesis_hash = B256::from_slice(
        &hex::decode("2b8bbb1ea1e04f9c9809b4b278a8687806edc061a356c7dbc491930d8e922503")
            .unwrap(),
    );
    let genesis_state_root = B256::from_slice(
        &hex::decode("05e9954443da80d86f2104e56ffdfd98fe21988730684360104865b3dc8191b4")
            .unwrap(),
    );

    let genesis_header = Header {
        state_root: genesis_state_root,
        number: 0,
        gas_limit: 30_000_000,
        timestamp: 1710338135,
        base_fee_per_gas: Some(0),
        excess_blob_gas: Some(0),
        blob_gas_used: Some(0),
        parent_beacon_block_root: Some(B256::ZERO),
        ..Default::default()
    };
    provider.add_header(genesis_hash, genesis_header.clone());

    let config = EvolvePayloadBuilderConfig::from_chain_spec(chain_spec.as_ref()).unwrap();
    let evm_config = EvolveEvmConfig::new(chain_spec);
    let evolve_builder = Arc::new(EvolvePayloadBuilder::new(
        Arc::new(provider),
        evm_config,
        config.clone(),
    ));

    let engine_builder = EvolveEnginePayloadBuilder {
        evolve_builder,
        config,
    };

    (collector, guard, engine_builder, genesis_hash, genesis_header)
}
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@crates/node/src/payload_service.rs` around lines 394 - 474, Extract the
duplicated test setup into a helper (e.g. create_test_engine_builder) and use it
from both build_empty_payload_span_has_expected_fields and
try_build_span_has_expected_fields: move the SpanCollector creation, genesis
deserialization, ChainSpecBuilder, MockEthProvider setup (including
provider.add_header), genesis_header/hash,
EvolvePayloadBuilderConfig/EvolveEvmConfig/EvolvePayloadBuilder instantiation
and EvolveEnginePayloadBuilder construction into the helper and return the
needed values (SpanCollector and its guard, EvolveEnginePayloadBuilder,
genesis_hash, genesis_header); then replace the duplicated block in both tests
with a call to create_test_engine_builder and use the returned symbols when
building payloads and assertions (match names like SpanCollector,
EvolveEnginePayloadBuilder, genesis_hash, genesis_header, payload_config).
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@crates/node/src/builder.rs`:
- Around line 1-2: The import grouping is misformatted: move the standalone `use
crate::tracing_ext::RecordDurationOnDrop;` into the same `crate::` import group
as `EvolvePayloadBuilderConfig` and `EvEvmConfig` (or otherwise group all
`crate::...` uses together) so the file's imports conform to rustfmt
expectations; alternatively run `cargo fmt --all` to auto-fix.

---

Nitpick comments:
In `@crates/node/src/payload_service.rs`:
- Around line 394-474: Extract the duplicated test setup into a helper (e.g.
create_test_engine_builder) and use it from both
build_empty_payload_span_has_expected_fields and
try_build_span_has_expected_fields: move the SpanCollector creation, genesis
deserialization, ChainSpecBuilder, MockEthProvider setup (including
provider.add_header), genesis_header/hash,
EvolvePayloadBuilderConfig/EvolveEvmConfig/EvolvePayloadBuilder instantiation
and EvolveEnginePayloadBuilder construction into the helper and return the
needed values (SpanCollector and its guard, EvolveEnginePayloadBuilder,
genesis_hash, genesis_header); then replace the duplicated block in both tests
with a call to create_test_engine_builder and use the returned symbols when
building payloads and assertions (match names like SpanCollector,
EvolveEnginePayloadBuilder, genesis_hash, genesis_header, payload_config).

ℹ️ Review info

Configuration used: defaults

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 6ea5e78 and 0bf3609.

📒 Files selected for processing (4)
  • crates/node/src/builder.rs
  • crates/node/src/payload_service.rs
  • crates/node/src/txpool.rs
  • crates/node/src/validator.rs
🚧 Files skipped from review as they are similar to previous changes (1)
  • crates/node/src/txpool.rs

Copy link
Contributor

@randygrok randygrok left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

only pending merge conflicts

…g-spans

# Conflicts:
#	crates/node/src/txpool.rs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants