feat: [metrics] Create metrics framework and add connection metrics by charlesdong1991 · Pull Request #407 · apache/fluss-rust

charlesdong1991 · 2026-03-01T17:50:09Z

Purpose

Linked issue: close #390

Brief change log

This PR adds a metrics framework and connection level metrics.

NOTE:
There is an parity from Java, which is the scopeguard for in-flight on cancellation, because Java callback model doesn't have equivalent to tokio future cancellation. And if a tokio future is dropped mid-execution, it would skip decrement and cause gauge drift. So i think it's important to add "scopeguard" to avoid that.

And I can have writer and scanner metrics as follow-up PRs.

Tests

All tests are passed locally.

fresh-borzoni

@charlesdong1991 Ty for the PR, left some comments
PTAL

crates/fluss/src/rpc/server_connection.rs

charlesdong1991 · 2026-03-04T20:38:15Z

Thanks for reviews, i made some changes and leave a question. PTAL @fresh-borzoni

fresh-borzoni

@charlesdong1991 LGTM overall, small comment about misleading doc, and we can do simple math to match Java here, at least semantically. PTAL

crates/fluss/src/metrics.rs

charlesdong1991 · 2026-03-05T19:47:59Z

and we can do simple math to match Java here, at least semantically. PTAL

btw, just to clarify before making changes, do you mean to update code to fix header overhead? or to update the doc to reflect this? @fresh-borzoni

fresh-borzoni · 2026-03-05T20:24:04Z

and we can do simple math to match Java here, at least semantically. PTAL

btw, just to clarify before making changes, do you mean to update code to fix header overhead? or to update the doc to reflect this? @fresh-borzoni

Sorry, I should have been more specific - I meant code @charlesdong1991

charlesdong1991 · 2026-03-05T20:33:44Z

Gotcha, thanks @fresh-borzoni will do a quick change, should be straightforward 👍

charlesdong1991 · 2026-03-05T20:45:22Z

i updated code to calculate total size "manually", PTAL @fresh-borzoni

thanks again!

fresh-borzoni · 2026-03-06T09:02:42Z

@charlesdong1991 TY, LGTM

charlesdong1991 · 2026-03-08T11:05:08Z

would be great if you can take a look! Thanks! @luoyuxia

Copilot

Pull request overview

Adds a client-side metrics framework (via the metrics facade) and instruments RPC connection requests to emit per-API-key counters, gauges, and latency histograms (issue #390).

Changes:

Introduces crates/fluss/src/metrics.rs with metric name constants and an api_key label filter for reportable APIs.
Instruments ServerConnectionInner::request to record request/response counts, bytes sent/received, in-flight gauge, and request latency (with drop-based cleanup on cancellation).
Adds unit/integration-style tests around connection metrics emission and failure paths; wires in metrics/metrics-util dependencies.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
crates/fluss/src/rpc/server_connection.rs	Emits connection-level metrics around request lifecycle; adds tests validating metrics behavior (success, send failure, API error).
crates/fluss/src/rpc/mod.rs	Re-exports `ApiKey` as `pub(crate)` to support metrics helpers.
crates/fluss/src/rpc/message/header.rs	Exposes `REQUEST_HEADER_LENGTH` for consistent byte accounting.
crates/fluss/src/metrics.rs	New metrics constants + `api_key` label helper and tests.
crates/fluss/src/lib.rs	Exposes `metrics` module from the crate root.
crates/fluss/Cargo.toml	Adds `metrics` dependency and `metrics-util` for dev/test support.
Cargo.toml	Adds workspace-level `metrics` dependency version.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

crates/fluss/src/rpc/server_connection.rs

Copilot

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

crates/fluss/src/rpc/server_connection.rs

crates/fluss/src/metrics.rs

charlesdong1991 · 2026-03-10T19:36:47Z

can i get another round of reviews? @luoyuxia @fresh-borzoni thanks!

fresh-borzoni

@charlesdong1991 Looked through one more time, LGTM

charlesdong1991 · 2026-03-14T11:08:35Z

Just rebased, it would be great if you can take a look as well! @luoyuxia @leekeiabstraction 🙏

Thank you!

leekeiabstraction

TY for the PR! Left a couple of small comments

leekeiabstraction · 2026-03-15T09:57:23Z

crates/fluss/src/metrics.rs

+        assert_eq!(find_counter(CLIENT_RESPONSES_TOTAL), Some(1));
+        assert_eq!(find_counter(CLIENT_BYTES_SENT_TOTAL), Some(256));
+        assert_eq!(find_counter(CLIENT_BYTES_RECEIVED_TOTAL), Some(128));
+        assert_eq!(find_histogram(CLIENT_REQUEST_LATENCY_MS), Some(vec![42.5]));


Seems like we're missing a couple of assertions on gauge metrics?

There is only 1 gauge metric now, and have tested in inflight_gauge_nets_to_zero_after_balanced_calls, but i added also here for completeness

Thanks!

crates/fluss/src/metrics.rs

charlesdong1991 · 2026-03-15T14:37:53Z

Thanks for the callout on tests @leekeiabstraction Addressed both! PTAL! cc @luoyuxia

leekeiabstraction

TY for the revision. Added a couple more comments.

(Sorry for reviewing in waves, don't have my computer with me atm so it's easy to miss things reviewing on the phone).

crates/fluss/src/rpc/server_connection.rs

charlesdong1991 · 2026-03-17T21:02:43Z

thanks for suggestions! @leekeiabstraction updated

leekeiabstraction

Approved. TY for your contribution!

charlesdong1991 added 2 commits March 1, 2026 18:40

Create metrics framework and add connection metrics

d62a1f8

Merge remote-tracking branch 'upstream/main' into metrics-framework

666bb78

fresh-borzoni reviewed Mar 3, 2026

View reviewed changes

crates/fluss/src/rpc/server_connection.rs Show resolved Hide resolved

crates/fluss/src/rpc/server_connection.rs Outdated Show resolved Hide resolved

crates/fluss/src/rpc/server_connection.rs Outdated Show resolved Hide resolved

charlesdong1991 added 2 commits March 4, 2026 21:01

Merge remote-tracking branch 'upstream/main' into metrics-framework

4dade11

address review feedback

0345c69

fresh-borzoni reviewed Mar 4, 2026

View reviewed changes

crates/fluss/src/metrics.rs Outdated Show resolved Hide resolved

charlesdong1991 added 2 commits March 5, 2026 13:59

Update comment to reflect correct behaviour

2c88f31

Merge remote-tracking branch 'upstream/main' into metrics-framework

7003c57

update response byte and request byte to exclude header and framing

44c4474

fresh-borzoni approved these changes Mar 6, 2026

View reviewed changes

Merge remote-tracking branch 'upstream/main' into metrics-framework

c197654

luoyuxia requested a review from Copilot March 9, 2026 01:37

Copilot started reviewing on behalf of luoyuxia March 9, 2026 01:37 View session

Copilot AI reviewed Mar 9, 2026

View reviewed changes

crates/fluss/src/rpc/server_connection.rs Outdated Show resolved Hide resolved

charlesdong1991 added 2 commits March 9, 2026 20:36

update error message and fix format

1d91fab

Merge remote-tracking branch 'upstream/main' into metrics-framework

98eb598

charlesdong1991 requested a review from Copilot March 9, 2026 20:16

Copilot started reviewing on behalf of charlesdong1991 March 9, 2026 20:17 View session

Copilot AI reviewed Mar 9, 2026

View reviewed changes

crates/fluss/src/rpc/server_connection.rs Show resolved Hide resolved

crates/fluss/src/rpc/server_connection.rs Outdated Show resolved Hide resolved

crates/fluss/src/metrics.rs Show resolved Hide resolved

update tests

9ab7f84

fresh-borzoni approved these changes Mar 11, 2026

View reviewed changes

Merge remote-tracking branch 'upstream/main' into metrics-framework

2aee3e2

leekeiabstraction reviewed Mar 15, 2026

View reviewed changes

enhance testing

063b9f5

leekeiabstraction reviewed Mar 15, 2026

View reviewed changes

crates/fluss/src/rpc/server_connection.rs Outdated Show resolved Hide resolved

crates/fluss/src/rpc/server_connection.rs Outdated Show resolved Hide resolved

charlesdong1991 added 2 commits March 17, 2026 22:01

address comments

11a8eeb

Merge remote-tracking branch 'upstream/main' into metrics-framework

5b0008c

leekeiabstraction approved these changes Mar 17, 2026

View reviewed changes

Conversation

charlesdong1991 commented Mar 1, 2026

Purpose

Brief change log

Tests

Uh oh!

fresh-borzoni left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

charlesdong1991 commented Mar 4, 2026

Uh oh!

fresh-borzoni left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

charlesdong1991 commented Mar 5, 2026

Uh oh!

fresh-borzoni commented Mar 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

charlesdong1991 commented Mar 5, 2026

Uh oh!

charlesdong1991 commented Mar 5, 2026

Uh oh!

fresh-borzoni commented Mar 6, 2026

Uh oh!

charlesdong1991 commented Mar 8, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

charlesdong1991 commented Mar 10, 2026

Uh oh!

fresh-borzoni left a comment

Choose a reason for hiding this comment

Uh oh!

charlesdong1991 commented Mar 14, 2026

Uh oh!

leekeiabstraction left a comment

Choose a reason for hiding this comment

Uh oh!

leekeiabstraction Mar 15, 2026

Choose a reason for hiding this comment

Uh oh!

charlesdong1991 Mar 15, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

charlesdong1991 commented Mar 15, 2026

Uh oh!

leekeiabstraction left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

charlesdong1991 commented Mar 17, 2026

Uh oh!

leekeiabstraction left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

fresh-borzoni left a comment •

edited

Loading

fresh-borzoni commented Mar 5, 2026 •

edited

Loading