feat: gRPC sync endpoint metrics

[alxckn]

This PR

Adds OpenTelemetry metrics for the gRPC flag sync service to enable monitoring of sync connections and operations.

Metrics

Custom Metrics:

Metric	Type	Description
feature_flag.flagd.sync.active_streams	Gauge	Currently active streaming connections
feature_flag.flagd.sync.stream.duration	Histogram	Duration of streaming connections (seconds)

Standard gRPC Metrics:

• Leverages otelgrpc.NewServerHandler() for comprehensive gRPC server metrics (request counts, latencies, status codes, etc.)

Changes

• Add custom sync metric definitions in core/pkg/telemetry/metrics.go
• Instrument SyncFlags streaming RPC with stream lifecycle metrics
• Enable standard gRPC metrics via OpenTelemetry gRPC instrumentation
• Add unit tests for metric collection, histogram buckets, and NoopMetricsRecorder

[netlify]

✅ Deploy Preview for polite-licorice-3db33c canceled.

Name	Link
🔨 Latest commit	8ab1c63
🔍 Latest deploy log	https://app.netlify.com/projects/polite-licorice-3db33c/deploys/69b9810e952c7f0008eed10d

[gemini-code-assist]

Summary of Changes

Hello @alxckn, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request enhances the observability of the gRPC flag synchronization endpoint by integrating comprehensive OpenTelemetry metrics. It provides critical insights into the performance and usage patterns of streaming connections and unary flag fetch operations, enabling better monitoring and troubleshooting of the synchronization service.

Highlights

• New gRPC Sync Metrics: Introduced new OpenTelemetry metrics for gRPC synchronization, including active streams, stream duration, events sent, and unary FetchAllFlags requests.
• Telemetry Interface Extension: The IMetricsRecorder interface and its NoopMetricsRecorder implementation have been extended to support the new gRPC sync metrics.
• gRPC Handler Instrumentation: The SyncFlags streaming handler and FetchAllFlags unary handler are now instrumented to record detailed metrics, including stream start/end, duration with exit reasons, and request status.
• Metrics Recorder Integration: The SyncService configuration now accepts a MetricsRecorder instance, allowing for the injection and use of the new telemetry capabilities.

Changelog

• core/pkg/telemetry/metrics.go
  • Added new constants for gRPC sync metric names (e.g., grpcSyncActiveStreamsMetric, grpcSyncStreamDurationMetric).
  • Extended IMetricsRecorder interface with new methods for gRPC sync operations (SyncStreamStart, SyncStreamEnd, SyncStreamDuration, SyncEventSent, FetchAllFlagsRequest).
  • Implemented the new IMetricsRecorder methods as no-ops in NoopMetricsRecorder.
  • Added new metric instruments (syncActiveStreams, syncStreamDuration, syncEventsSent, fetchAllFlagsRequests) to the MetricsRecorder struct.
  • Implemented the logic for recording these new metrics within the MetricsRecorder methods.
  • Configured a new OpenTelemetry view for grpcSyncStreamDurationMetric and initialized the new gRPC sync metric instruments in NewOTelRecorder.
• flagd/pkg/runtime/from_config.go
  • Modified FromConfig to pass the recorder (metrics recorder) to the SyncService configuration.
• flagd/pkg/service/flag-sync/handler.go
  • Imported telemetry and attribute packages.
  • Added metricsRecorder field to the syncHandler struct.
  • Instrumented the SyncFlags streaming handler to record SyncStreamStart, SyncStreamEnd, SyncStreamDuration (with exit reasons), and SyncEventSent metrics.
  • Instrumented the FetchAllFlags unary handler to record FetchAllFlagsRequest metrics, including request status (ok/error).
• flagd/pkg/service/flag-sync/handler_test.go
  • Imported telemetry package.
  • Updated test cases for TestSyncHandler_SyncFlags and TestSyncHandler_SelectorLocationPrecedence to inject a telemetry.NoopMetricsRecorder into the syncHandler for testing purposes.
• flagd/pkg/service/flag-sync/sync_service.go
  • Imported telemetry package.
  • Added MetricsRecorder field to the SvcConfigurations struct.
  • Modified NewSyncService to accept and pass the MetricsRecorder from SvcConfigurations to the syncHandler, defaulting to NoopMetricsRecorder if not provided.

Activity

• This pull request introduces a new feature related to gRPC sync endpoint metrics.
• It addresses and fixes issue #1234523.
• No specific human activity (e.g., comments, reviews) has been recorded for this pull request yet.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces metrics for the gRPC sync endpoint, which is a great addition for observability. The implementation is solid, adding new metrics, updating the recorder interface, and integrating metric collection into the sync handlers. I've suggested one refactoring to improve code clarity and reduce duplication in the FetchAllFlags handler. Overall, this is a well-executed feature.

[toddbaert]

Good idea @alxckn , but I think some of these are close to the OTel defaults. I propose we:

use otelgrpc.NewServerHandler() on the sync service's gRPC server. IIUC, this gives us automatically:

• rpc.server.duration
• rpc.server.request.size / rpc.server.response.size
• Message counts per method
• Error rates by code

custom from this PR, use:

• grpc.sync.active_streams
• grpc.sync.stream.duration

I think this will give us something more standard for the standard metrics, but also the really nice custom metrics you've invented that are specific to flagd.

WDYT?

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[alxckn]

@toddbaert @erka thank you for your reviews. I replaced custom metrics with oTel defaults and kept the 2 custom metrics that add value (renamed to be more coherent with custom metrics defined in the connect service).

Other review comments and sonar recommendations adressed.

[alxckn]

@toddbaert do you think this PR may be merged and released soon ? I see you requested additional reviews

I will take care of the conflicts

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud