feat: enhance observability of flag updates, and especially sync streams

[aepfli]

This pull request introduces tracing capabilities to the synchronization streams within the flagd service. By integrating OpenTelemetry tracing into the gRPC sync mechanisms, we aim to enhance observability and facilitate easier debugging and performance monitoring.

A span is started as soon as we start emitting and updating a Flagset/configuration.

Furthermore, we improve the emission of updates from the sync server. As spans are not functional for gRPC messages, we are adding the spanId and traceId to the flag set metadata. This allows us to create child spans on all services receiving these updates. Furthermore, we can even link flag evaluations via traces to the correlating update event, generating a deep insight into propagation and events.

This adds the traceId and the spanId as metadata to the flagset, and will allow us to link flag evaluations back to flagset updates.

next steps

• Define if flagd providers should enhance the event details metadata emitted on a CONFIGURATION_CHANGED event with flagset metadata like version, flagset-id, propagationSpanId, and propagationTraceId
• Depending on the task above, add a Gherkin test suite to verify the behavior
• Implement based on Gherkin suite

[netlify]

✅ Deploy Preview for polite-licorice-3db33c canceled.

Name	Link
🔨 Latest commit	f11896b
🔍 Latest deploy log	https://app.netlify.com/projects/polite-licorice-3db33c/deploys/683d39069fc65b000813486e

[aepfli]

@beeme1mr this would be the same thing as for ofrep, but excluding the stream endpoint, as this is a long span - we could manually create a span per each configuration update, not sure if that is worth it

[aepfli]

do we want to keep this, or should we close this pr @beeme1mr

[toddbaert]

I'll check this out this weekend!

[toddbaert]

I think a more consistent attr name would be feature_flag.set.change_count or feature_flag.set.flags_changed_count. We already use feature_flag.set.xxx for properties of the flag set itself. I think this is another property of the flag set, same as the id: feature_flag.set.id.

[toddbaert]

Should we put other attributes here as well, like the flag set version and id?

[aepfli]

This code was only copy pasted fro the json.go file, so I did not question the naming :).

But I agree with combining them within the feature_flag.set attribute. Maybe we should also rethink the naming, because notifications are actually the whole flags, and not just the changed ones. So it might be worth to just say flag_count

[aepfli]

I tried to add version but it is not part of the DataSync payload, parsing the json string would be too much effort, and changing all the syncs to add this information is - IMHO - out of scope. This seems to be a good follow-up, wdyt?

[toddbaert]

I think this attributes are very flagd specific, I'm not sure if we should perhaps use our own custom namespace for them?

[aepfli]

i like the idea, what is our preference here. Should it be feature_flag.flagd.<attribute> or simply flagd.<attribute>

[aepfli]

adapted based on my perception, please check if you agree with this format

[toddbaert]

I love this idea. I want to think very carefully about our naming though. I think propagationTraceId and propagationSpanId might be a bit generic. Maybe something more like configurationChangedTraceId and configurationChangedSpanId since our event name is PROVIDER_CONFIGURATION_CHANGED.

[aepfli]

I am not sure I agree, because the PROVIDER_CONFIGURATION_CHANGED is also a provider implementation detail. we also add this information on the first connection, which is a provider ready event.

[toddbaert]

I love this idea and I want to merge it, but first I want us to carefully consider all our naming here. I've left a couple suggestions. The most important considerations to me are:

• https://github.com/open-feature/flagd/pull/1595/files#r2105837735
• https://github.com/open-feature/flagd/pull/1595/files#r2105839471