Question: Observability and exposing real-time proxy metrics for transparent data distribution

[diegoripley]

Context

I'm planning on using multistore as the S3 gateway for a data distribution project (Data for Canada). The end-goal is to create a fully transparent system so that other data providers — and data consumers — can make optimal, informed decisions about how to download datasets, whether that means pulling through my multistore deployment or choosing an alternative provider closer to them.

To get there, I'd like to understand how the maintainers see multistore fitting into the observability story. Specifically, I'm interested in exposing real-time operational metrics from the proxy and making them publicly visible.

What I'd like to do

I want to send metrics from multistore to a real-time dashboard/dataset — things like:

• Current active downloads (count)
• Download speed (bytes/sec, aggregate and per-request)
• Upload speed (bytes/sec)
• Breakdown by region (e.x, which Cloudflare colo or AWS region is serving traffic)
• Bytes transferred per bucket over time

I've been looking at Aether as a lightweight, real-time metrics visualization layer. Here's a live demo of what that looks like: https://aether.saphal.me/dashboard/default

The idea would be to have a UsageRecorder implementation (or a separate middleware) that pushes these metrics to an metrics sink, giving anyone a live view of the system's throughput and load characteristics.

Why this matters

Transparency is the key value proposition here. If data providers and consumers can see:

1. How much bandwidth is currently being consumed through a given proxy deployment
2. Which regions are seeing the most traffic
3. Whether the system is under heavy load or has spare capacity

…then they can make data-informed decisions about the best route to fetch a dataset. A provider in Europe might see that my North American deployment is saturated and decide to mirror the dataset themselves. A consumer might see low traffic on a particular region and know it's a good time to start a large download.

Question for maintainers

How do you see observability fitting into Multistore's roadmap? Is there a plan for a metrics export interface (e.x, OpenTelemetry metrics, Prometheus endpoint, or a push-based approach)?

Would love to hear your thoughts on the right direction here as I am not too well versed in observability.

[alukach]

I'm planning on using multistore as the S3 gateway for a data distribution project (Data for Canada).

This is great! Please be aware that you're very early here though, so things are definitely still developing rapidly (we aren't really doing versioning yet, each minor version may be drastically different than the last). With that said, I think that we're circling on some patterns that feel right and hopefully there won't be too much whiplash between versions. Furthermore, being early can have some benefits as it's super useful to take your use cases to inform us how this project should be developed.

I'd like to understand how the maintainers see multistore fitting into the observability story.

Tracking usage is very much a target of this project. We recently added 20395f7 which is an attempt at a generic metering middleware (ie developer can implement what to do with the UsageEvents (ie send to their backend of choice) and potentially deny requests if a quota limit is met (not necessarily relevant to your needs). I'm not 100% certain that it makes sense to provide this level of utility within the project, however it demonstrates how the Middleware trait can be utilized to track usage.

multistore/crates/core/src/middleware.rs

Lines 171 to 217 in f1f2886

	// ---------------------------------------------------------------------------
	// Middleware — the public trait implementors use.
	// ---------------------------------------------------------------------------
	/// Composable post-auth middleware for the dispatch chain.
	///
	/// Implement this trait to intercept requests after identity resolution and
	/// authorization but before (or instead of) backend dispatch. Each
	/// middleware receives the [`DispatchContext`] and a [`Next`] handle to
	/// continue the chain.
	///
	/// ```rust,ignore
	/// struct RateLimiter;
	///
	/// impl Middleware for RateLimiter {
	/// async fn handle<'a>(
	/// &'a self,
	/// ctx: DispatchContext<'a>,
	/// next: Next<'a>,
	/// ) -> Result<HandlerAction, ProxyError> {
	/// if self.is_over_limit(&ctx) {
	/// Ok(HandlerAction::Response(ProxyResult { status: 429, .. }))
	/// } else {
	/// next.run(ctx).await
	/// }
	/// }
	/// }
	/// ```
	pub trait Middleware: MaybeSend + MaybeSync + 'static {
	/// Handle a request, optionally delegating to the next middleware via
	/// [`Next::run`].
	fn handle<'a>(
	&'a self,
	ctx: DispatchContext<'a>,
	next: Next<'a>,
	) -> impl Future<Output = Result<HandlerAction, ProxyError>> + MaybeSend + 'a;
	/// Called after the request has been fully dispatched and the response is
	/// available. Use this for logging, metering, or other post-dispatch
	/// side effects. The default implementation is a no-op.
	fn after_dispatch(
	&self,
	_completed: &CompletedRequest<'_>,
	) -> impl Future<Output = ()> + MaybeSend + '_ {
	async {}
	}
	}