Skip to content

perf: Speed up VersionedIntervalTimeline operations#19278

Draft
jtuglu1 wants to merge 1 commit intoapache:masterfrom
jtuglu1:versionized-timeline-perf-improvements
Draft

perf: Speed up VersionedIntervalTimeline operations#19278
jtuglu1 wants to merge 1 commit intoapache:masterfrom
jtuglu1:versionized-timeline-perf-improvements

Conversation

@jtuglu1
Copy link
Copy Markdown
Contributor

@jtuglu1 jtuglu1 commented Apr 8, 2026

Description

We have observed severe performance degradation in the VersionedIntervalTimeline when ingesting datasource at the MINUTE segment grain. Due to query runners needing to acquire an exclusive lock here on the query hot-path, if frequent segment add/remove callbacks that touch many intervals occur, the volume + latency of each of these timeline operations can cause the BrokerServerView lock to stall indefinitely, preventing queries from running. This giant lock in BrokerServerView is IMO another issue that should be dealt with in a separate change (at the very least, should change this lock granularity to per-datasource).

This patch speeds up a few timeline operations to significantly increase timeline throughput to minimize contention on the BrokerServerView lock.

Results

Params: numInitialRootGenSegmentsPerInterval=10, numNonRootGenerations=1, useSegmentLock=false

benchLookup

Granularity Intervals Baseline (ops/s) Optimized (ops/s) Speedup
DAY 365 401,045 1,704,230 4.2x
HOUR 8,760 19,265 960,835 49.9x
MINUTE 525,600 68 243,950 3,588x

benchRemoveWithReplacement

Granularity Intervals Baseline (ops/s) Optimized (ops/s) Speedup
DAY 365 581,271 662,742 ~1x (no visible change)
HOUR 8,760 19,427 19,136 ~1x (no visible change)
MINUTE 525,600 45 264 5.9x

Release note

Speed up VersionedIntervalTimeline operations


This PR has:

  • been self-reviewed.
  • added documentation for new or modified features or behaviors.
  • a release note entry in the PR description.
  • added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
  • added or updated version, license, or notice information in licenses.yaml
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
  • added integration tests.
  • been tested in a test Druid cluster.

@jtuglu1 jtuglu1 force-pushed the versionized-timeline-perf-improvements branch 2 times, most recently from 585c258 to dc73607 Compare April 8, 2026 20:22
@jtuglu1 jtuglu1 force-pushed the versionized-timeline-perf-improvements branch from dc73607 to cc6d8a1 Compare April 8, 2026 21:01
@jtuglu1 jtuglu1 force-pushed the versionized-timeline-perf-improvements branch from cc6d8a1 to 19e327a Compare April 8, 2026 21:04
@jtuglu1
Copy link
Copy Markdown
Contributor Author

jtuglu1 commented Apr 8, 2026

I think #19138 could solve this problem more cleanly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant