Skip to content

perf: use in-place Arrays.sort for std.sort/std.set/std.uniq#664

Closed
He-Pin wants to merge 1 commit intodatabricks:masterfrom
He-Pin:perf/inplace-sort
Closed

perf: use in-place Arrays.sort for std.sort/std.set/std.uniq#664
He-Pin wants to merge 1 commit intodatabricks:masterfrom
He-Pin:perf/inplace-sort

Conversation

@He-Pin
Copy link
Copy Markdown
Contributor

@He-Pin He-Pin commented Apr 4, 2026

Motivation

std.sort uses Scala's sortWith which creates a new sorted copy. For homogeneous arrays (all numbers or all strings), we can use Java's optimized Arrays.sort in-place on a copied array, avoiding the overhead of Scala's generic sorting.

Key Design Decision

Detect homogeneous arrays at sort time and dispatch to Arrays.sort(Double[]) or Arrays.sort(String[]) for maximum performance. Fall back to the generic sort for mixed-type arrays.

Modification

Added homogeneous array detection in SetModule.scala sortArr method. Uses Arrays.sort with primitive specialization for numeric and string arrays.

Benchmark Results

JMH Regression Suite (1 fork, 3 warmup, 1 measurement)

Benchmark Master This PR Change
bench.06 0.367 0.312 -15.0%
setDiff 0.425 similar noise
setInter 0.372 similar noise
setUnion 0.676 similar noise

All other benchmarks within noise margin.

Scala Native Hyperfine (-N -w4 -m20)

Benchmark sjsonnet master This PR jrsonnet vs master vs jrsonnet
bench.06 (array sort) 8.2 ± 1.1 ms 7.4 ± 1.0 ms 7.7 ± 9.1 ms 1.10x faster 1.05x faster

Analysis

Arrays.sort uses dual-pivot Quicksort which is highly optimized for primitive arrays. The improvement is most visible on sort-heavy benchmarks. On native, sjsonnet with this PR is now slightly faster than jrsonnet on bench.06.

References

Upstream jit branch exploration at he-pin/sjsonnet@jit

Result

15% improvement on JMH bench.06, 10% on native. sjsonnet is now 1.05x faster than jrsonnet for array sorting, closing a previous 1.36x gap.

@He-Pin He-Pin force-pushed the perf/inplace-sort branch 2 times, most recently from e4fc0b5 to 21ab9c0 Compare April 4, 2026 12:04
@He-Pin He-Pin marked this pull request as ready for review April 4, 2026 12:42
Replace map+sortBy chain with java.util.Arrays.sort for numeric, string,
and array sorting in the identity-key path. This eliminates intermediate
array allocations from the Schwartzian transform in Scala's sortBy and
sorts the strict array in-place.

Upstream: jit branch commit 926a6d0
@He-Pin He-Pin force-pushed the perf/inplace-sort branch from 21ab9c0 to 23ef1b1 Compare April 4, 2026 17:51
@He-Pin
Copy link
Copy Markdown
Contributor Author

He-Pin commented Apr 6, 2026

Closing: Independent verification shows a 12% regression on setUnion (the primary sort benchmark). java.util.Arrays.sort with Comparator lambdas has more overhead per comparison than Scala's sortBy, which extracts keys once via map. The allocation savings don't offset the per-comparison overhead.

Verified with Hyperfine (warmup 5, runs 20, --shell=none):

  • setUnion: master 8.0±0.6ms → in-place 8.9±0.8ms = 1.12x slower
  • comparison2: neutral (175.2 vs 176.1ms)
  • bench.02: neutral (75.2 vs 74.8ms)

@He-Pin He-Pin closed this Apr 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant