feat: Add array data type support by charlesdong1991 · Pull Request #433 · apache/fluss-rust

charlesdong1991 · 2026-03-07T16:02:37Z

Purpose

Linked issue: close #386

Brief change log

Add end-to-end ARRAY column support in fluss-rust

Tests

Yes all tests are passed locally

charlesdong1991 · 2026-03-07T16:06:56Z

crates/fluss/src/row/compacted/compacted_row_reader.rs

                        let (val, next) = reader.read_long(cursor);
-                        let decimal = Decimal::from_unscaled_long(val, precision, scale)
-                            .expect("Failed to create decimal from unscaled long");
+                        let decimal =


i think we should convert all expect to typed errors so malformed array payload handling does not leave panic backdoor

charlesdong1991 · 2026-03-07T16:10:02Z

crates/fluss/src/row/compacted/compacted_key_writer.rs

+        // silently widen key semantics.
+        if matches!(
+            field_type,
+            DataType::Array(_) | DataType::Map(_) | DataType::Row(_)


this applies to all array/map/row, so i understand this PR only adds Array, but i think better to add all of them.
if there is objection, i can remove

luoyuxia · 2026-03-08T02:31:21Z

@charlesdong1991 Thanks for the great pr. I'll find some time to review

Copilot

Pull request overview

Adds end-to-end ARRAY column support to the Fluss Rust client, including a Java-compatible binary array representation and integration across row encoders/decoders, Arrow interop, and public APIs.

Changes:

Introduces FlussArray / FlussArrayWriter (BinaryArray layout) and wires it into Datum, InternalRow, and runtime field/value writers.
Adds compacted row read/write support for arrays (length-prefixed BinaryArray bytes) and rejects ARRAY as a key column type.
Extends Arrow conversion logic and updates user/docs + tests for primitive, nullable, empty, and nested arrays.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
website/docs/user-guide/rust/data-types.md	Documents `ARRAY<T>` type mapping and how to construct/read arrays.
website/docs/user-guide/rust/api-reference.md	Adds `InternalRow::get_array` and documents `FlussArray` API.
crates/fluss/src/row/mod.rs	Exposes `binary_array` module and re-exports `FlussArray`; adds `InternalRow::get_array`.
crates/fluss/src/row/field_getter.rs	Adds `Array` field getter + tests.
crates/fluss/src/row/encode/compacted_key_encoder.rs	Adds test ensuring array types are rejected in key encoding.
crates/fluss/src/row/datum.rs	Adds `Datum::Array` and Arrow ListBuilder append/conversion logic.
crates/fluss/src/row/compacted/compacted_row_writer.rs	Adds `write_array` (delegates to length-prefixed bytes).
crates/fluss/src/row/compacted/compacted_row_reader.rs	Makes deserialization fallible and adds array decoding via `FlussArray::from_bytes`.
crates/fluss/src/row/compacted/compacted_row.rs	Propagates fallible deserialization and adds `InternalRow::get_array` + array tests.
crates/fluss/src/row/compacted/compacted_key_writer.rs	Explicitly rejects complex key types and adds `write_array` to the delegated writer surface.
crates/fluss/src/row/column.rs	Implements `ColumnarRow::get_array` by converting Arrow `ListArray` → `FlussArray`; adds tests.
crates/fluss/src/row/binary_array.rs	New Java-compatible binary array implementation + writer + tests.
crates/fluss/src/row/binary/binary_writer.rs	Adds `BinaryWriter::write_array` and `InnerValueWriter::Array`.
crates/fluss/src/record/arrow.rs	Adds Arrow `List` builder support and `from_arrow_type` mapping for list element conversion.
bindings/cpp/src/types.rs	Propagates `Datum::Array` handling in type resolution and row ownership conversion.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

crates/fluss/src/row/binary_array.rs

crates/fluss/src/row/datum.rs

crates/fluss/src/row/compacted/compacted_row_reader.rs

crates/fluss/src/row/compacted/compacted_key_writer.rs

crates/fluss/src/row/column.rs

crates/fluss/src/row/binary_array.rs

fresh-borzoni

@charlesdong1991 LGTM overall, left minor comments
If you fixed smth in the meantime - good, sorry for the noise, as I started reviewing before the new code :)

crates/fluss/src/row/datum.rs

crates/fluss/src/row/column.rs

crates/fluss/src/record/arrow.rs

crates/fluss/src/row/binary_array.rs

bindings/cpp/src/types.rs

crates/fluss/src/row/datum.rs

charlesdong1991 · 2026-03-08T14:48:21Z

Hi @fresh-borzoni thanks a lot!
It would be great if you can give another round of reviews since i made some changes during your review ^^

leekeiabstraction

Thank you for the PR! Left some comments. PTAL

crates/fluss/src/record/arrow.rs

crates/fluss/src/row/encode/compacted_key_encoder.rs

fresh-borzoni

@charlesdong1991 TY, looked through again, left comments
PTAL

crates/fluss/src/row/binary_array.rs

crates/fluss/src/row/encode/compacted_key_encoder.rs

crates/fluss/src/row/column.rs

crates/fluss/src/row/binary_array.rs

crates/fluss/src/record/arrow.rs

crates/fluss/src/row/mod.rs

crates/fluss/src/record/arrow.rs

Copilot

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated 4 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

website/docs/user-guide/rust/data-types.md

crates/fluss/src/row/compacted/compacted_key_writer.rs

crates/fluss/src/row/encode/compacted_key_encoder.rs

fresh-borzoni

@charlesdong1991 TY, Looked through, LGTM overall, one nit comment 👍

crates/fluss/src/row/compacted/compacted_row.rs

crates/fluss/src/row/encode/compacted_key_encoder.rs

charlesdong1991 · 2026-03-11T19:57:33Z

Thanks both! PTAL @fresh-borzoni @leekeiabstraction @luoyuxia

leekeiabstraction · 2026-03-11T20:35:58Z

Test case changes looks good to me ~~but diff seems larger, seems to be squashed with is retriable changes from another PR? Not sure if it will conflict~~. LGTM otherwise.

Edit: for some reason I mistaken the diff between commits as overall diff.

fresh-borzoni

@charlesdong1991 TY, LGTM

leekeiabstraction

Approved. TY for the PR!

luoyuxia · 2026-03-15T10:24:52Z

@charlesdong1991 Hi, is it possible to add IT for covering the support for array? Or we can merge this first, but not expose it to user in document util we add it to make sure it work.
I don't mind add it in another pr (also fix any possible issue found by IT )since the pr is already to big to review.

luoyuxia · 2026-03-15T10:33:00Z

website/docs/user-guide/rust/api-reference.md

+
+Element getters mirror `InternalRow` typed getters and return `Result<T>`. For example, use `get_int()`, `get_long()`, and `get_double()` for primitive elements, and `get_string()`, `get_binary()`, `get_decimal()`, `get_timestamp_ntz()`, `get_timestamp_ltz()`, and `get_array()` for variable-length or nested elements.
+
+TODO: `FlussArray` currently exposes the fallible getter surface as the stable API. Infallible fast-path variants may be added later as non-breaking extensions if needed.


why left todo in user-face document?

it was added to address a comment in Copilot: 66359e4 for the design decision we have now that fluss array only has fallible getters now, and we might add infallible variants in future as non-breaking change.

But i think indeed, it's confusing to leave TODO in user facing doc, and let me just remove this part completely, and move the note to codebase instead @luoyuxia

charlesdong1991 · 2026-03-15T12:07:17Z

Hi @luoyuxia

if you don't mind, i'd prefer to have integration tests part in a separate PR, and i created this issue to track: #441 and i can work on it.

Thanks!

charlesdong1991 added 2 commits March 7, 2026 16:59

add array data type support

5a8ca2e

add docs

e35adce

charlesdong1991 commented Mar 7, 2026

View reviewed changes

luoyuxia requested a review from Copilot March 8, 2026 02:29

Copilot started reviewing on behalf of luoyuxia March 8, 2026 02:29 View session

Copilot AI reviewed Mar 8, 2026

View reviewed changes

charlesdong1991 added 3 commits March 8, 2026 11:45

resolve conflicts

a8fdac0

address comments

8cacf63

avoid intermediate record batch in get-array

8208023

fresh-borzoni reviewed Mar 8, 2026

View reviewed changes

Address comments

c29c5ad

leekeiabstraction reviewed Mar 8, 2026

View reviewed changes

crates/fluss/src/record/arrow.rs Outdated Show resolved Hide resolved

crates/fluss/src/record/arrow.rs Outdated Show resolved Hide resolved

crates/fluss/src/row/encode/compacted_key_encoder.rs Show resolved Hide resolved

fresh-borzoni reviewed Mar 9, 2026

View reviewed changes

charlesdong1991 added 2 commits March 9, 2026 20:39

Merge remote-tracking branch 'upstream/main' into issue-386

44336c2

address review comments

750799c

charlesdong1991 requested a review from Copilot March 9, 2026 20:17

Copilot started reviewing on behalf of charlesdong1991 March 9, 2026 20:17 View session

Copilot AI reviewed Mar 9, 2026

View reviewed changes

charlesdong1991 added 3 commits March 9, 2026 21:41

address comments and improve doc

83c8093

address comments

d0b7cde

add todo

66359e4

fresh-borzoni reviewed Mar 11, 2026

View reviewed changes

crates/fluss/src/row/compacted/compacted_row.rs Show resolved Hide resolved

leekeiabstraction reviewed Mar 11, 2026

View reviewed changes

crates/fluss/src/row/encode/compacted_key_encoder.rs Show resolved Hide resolved

address comments

ca96551

fresh-borzoni approved these changes Mar 14, 2026

View reviewed changes

leekeiabstraction approved these changes Mar 14, 2026

View reviewed changes

luoyuxia reviewed Mar 15, 2026

View reviewed changes

charlesdong1991 added 2 commits March 15, 2026 13:03

Merge remote-tracking branch 'upstream/main' into issue-386

3ca1ee1

remove todo from reference.md to code

e93fa3c


		Element getters mirror `InternalRow` typed getters and return `Result<T>`. For example, use `get_int()`, `get_long()`, and `get_double()` for primitive elements, and `get_string()`, `get_binary()`, `get_decimal()`, `get_timestamp_ntz()`, `get_timestamp_ltz()`, and `get_array()` for variable-length or nested elements.

		TODO: `FlussArray` currently exposes the fallible getter surface as the stable API. Infallible fast-path variants may be added later as non-breaking extensions if needed.

Conversation

charlesdong1991 commented Mar 7, 2026

Purpose

Brief change log

Tests

Uh oh!

charlesdong1991 Mar 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

charlesdong1991 Mar 7, 2026

Choose a reason for hiding this comment

Uh oh!

luoyuxia commented Mar 8, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fresh-borzoni left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

charlesdong1991 commented Mar 8, 2026

Uh oh!

leekeiabstraction left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fresh-borzoni left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fresh-borzoni left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

charlesdong1991 commented Mar 11, 2026

Uh oh!

leekeiabstraction commented Mar 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fresh-borzoni left a comment

Choose a reason for hiding this comment

Uh oh!

leekeiabstraction left a comment

charlesdong1991 Mar 7, 2026 •

edited

Loading

fresh-borzoni left a comment •

edited

Loading

leekeiabstraction commented Mar 11, 2026 •

edited

Loading

luoyuxia commented Mar 15, 2026 •

edited

Loading