PLT-923: Schema ↔ Arrow nullability correspondence + logical equality tests#126

Merged

eywalker merged 28 commits intodevfrom

eywalker/plt-923-add-tests-verifying-schema-equality-corresponds-to-arrow

Apr 3, 2026

Commits on Apr 2, 2026

docs(specs): add PLT-923 schema-arrow equality design spec
kurodo3[bot]
committed
docs(specs): fix review issues in PLT-923 schema-arrow equality spec
kurodo3[bot]
committed
docs(specs): expand PLT-923 spec to include full nullability treatment
kurodo3[bot]
committed
feat(types): add Schema.as_required() to expose structural Arrow-level view

kurodo3[bot]
and
claude
committed
fix(converter): derive Arrow field nullable from Python type in python_schema_to_arrow_schema

kurodo3[bot]
and
claude
committed
fix(converter): reconstruct T | None from nullable Arrow fields in arrow_schema_to_python_schema

kurodo3[bot]
and
claude
committed
refactor(converter): guard Any in arrow_schema_to_python_schema and extract normalization helper

kurodo3[bot]
and
claude
committed
test(schema): add comprehensive Schema ↔ Arrow logical equality test suite (PLT-923)

kurodo3[bot]
and
claude
committed
style(tests): remove unused pytest import from test_schema_arrow_equality
kurodo3[bot]
committed
fix(utils): preserve field metadata in make_schema_non_nullable
kurodo3[bot]
committed
docs(plans): add PLT-923 schema-arrow equality implementation plan
kurodo3[bot]
committed
feat(nullable): make nullable-stripping configurable in ArrowTableStream, Datagram, SourceStreamBuilder

kurodo3[bot]
and
claude
committed
refactor(nullable): centralise nullable-stripping at SourceStreamBuilder; operators preserve Arrow nullable flags

kurodo3[bot]
and
claude
committed
refactor(batch): derive batched schema from input table instead of post-hoc rebuild

kurodo3[bot]
and
claude
committed
fix(nullable): eliminate remaining nullable=True leaks at construction sites

kurodo3[bot]
and
claude
committed
refactor(arrow_utils): derive prefix column nullable from intent, not null counts

kurodo3[bot]
and
claude
committed
feat(precomputed_converters): add arrow_schema override to python_dicts_to_arrow_table

kurodo3[bot]
and
claude
committed
refactor(sources): move nullable inference out of builder, into each source

kurodo3[bot]
and
claude
committed
feat(arrow_table_source): add infer_nullable parameter

kurodo3[bot]
and
claude
committed
fix(test-objective): replace raw ArrowTableStream fixtures with ArrowTableSource

kurodo3[bot]
and
claude
committed
fix(review): address Copilot review comments

kurodo3[bot]
and
claude
committed
fix(review): preserve schema-level metadata and document prepare_prefixed_columns

kurodo3[bot]
and
claude
committed
test(arrow_utils): add metadata preservation regression tests

kurodo3[bot]
and
claude
committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PLT-923: Schema ↔ Arrow nullability correspondence + logical equality tests#126

PLT-923: Schema ↔ Arrow nullability correspondence + logical equality tests#126
eywalker merged 28 commits intodevfrom
eywalker/plt-923-add-tests-verifying-schema-equality-corresponds-to-arrow

Commits on Apr 2, 2026

docs(specs): add PLT-923 schema-arrow equality design spec

docs(specs): fix review issues in PLT-923 schema-arrow equality spec

docs(specs): expand PLT-923 spec to include full nullability treatment

feat(types): add Schema.as_required() to expose structural Arrow-level view

fix(converter): derive Arrow field nullable from Python type in python_schema_to_arrow_schema

fix(converter): reconstruct T | None from nullable Arrow fields in arrow_schema_to_python_schema

refactor(converter): guard Any in arrow_schema_to_python_schema and extract normalization helper

test(schema): add comprehensive Schema ↔ Arrow logical equality test suite (PLT-923)

style(tests): remove unused pytest import from test_schema_arrow_equality

fix(utils): preserve field metadata in make_schema_non_nullable

docs(plans): add PLT-923 schema-arrow equality implementation plan

feat(nullable): make nullable-stripping configurable in ArrowTableStream, Datagram, SourceStreamBuilder

refactor(nullable): centralise nullable-stripping at SourceStreamBuilder; operators preserve Arrow nullable flags

refactor(batch): derive batched schema from input table instead of post-hoc rebuild

fix(nullable): eliminate remaining nullable=True leaks at construction sites

refactor(arrow_utils): derive prefix column nullable from intent, not null counts

feat(precomputed_converters): add arrow_schema override to python_dicts_to_arrow_table

refactor(sources): move nullable inference out of builder, into each source

feat(arrow_table_source): add infer_nullable parameter

fix(test-objective): replace raw ArrowTableStream fixtures with ArrowTableSource

fix(review): address Copilot review comments

fix(review): preserve schema-level metadata and document prepare_prefixed_columns

test(arrow_utils): add metadata preservation regression tests

Commits on Apr 3, 2026

refactor(review): address round-3 review comments

fix: update test-objective test to import from arrow_utils instead of deleted arrow_data_utils

fix(test): add NOT NULL constraints to pipeline integration test table DDL

style(review): remove quoted type hints and consolidate dict_source schema param

style(review): inline raw_schema intermediate variable in datagram.py