V3 Tracking issue

[Fokko]

Tracks the progress of the V3 implementation

• New DataTypes
  • UnknownType: Added in feat: Add unknown type #1681
  • Timestamp(Tz)NanoType: Introduced in V3: Introduce timestamp_ns and timestamptz_ns #1632
  • VariantType: Add VariantType #1819
  • Geometry/Geography: Add GeometryType / GeographyType #1820
• Default value:
  • Implement default-value projection #1836 / Update schema projection to support initial-defaults #1644
  • Update-schema: Add support for initial-default #1770
  • Prerequisite: Add JSON single value encoding #1805
• Multi-arg transforms
  • Reading of the metadata has been added in Add V3 read support #1554
  • Supporting multi-arg transforms still needs to happen
• Row-level lineage
  • Make PyIceberg aware of row-level lineage #1821
• Deletion vectors
  • Read support is being added in Deletion Vectors read support #1516
  • Write support

[sungwy]

I'd love to get my hands on working with some more new types (when the dependencies are resolved).

Could you assign me to either the VariantType or GeoTypes? (or both? haha) :)

[Fokko]

Hey @sungwy it is first come first serve! :D Please check out the issues of the types itself. I think there are some dependencies, but curious to learn what you think.

[smaheshwar-pltr]

Happy to see an issue tracking this 🚀 !

The V3 spec changes for encryption have merged in apache/iceberg#12162. There's an open PR (edit: now merged) for the resultant changes to table metadata on the Java side here - apache/iceberg#12927.

@Fokko, can we add encryption to the list here, so we're tracking it somewhere for PyIceberg? (Update: I put up #1972 for metadata read change tracking)

[yingjianwu98]

@Fokko

I see that read for equality deletes are not implemented yet and write for both positional/equality deletes are also not implemented.

I am aware that the community deprecates position deletes in v3 and might deprecate equality deletes as well in the future so want to ask what is the vision for pyiceberg on these topics, thanks!

[kevinjqliu]

@stevie9868

I am interested in Deletion vectors work if no one is working on it.

DV read is implemented in #1516. I dont think anyone has started on write support

Also, I see that read for equality deletes are not implemented yet and write for both positional/equality deletes are also not implemented.

Even with the deprecation, I think it would be nice to support reading equality deletes for feature completeness.
Writing positional deletes would be nice too.
Writing equality deletes is nice but we'd like to discourage writing equality deletes due to deprecation

[rambleraptor]

@stevie9868 are you still planning on taking on the deletion vector work? I'm happy to contribute some cycles if you don't have any

[yingjianwu98]

@rambleraptor

yeah, I am planning to work on it soon.
There is an ongoing work to build the DeleteFileIndex, which the DV work might depend on.