Bug: UploadPart fails with Aliyun OSS due to unsupported aws-chunked encoding

[Dludora]

Apache Iceberg version

0.10.0 (latest release)

Please describe the bug 🐞

Description

When using PyIceberg with Aliyun OSS (Object Storage Service) via the S3-compatible endpoint, creating a table or appending data fails during the metadata/data upload phase. The underlying PyArrow S3 filesystem attempts to use aws-chunked encoding with x-amz-content-sha256, which is not supported by Aliyun OSS's S3 compatibility layer for certain operations.

This leads to the following error:

OSError: When uploading part for key '...' in bucket '...': 
AWS Error UNKNOWN (HTTP status 400) during UploadPart operation: 
Unable to parse ExceptionName: InvalidArgument 
Message: aws-chunked encoding is not supported with the specified x-amz-content-sha256 value.

According to Aliyun OSS Documentation, the S3-compatible API has specific constraints regarding signature versions and chunked encoding.

Steps to Reproduce

Initialize a Catalog with Aliyun OSS S3-compatible endpoint.

1. Set s3.force-virtual-addressing: "true" as required by OSS.

2. Attempt to create a table or write data.

3. Reproducible Code Snippet:

import os
from pyiceberg.catalog import load_catalog

catalog = load_catalog(
    "oss_catalog",
    **{
        "type": "sql",
        "uri": "sqlite:///oss_catalog.db",
        "warehouse": "s3://your-bucket/path/",
        "s3.endpoint": "https://s3.oss-cn-wulanchabu.aliyuncs.com",
        "s3.region": "cn-wulanchabu",
        "s3.access-key-id": os.environ["OSS_AK"],
        "s3.secret-access-key": os.environ["OSS_SK"],
        "s3.force-virtual-addressing": "true",
    }
)

# This triggers the UploadPart error during metadata write
catalog.create_table("default.test_table", schema=pa_schema)

Willingness to contribute

• I can contribute a fix for this bug independently
• I would be willing to contribute a fix for this bug with guidance from the Iceberg community
• I cannot contribute a fix for this bug at this time

[kevinjqliu]

Thanks for reporting the issue!

Which FileIO implementation are you using? Fsspec or Pyarrow?
Are you using the S3 Signer? The only place in the codebase i see mentions of x-amz-content-sha256

Heres the translation from the aliyun docs, via claude:

Solutions
If using aws-chunked encoding, you need to set x-amz-content-sha256 to STREAMING-AWS4-HMAC-SHA256-PAYLOAD.
If you don't need to use aws-chunked encoding, remove the Content-Encoding request header or set it to another value.
It is recommended to use OSS's standard Multipart Upload method rather than relying on the S3 SDK's aws-chunked encoding feature.
Ensure that the S3 SDK version you're using supports OSS's aws-chunked encoding implementation, and update to the latest version if necessary.

Are you able to try one of the above? Would be helpful to see which one works. I do not have an Aliyun env 😄

[Dludora]

Hello, I use PyArrowIO by default.

Traceback (most recent call last):
  File "/mnt/data/ksr/iceberg/./creare_catalog.py", line 40, in <module>
    table = catalog.create_table(identifier, schema=data.schema)
  File "/mnt/data/iceberg/lib/python3.10/site-packages/pyiceberg/catalog/sql.py", line 221, in create_table
    self._write_metadata(metadata, io, metadata_location)
  File "/mnt/data/iceberg/lib/python3.10/site-packages/pyiceberg/catalog/__init__.py", line 1005, in _write_metadata
    ToOutputFile.table_metadata(metadata, io.new_output(metadata_path))
  File "/mnt/data/iceberg/lib/python3.10/site-packages/pyiceberg/serializers.py", line 130, in table_metadata
    with output_file.create(overwrite=overwrite) as output_stream:
  File "pyarrow/io.pxi", line 136, in pyarrow.lib.NativeFile.__exit__
  File "pyarrow/io.pxi", line 206, in pyarrow.lib.NativeFile.close
  File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status

This is a more concise traceback.

[Dludora]

This bug is handled when I set environment variable

os.environ["AWS_REQUEST_CHECKSUM_CALCULATION"] = "when_required"
os.environ["AWS_RESPONSE_CHECKSUM_VALIDATION"] = "when_required"

[martyngigg]

I'm also using PyArrowIO and on upgrading to 22.0.0 (it's fine with <= 21.0.0) I have received:

AWS Error UNKNOWN (HTTP status 400) during UploadPart operation: Unable to parse ExceptionName: 
XAmzContentSHA256Mismatch Message:  (Request ID: tx000004f6eb4a91c740bec-00695e3fe7-30b07c552-default).
The package is aborted and cannot be retried.

Using the same environment variables as @Dludora fixes the issue. See ISISNeutronMuon/analytics-data-platform#183 for more details. We have a Ceph-based object store.

Just a note that something similar happened with botocore that required those environment variables. Perhaps this is a pyarrow issue?

[kevinjqliu]

I believe #2827 resolves this.
@Dludora could you confirm? The change should already be in the nightly bulid https://test.pypi.org/project/pyiceberg

Closing for now, will reopen if necessary