signing error when using s3 alternatives

[fboerman]

Apache Iceberg version

0.10.0 (latest release)

Please describe the bug 🐞

when I am using pyiceberg with a rest catalogue and a s3 backend on ovhcloud or hetzner I always get this error when pushing data (example ovhcloud but hetzner is similar):

pyiceberg.exceptions.SignError: Failed to sign request 401: {'method': 'PUT', 'region': 'de', 'uri': 'https://s3.de.io.cloud.ovh.net/dl-amunanalytics/iceberg/019ad68e-74d5-7b63-b17e-72ee4e468d3d/019ad68e-dd6d-7951-b5f5-435e38041856/metadata/snap-3963935388477483242-0-78604abf-4fac-4072-bec7-a33d40a3cabd.avro', 'headers': {'User-Agent': ['aiobotocore/2.17.0 md/Botocore#1.35.93 ua/2.0 os/macos#25.1.0 md/arch#arm64 lang/python#3.12.11 md/pyimpl#CPython cfg/retry-mode#legacy botocore/1.35.93'], 'Content-MD5': ['FRHffmvppTAlGR6yOEaWoQ=='], 'Expect': ['100-continue']}}

Spark works just fine. It seems that pyiceberg is expecting some form of signing which is not available and cannot get around it, while spark can. Is there a setting that fixes this? is this expected behaviour? I am a bit at a loss on this. Thanks in advance for any help!

Willingness to contribute

• I can contribute a fix for this bug independently
• I would be willing to contribute a fix for this bug with guidance from the Iceberg community
• I cannot contribute a fix for this bug at this time

[Fokko]

Thanks @fboerman for raising this issue. My first feeling is that there is a discrepancy between Java and PyIceberg. Probably around what's included in the signature, and what's being excluded. Unfortunally, there is no easy way to check this without seeing how the requests are being signed.

Maybe @danielcweeks or @c-thiel have seen anything similar?

[ChangxingJiang]

This issue look like #2544 .

[Fokko]

Thanks @ChangxingJiang, in that case @fboerman could try to check if the error persists with 0.9.1

[fboerman]

thanks for this! this is definitely the same issue because I also use lakekeeper. downgrading indeed fixed this. I will close for duplicate.

[fboerman]

sorry to bother you again @Fokko @ChangxingJiang but I reinstalled my venv and did pip install pyiceberg==0.9.1 and suddenly its broken again:
botocore.exceptions.ClientError: An error occurred (SignatureDoesNotMatch) when calling the PutObject operation: None

I trippled checked my access key, in cli it works. how can this happen? is there a specific version combination I need?

[fboerman]

I have by now spend hours on this..... I have written a quick test file that uploads a random file with s3fs to the bucket. this works so my access credentials are clearly correct. but if I do it with pyiceberg 0.9.1 I get a signature does not match. I really dont get this. is pyiceberg broken even further back? really at a loss here, any help is very appreciated

[ChangxingJiang]

@fboerman Are you using S3 do not support path-style? like OSS or TOS?

You can look at #2422 .

And then, you may meet #2529

[fboerman]

@ChangxingJiang I am using hetzner which uses Ceph which should support path style. I have it enabled in lakekeeper as well. spark can write to it so not sure what is going wrong here. I tried the trick with setting the lambda to None but that does not do anything for me
also changing path style to false does not do anything

[ChangxingJiang]

@ChangxingJiang I am using hetzner which uses Ceph which should support path style. I have it enabled in lakekeeper as well. spark can write to it so not sure what is going wrong here. I tried the trick with setting the lambda to None but that does not do anything for me also changing path style to false does not do anything

This might not appear to be an issue at first glance. My problem arises due to the path style, and after downgrading to an earlier version, I need to reapply the changes made in #2517 .

[fboerman]

@ChangxingJiang I am using hetzner which uses Ceph which should support path style. I have it enabled in lakekeeper as well. spark can write to it so not sure what is going wrong here. I tried the trick with setting the lambda to None but that does not do anything for me also changing path style to false does not do anything

This might not appear to be an issue at first glance. My problem arises due to the path style, and after downgrading to an earlier version, I need to reapply the changes made in #2517 .

I am sorry I dont fully understand, is this for path style true or false? you say to downgrade and then reapply the patch of #2517 and then put S3_FORCE_VIRTUAL_ADDRESSING to true or false? What is the combination of config then you would recommend?

[fboerman]

okay @ChangxingJiang I figured out what you meant, I now use 0.9.1 with the manuall patch, now I am suddenly back at the signing error again:
pyiceberg.exceptions.SignError: Failed to sign request 403: {'method': 'PUT', 'region': 'nbg1', 'uri': 'https://dl-amunanalytics.nbg1.your-objectstorage.com/iceberg/019ad9c2-4e8c-7b00-a3a4-2e4310e9638d/019ae651-c3c9-7c20-8ed9-346a62a35bcc/metadata/snap-1791227160749482219-0-350a8033-21dc-4bca-bbdb-abb6010f55ac.avro', 'headers': {'User-Agent': ['aiobotocore/2.26.0 md/Botocore#1.41.5 ua/2.1 os/macos#25.1.0 md/arch#arm64 lang/python#3.12.11 md/pyimpl#CPython m/b,D,Z,N cfg/retry-mode#legacy botocore/1.41.5'], 'Expect': ['100-continue'], 'Transfer-Encoding': ['chunked'], 'Content-Encoding': ['aws-chunked'], 'X-Amz-Trailer': ['x-amz-checksum-crc32'], 'X-Amz-Decoded-Content-Length': ['1656'], 'x-amz-sdk-checksum-algorithm': ['CRC32'], 'X-Amz-Content-SHA256': ['STREAMING-UNSIGNED-PAYLOAD-TRAILER']}}

[fboerman]

okay I added 'header.X-Iceberg-Access-Delegation': 'remote-signing' to the config on advice of a community member of lakekeeper and now it seems stuff is working!

[kevinjqliu]

Thank you @fboerman for the detailed report. I think we've fixed the underlying issue (perhaps related to #2827), and should be released in 0.11 (coming soon)
You can also try the nightly build, https://test.pypi.org/project/pyiceberg

Do you think this is an area we can improve the documentation for? Perhaps in https://py.iceberg.apache.org/configuration/#r2-data-catalog