Validation error related to content-md5 when uploading OME-Zarr

[kabilar]

Reposting an error from dandi/dandi-hub#289 (comment):

When running dandi upload on the DANDI Hub for an OME-Zarr asset the error below occurs.

@CodyCBakerPhD @jjnesbitt @yarikoptic Any suggestions? Thank you.

Tail of log:

2026-03-18T08:49:36-0500 [ERROR   ] dandi 26563:140641760982784 Failed to upload /home/dstansby/data/linc/001411/sourcedata/raw/sub-I74/micr/sub-I74_sample-hemi_acq-zoom_chunk-02_XPCT.ome.zarr/0/c/0/0/186: Error 400 while sending PUT request to https://dandiarchive.s3.amazonaws.com/zarr/427e2ade-7a38-4f31-934b-6a74188120d6/0/c/0/0/186?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAUBRWC5GAEKH3223E%2F20260318%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20260318T131131Z&X-Amz-Expires=21600&X-Amz-SignedHeaders=content-md5%3Bhost%3Bx-amz-tagging&X-Amz-Signature=c8932c80b50642976753587f682fd6c9b2fb14a487506964d9099a7692e03bc3: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>BadDigest</Code><Message>The Content-MD5 you specified did not match what we received.</Message><CalculatedDigest>DvNGyXv56F0PdLculLqPdg==</CalculatedDigest><ExpectedDigest>5a1b1a6ce22af597da3e7549b9100235</ExpectedDigest><RequestId>SF2PBVY91D8WYS3W</RequestId><HostId>68PXcoPIpKamMD+KnrekF+CvXLZgCPccko/BW3bgsgJVjMWG/OHnc9pHx6HHr35or1aghLPWU16JcROUcGSCnM2F4vQhdSBU</HostId></Error>
2026-03-18T08:49:36-0500 [ERROR   ] dandi 26563:140641760982784 Error uploading /home/dstansby/data/linc/001411/sourcedata/raw/sub-I74/micr/sub-I74_sample-hemi_acq-zoom_chunk-02_XPCT.ome.zarr:
Traceback (most recent call last):
  File "/home/dstansby/envs/dandi/.venv/lib/python3.11/site-packages/dandi/upload.py", line 386, in process_path
    for r in dfile.iter_upload(
  File "/home/dstansby/envs/dandi/.venv/lib/python3.11/site-packages/dandi/files/zarr.py", line 813, in iter_upload
    _handle_failed_items_and_raise(
  File "/home/dstansby/envs/dandi/.venv/lib/python3.11/site-packages/dandi/files/zarr.py", line 903, in _handle_failed_items_and_raise
    raise failed_items[0][1]
  File "/home/dstansby/envs/dandi/.venv/lib/python3.11/site-packages/dandi/files/zarr.py", line 942, in _upload_zarr_file
    storage_session.put(
  File "/home/dstansby/envs/dandi/.venv/lib/python3.11/site-packages/dandi/dandiapi.py", line 325, in put
    return self.request("PUT", path, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/dstansby/envs/dandi/.venv/lib/python3.11/site-packages/dandi/dandiapi.py", line 289, in request
    raise requests.HTTPError(msg, response=result)
requests.exceptions.HTTPError: Error 400 while sending PUT request to https://dandiarchive.s3.amazonaws.com/zarr/427e2ade-7a38-4f31-934b-6a74188120d6/0/c/0/0/169?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAUBRWC5GAEKH3223E%2F20260318%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20260318T131131Z&X-Amz-Expires=21600&X-Amz-SignedHeaders=content-md5%3Bhost%3Bx-amz-tagging&X-Amz-Signature=6e3c6f4edb2aa4fa0d2f6a9c8b0426235a836aa8e2dc330a910a50506f325830: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>BadDigest</Code><Message>The Content-MD5 you specified did not match what we received.</Message><CalculatedDigest>ffRj2mH4Wvm8V8SHTc+k0w==</CalculatedDigest><ExpectedDigest>d29ef077bb80cb57f588c15074f6582a</ExpectedDigest><RequestId>5RWSTAAZ5WAH5E05</RequestId><HostId>3FuSqNnmWJEa7oUosQztGcVUoVh4309SHbdy2Wt9z+zY73sRiOG+2v5OtnybkCbMxdHzhl+nb+fIliN1kuzBxHNLe1LiSqOU</HostId></Error>
2026-03-18T08:49:36-0500 [WARNING ] dandi 26563:140642653751104 One or more assets failed validation.  Consult the logfile for details.
2026-03-18T08:49:36-0500 [DEBUG   ] urllib3.connectionpool 26563:140642653751104 Starting new HTTPS connection (1): rig.mit.edu:443
2026-03-18T08:49:37-0500 [DEBUG   ] dandi 26563:140642653751104 Caught exception failed validation
Traceback (most recent call last):
  File "/home/dstansby/envs/dandi/.venv/lib/python3.11/site-packages/dandi/cli/base.py", line 126, in wrapper
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/home/dstansby/envs/dandi/.venv/lib/python3.11/site-packages/dandi/cli/cmd_upload.py", line 107, in upload
    upload(
  File "/home/dstansby/envs/dandi/.venv/lib/python3.11/site-packages/dandi/upload.py", line 488, in upload
    raise upload_err
  File "/home/dstansby/envs/dandi/.venv/lib/python3.11/site-packages/dandi/upload.py", line 309, in process_path
    raise UploadError("failed validation")
dandi.exceptions.UploadError: failed validation
2026-03-18T08:49:37-0500 [INFO    ] dandi 26563:140642653751104 Logs saved in /home/dstansby/.local/state/dandi-cli/log/2026.03.18-12.34.26Z-26563.log

Full log: 2026.03.18-12.34.26Z-26563.log

cc @satra @ayendiki @dstansby

[yarikoptic]

Quick one from the phone for now: Need to search issues anals, I think we chased similar issues before on files flipping their checksums or alike. Is nfs involved?

Give log to Claude code and ask to try to reestablish the chronology of the failure and look through issues on GitHub for related?

[yarikoptic]

prompt to claude code

User reported difference in digest in a file before and after upload as reported by AWS based on minted URL in #1821. I believe we had similar issues before -- please search for
similar ones, and investigate the log 2026.03.18-12.34.26Z-26563.log I copied here to get a better picture on what might be going on. We are in the dandi-cli repo so you can investigate also code paths as to establish
a better picture of the events

lead to this summary congruent with my prior idea (although I did prime claude with it in above) of files changing during upload

Analysis of Issue #1821: Content-MD5 Digest Mismatch During Zarr Upload

Summary

The upload of dandiset 001411 by dstansby fails because every level-0 (full resolution) zarr chunk has a different MD5 digest at upload time vs digest computation time, while all level-2 (downsampled) chunks upload successfully.

Key Facts from the Log

Metric	Value
dandi version	0.74.3, Python 3.11.10
Dataset	DANDI:001411, OME-Zarr (sub-I74_sample-hemi_acq-zoom_chunk-02_XPCT.ome.zarr)
Data path	/home/dstansby/data/linc/001411/sourcedata/raw/sub-I74/micr/
Level 0 chunks	188 PUT attempts → ALL 400 BadDigest
Level 2 chunks	67 PUT attempts → ALL 200 OK
Level 1 chunks	380 digested → 0 uploads (aborted because level 0 failed)
File size changes	None detected (no post_upload_size_check warnings)
Batch size	255 = 188 (level 0) + 67 (level 2) — single batch

Timeline

07:34:26  Upload starts
07:37:22  Digest computation begins (level 0 chunks: /0/c/0/0/*)
07:37:30  Digest computation begins (level 2 chunks: /2/c/0/0/*)
08:10:46  Last level 0 digests computed
08:11:17  Last level 2 digests computed
08:12:47  Level 0 uploads begin → ALL fail with BadDigest
08:48:01  Level 2 uploads begin → ALL succeed
08:49:36  Upload aborts with error

Both level 0 and level 2 have similar time gaps between digest and upload (~2-37 minutes). Yet level 0 fails 100% and level 2 succeeds 100%.

Root Cause: File Content Changed On Disk

The architecture reads each file twice: once for MD5 digest computation (Digester.__call__ in dandi/support/digests.py:69-78), then again during the actual PUT upload (_upload_zarr_file in dandi/files/zarr.py:941-948). If file content changes between these reads, S3 rejects with BadDigest.

The error pairs are real content mismatches (not encoding bugs):

• ExpectedDigest = hex representation of the client's pre-computed MD5 (AWS decodes the base64 Content-MD5 header back to hex in error messages)
• CalculatedDigest = base64 MD5 of what S3 actually received
• All 11+ pairs checked have completely unrelated hashes — not partial corruption but wholesale different file content

The fact that all level-0 chunks fail while all level-2 chunks succeed strongly indicates the full-resolution data was being rewritten on disk during the upload, while the downsampled pyramids were not. This is consistent with an active OME-Zarr conversion or reprocessing touching only level 0.

Historical Context

This project has extensive prior work on NFS-related flakiness:

• pre_upload_size_check / post_upload_size_check in dandi/utils.py:883-905 with explicit "flaky NFS" comments
• PR Set Content-MD5 header when uploading asset parts #409 added Content-MD5 header, immediately reverted in PR Revert PR #409 (Content-MD5 header) #419 (2021) — but it was later re-added for zarr uploads
• Multiple NFS-related CI commits (596b97f, a07aa37, 4f217c9, 302079b)
• Issue Allow Retrying when Uploading Big Assets #1662 handling S3 timeout 400 errors in zarr uploads

What Needs Clarification From the Reporter

1. Was the data being actively processed/converted during the upload? The pattern of only level 0 changing is consistent with OME-Zarr pyramid regeneration.
2. What filesystem is /home/dstansby/data/linc/? NFS, GPFS, Lustre, or local? Network filesystems are known to cause stale reads.
3. Can they reproduce by re-running the upload when data is guaranteed stable?

Potential Improvements to dandi-cli

The fundamental vulnerability is the two-read architecture (digest then upload as separate file reads). Options:

1. Compute digest while streaming upload: Read the file once, tee the stream to both MD5 computation and the HTTP body. This eliminates the race window entirely. However, S3 presigned PUT URLs require the Content-MD5 header upfront, so this would require a different upload approach (e.g., computing MD5 first, then reading again — same issue, or using S3 multipart upload where each part can be verified).

2. Detect and retry on BadDigest: Currently _retry_zarr_file only retries on 501 and timeout-400. Adding BadDigest 400 as a retryable condition (with re-digest) would help with transient issues, though it wouldn't help if files are systematically changing.

3. Verify digest immediately before upload: Re-hash the file right before the PUT, compare with pre-computed digest, and skip/warn if they differ. This narrows the race window.

4. Read file into memory, digest that, upload that: For zarr chunks (typically small-to-medium sized), read the entire chunk into a bytes buffer, compute MD5 from the buffer, then upload the buffer. This guarantees consistency since there's only one read. This is the most robust solution for zarr chunks which are typically well under 100MB each.

Option 4 is probably the most practical fix for zarr chunk uploads.

relevant questions to be asked from above

1. Was the data being actively processed/converted during the upload? The pattern of only level 0 changing is consistent with OME-Zarr pyramid regeneration.
2. What filesystem is /home/dstansby/data/linc/? NFS, GPFS, Lustre, or local? Network filesystems are known to cause stale reads.
3. Can they reproduce by re-running the upload when data is guaranteed stable?

[kabilar]

Thanks @yarikoptic. @dstansby please correct me if I am wrong. This was being uploaded from DANDI Hub. I don't think anything was being processed during upload. Files were downloaded from GCP, and then uploaded to DANDI.

[yarikoptic]

Files were downloaded from GCP, and then uploaded to DANDI.

in how rapid succession between those two steps?
what command was used to 'download'?
was by any chance sync ran to flush "filesystem" (unlikely but why not to ask)?

NB a little odd that filesize check though passed... I wonder how files could have been changed... may be they were initiated 'sparse' or alike for download...

[dstansby]

I'm running into this again. I'm on a Windows machine, and definitely not (deliberately) changing the file content on my local hard drive. I managed a few uploads fine before this one, but the difference with the failed upload is that I tried it once, interrupted it, then tried it again.

My suspicion is that there's a half uploaded OME-Zarr file on the remote dandi server, whose MD5 doesn't match my local file (because the remote file is truncated, my local one is complete). In this instance I also deleted the partial upload through the web interface beforehand.

For completeness, to answer questions above

Was the data being actively processed/converted during the upload?

No

What filesystem is /home/dstansby/data/linc/? NFS, GPFS, Lustre, or local?

In the latest case a standard Windows filesystem on a hard drive. No network drives involved.

Can they reproduce by re-running the upload when data is guaranteed stable?

Yes, I believe I've just reproduced this.

Files were downloaded from GCP, and then uploaded to DANDI. in how rapid succession between those two steps?

More than a few hours

what command was used to 'download'?

gcloud storage cp -r <source> <dest>

was by any chance sync ran to flush "filesystem" (unlikely but why not to ask)?

No

[dstansby]

Here is the full most recent log: dandi.log

[kabilar]

My suspicion is that there's a half uploaded OME-Zarr file on the remote dandi server, whose MD5 doesn't match my local file (because the remote file is truncated, my local one is complete). In this instance I also deleted the partial upload through the web interface beforehand.

1. Thanks David. Since you deleted the partial upload I would assume that the restarted upload should be fine.
   @yarikoptic @jjnesbitt Perhaps something is not getting cleaned up upon delete of the partial upload?
2. It would be ideal for a partially uploaded Zarr (where the upload was interrupted) to pick up where it left off upon rerunning dandi upload.

[jjnesbitt]

The error being mentioned in the issue description has nothing to do with any existing files, or anything that the archive manages. That error strictly pertains to the S3 upload process for a single file, and is simply saying that the provided value of the Content-MD5 header doesn't match the actual MD5 checksum that S3 is computing from the uploaded data.

There are a variety of reasons this could be occurring, but it doesn't seem to have anything to do with the archive itself.

[yarikoptic]

#1824 addressing an aspect of restarting the failed upload might be a solution, I hope @dstansby would have a chance to try it out some time soon.

[dstansby]

I tried with #1824, and that seems to remove the mismatched digest errors, full log here: 2026-03-27T12:50:29+0000.log

The upload still fails though, because of connections being dropped. I'm fairly sure this is a particular issue with this upload (and uploading previously aborted OME-Zarr uploads in general), because in between failed attemptes on this particular failing dataset I managed to upload anther large OME-Zarr dataset fine.