[Bug]: InvalidPart: All non-trailing parts must have the same length

[pablo-salad]

zot version

ghcr.io/project-zot/zot-minimal-linux-amd64:v2.0.0

Describe the bug

Pushing an image to Zot configured with remote store (Cloudflare R2) is reporting InvalidPart: All non-trailing parts must have the same length when is trying to complete a multipart upload.

To reproduce

We prepared a Docker compose to reproduce this error. Just clone this repository.
You'll find a Zot image with the remote configuration and an Ubuntu image with Skopeo installed to push the image.

Configuration
To setup the images, you'll need to have a Cloudflare account, create an R2 bucket and create a R2 API Token to access the bucket.
After this, open compose.yaml and fill these fields:
- R2_ID (it's an alphanumeric in this R2 Dashboard Url https://dash.cloudflare.com/HERE/r2)
- R2_BUCKET
- R2_ACCESS_KEY
- R2_SECRET_KEY

To run the test, execute docker compose up in the folder where the repository was cloned.

Error
After a few minutes, usually uploading a large layer, you will see InvalidPart: All non-trailing parts must have the same length error in the console.

Expected behavior

Able to push any image without errors using Zot with remote storage.

Screenshots

Additional context

The current Zot configuration uses remote storage, in our case, Cloudflare R2 which has S3 API compatibility.
The size of the images uploaded to the registry have an average of 5GB.
We had some luck pushing small images, less than 100MB, changing chunk size in the config but no success with larger ones.

[eusebiu-constantin-petu-dbk]

Hello @pablo-salad

Thank you for trying zot and for the feedback.

This seems to be a known issue in the library we are using for s3. We are waiting for a fix on their side.

Here is the issue: distribution/distribution#3873
Here is the PR which is fixing it: distribution/distribution#3940

[rchincha]

opencontainers/distribution-spec#485

^ also this.