The Content-MD5 you specified is invalid for multi-part uploads - amazon-web-services

Question

I am using the awd-sdk for nodejs and have a working upload.
const checksum = await this.getChecksum(path);
const payload: S3.PutObjectRequest = {
Bucket: bucket,
Key: key,
Body: fs.createReadStream(path),
ContentMD5: checksum,
};
return this.s3.upload(payload).promise();
This piece of code works great for small files and takes advantage of ContentMD5 which automatically verifies the file integrity.
Content-MD5
The base64-encoded 128-bit MD5 digest of the message (without the headers) according to RFC 1864. This header can be used as a message integrity check to verify that the data is the same data that was originally sent. Although it is optional, we recommend using the Content-MD5 mechanism as an end-to-end integrity check. For more information about REST request authentication, see REST Authentication.
https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html
However it doesn't work for multipart uploads.
The Content-MD5 you specified is invalid for multi-part uploads.
That makes sense because we send the file chunk by chunk but then, I am wondering how am I supposed to use this feature with multipart uploads?

Answer 1

I too faced this and did multiple testing. Finally found the answer and verified it in my own way. If you guys know a much better way please let me know here.
Being said that, here's how I solved the issue.
When you create the S3 client you have to create it as below.
const s3 = new AWS.S3({ computeChecksums: true });
Then you can define the s3 upload parameters like below
var fileData = Buffer.from(fileContentString, 'binary');
var s3params = {
Bucket: bucketName,
Key: folderName + "/" + fileName,
ContentType: 'binary',
Body: fileData
};
Then do the upload as below.
await s3.upload(s3params).promise()
.then(data => {
// use this log to verify whether the md5 checksum was verified
console.log(`File uploadedd successfully ${JSON.stringify(data)}`);
// handle success upload
})
.catch(error => {
//handle the error
});
After upload is successful here's how I verified its working.
Check the item in the S3. In the document details, Object Overview section check the E-tag. You should see something like this 7e35f58f134d8914604c9fc6c35b2db7-9. This number after the - means how many parts were uploaded. For bigger files there should be a number bigger than 1. In this case it is 9.
Check the log in the console. (The log with the comment in above code.) You'll see something like below.
{
"Location": "https://bucketname.s3.region.amazonaws.com/folderName/fileName",
"Bucket": "bucketName",
"Key": "folderName/fileName",
"ETag": "\"7e35f58f134d8914604c9fc6c35b2db7-9\""
}
If you are debugging you can further test it by printing the s3 upload request.
var uploadRequest = s3.upload(s3params);
console.log(`S3 Request ${JSON.stringify(uploadRequest)}`);
This will print the s3 client configurations. Check whether the 'computeChecksums' is set to true.
I tried to verify with the s3.putObject as well. But when I print the request it didn't show me the md5Checksum config in header as it is intended to be. Also it gave me cyclic JSON stringifying error when I tried to log the whole object in the same way as in the third point for upload. So I printed httpRequest only.
var uploadRequest = s3.putObject(s3params);
console.log(`S3 Request ${JSON.stringify(uploadRequest.httpRequest)}`
//Following gives the JSON stringifying cyclic issue
//console.log(`S3 Request ${JSON.stringify(uploadRequest)}`);
Appreciate if someone can tell how to do this and verify with putObject as well.