Multipart upload with presigned urls - Scaleway S3-compatible object storage - amazon-web-services

Question

I’m trying to have multipart upload working on Scaleway Object Storage (S3 compatible) with presigned urls and I’m getting errors (403) on preflight request generated by the browser but my CORS settings seems correctly set. (Basically wildcard on allowed headers and origins).
The error comes with a 403 status code and is as follow:
<?xml version='1.0' encoding='UTF-8'?>
<Error><Code>AccessDenied</Code><Message>Access Denied.</Message><RequestId>...</RequestId></Error>
I’m stuck on this one for a while now, I tried to copy the pre-flight request from my browser to reproduce it elsewhere and tried to tweak it a little bit.
Removing the query params from the url of the pre-flight request make the request successful (returns a 200 with Access-Control-Allow-* response headers correctly set) but this is obviously not the browser behavior...
This Doesn’t work (secrets, keys and names have been changed)
curl 'https://bucket-name.s3.fr-par.scw.cloud/tmp-screenshot-2021-01-20-at-16-21-33.png?AWSAccessKeyId=XXXXXXXXXXXXXXXXXXXX&Expires=1638217988&Signature=NnP1XLlcvPzZnsUgDAzm1Uhxri0%3D&partNumber=1&uploadId=OWI1NWY5ZGrtYzE3MS00MjcyLWI2NDAtNjFkYTM1MTRiZTcx' -X OPTIONS -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:94.0) Gecko/20100101 Firefox/94.0' -H 'Accept: */*' -H 'Accept-Language: en-US,en;q=0.5' --compressed -H 'Referer: http://domain.tech/' -H 'Access-Control-Request-Method: PUT' -H 'Origin: http://domain.tech' -H 'DNT: 1' -H 'Connection: keep-alive' -H 'Sec-Fetch-Dest: empty' -H 'Sec-Fetch-Mode: no-cors' -H 'Sec-Fetch-Site: cross-site' -H 'Pragma: no-cache' -H 'Cache-Control: no-cache'
This Works (secrets, keys and names have been changed)
curl 'https://bucket-name.s3.fr-par.scw.cloud/tmp-screenshot-2021-01-20-at-16-21-33.png' -X OPTIONS -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:94.0) Gecko/20100101 Firefox/94.0' -H 'Accept: */*' -H 'Accept-Language: en-US,en;q=0.5' --compressed -H 'Referer: http://domain.tech/' -H 'Access-Control-Request-Method: PUT' -H 'Origin: http://domain.tech' -H 'DNT: 1' -H 'Connection: keep-alive' -H 'Sec-Fetch-Dest: empty' -H 'Sec-Fetch-Mode: no-cors' -H 'Sec-Fetch-Site: cross-site' -H 'Pragma: no-cache' -H 'Cache-Control: no-cache'
The url comes from the aws-sdk and is generated this way :
const S3Client = new S3({
credentials: {
accessKeyId: env.SCW_ACCESS_KEY,
secretAccessKey: env.SCW_SECRET_KEY,
},
endpoint: `https://s3.${env.SCW_REGION}.scw.cloud`,
})
S3Client.getSignedUrlPromise('uploadPart', {
Bucket: bucket,
Key: key,
UploadId: multipartUpload.UploadId,
PartNumber: idx + 1,
})
and used this way in frontend:
// url being the url generated in backend as demonstrated above
const response = await fetch(url, {
method: 'PUT',
body: filePart,
signal: abortController.signal,
})
If anyone can give me a hand at this or that would be great!

Answer 1

As it turns out, Scaleway Object Storage is not fully S3-compatible on this case.
Here is a workaround:
Install aws4 library to sign request easily (or follow this scaleway doc to manually sign your request)
Form your request exactly as per stated in this other scaleway doc (this is where aws-sdk behavior differs, it generates an url with AWSAccessKeyId, Expires and Signature query params that cause the scaleway API to fail. Scaleway API only wants partNumber and uploadId).
Return the generated url and headers to the frontend
// Backend code
const signedRequest = aws4.sign(
{
method: 'PUT',
path: `/${key}?partNumber=${idx + 1}&uploadId=${
multipartUpload.UploadId
}`,
service: 's3',
region: env.SCW_REGION,
host: `${bucket}.s3.${env.SCW_REGION}.scw.cloud`,
},
{
accessKeyId: env.SCW_ACCESS_KEY,
secretAccessKey: env.SCW_SECRET_KEY,
},
)
return {
url: `https://${signedRequest.host}${signedRequest.path}`,
headers: Object.keys(signedRequest.headers).map((key) => ({
key,
value: signedRequest.headers[key] as string,
})),
}
And then in frontend:
// Frontend code
const headers = signedRequest.headers.reduce<Record<string, string>>(
(acc, h) => ({ ...acc, [h.key]: h.value }),
{},
)
const response = await fetch(signedRequest.url, {
method: 'PUT',
body: filePart,
headers,
signal: abortController.signal,
})
Scaleway knows this issue as I directly discussed with their support team and they are putting some effort in order to be as compliant as possible with S3. This issue might be fixed by the time you read this.
Thanks to them for the really quick response time and for taking this seriously.