I am using AWS javascript SDK to upload a file to S3 using multipart upload.
// Use S3 ManagedUpload class as it supports multipart uploads
var upload = new AWS.S3.ManagedUpload({
params: {
Bucket: albumBucketName,
Key: photoKey,
Body: file,
ACL: "public-read"
}
});
But I would like to also show the speed at which the upload is happening in the UI. Document doesn't provide any API to get the speed. So would like to know how to calculate the upload speed.
regards
Achuth
.on('httpUploadProgress', function(e) {
console.log(e.loaded);
});
You can use .on listner , and e.loaded will provide you the uploaded bytes value, which can be used to calculate percentage of upload.
new AWS.S3.ManagedUpload({
params: {
Bucket: albumBucketName,
Key: photoKey,
Body: file,
ACL: "public-read"
}
}).on('httpUploadProgress', function(e) {
console.log(e.loaded);
});
Related
Is there a way to do a multipart upload via the browser using a generated presigned URL?
Angular - Multipart Aws Pre-signed URL
Example
https://multipart-aws-presigned.stackblitz.io/
https://stackblitz.com/edit/multipart-aws-presigned?file=src/app/app.component.html
Download Backend:
https://www.dropbox.com/s/9tm8w3ujaqbo017/serverless-multipart-aws-presigned.tar.gz?dl=0
To upload large files into an S3 bucket using pre-signed url it is necessary to use multipart upload, basically splitting the file into many parts which allows parallel upload.
Here we will leave a basic example of the backend and frontend.
Backend (Serveless Typescript)
const AWSData = {
accessKeyId: 'Access Key',
secretAccessKey: 'Secret Access Key'
};
There are 3 endpoints
Endpoint 1: /start-upload
Ask S3 to start the multipart upload, the answer is an UploadId associated to each part that will be uploaded.
export const start: APIGatewayProxyHandler = async (event, _context) => {
const params = {
Bucket: event.queryStringParameters.bucket, /* Bucket name */
Key: event.queryStringParameters.fileName /* File name */
};
const s3 = new AWS.S3(AWSData);
const res = await s3.createMultipartUpload(params).promise()
return {
statusCode: 200,
headers: {
'Access-Control-Allow-Origin': '*',
'Access-Control-Allow-Credentials': true,
},
body: JSON.stringify({
data: {
uploadId: res.UploadId
}
})
};
}
Endpoint 2: /get-upload-url
Create a pre-signed URL for each part that was split for the file to be uploaded.
export const uploadUrl: APIGatewayProxyHandler = async (event, _context) => {
let params = {
Bucket: event.queryStringParameters.bucket, /* Bucket name */
Key: event.queryStringParameters.fileName, /* File name */
PartNumber: event.queryStringParameters.partNumber, /* Part to create pre-signed url */
UploadId: event.queryStringParameters.uploadId /* UploadId from Endpoint 1 response */
};
const s3 = new AWS.S3(AWSData);
const res = await s3.getSignedUrl('uploadPart', params)
return {
statusCode: 200,
headers: {
'Access-Control-Allow-Origin': '*',
'Access-Control-Allow-Credentials': true,
},
body: JSON.stringify(res)
};
}
Endpoint 3: /complete-upload
After uploading all the parts of the file it is necessary to inform that they have already been uploaded and this will make the object assemble correctly in S3.
export const completeUpload: APIGatewayProxyHandler = async (event, _context) => {
// Parse the post body
const bodyData = JSON.parse(event.body);
const s3 = new AWS.S3(AWSData);
const params: any = {
Bucket: bodyData.bucket, /* Bucket name */
Key: bodyData.fileName, /* File name */
MultipartUpload: {
Parts: bodyData.parts /* Parts uploaded */
},
UploadId: bodyData.uploadId /* UploadId from Endpoint 1 response */
}
const data = await s3.completeMultipartUpload(params).promise()
return {
statusCode: 200,
headers: {
'Access-Control-Allow-Origin': '*',
'Access-Control-Allow-Credentials': true,
// 'Access-Control-Allow-Methods': 'OPTIONS,POST',
// 'Access-Control-Allow-Headers': 'Content-Type',
},
body: JSON.stringify(data)
};
}
Frontend (Angular 9)
The file is divided into 10MB parts
Having the file, the multipart upload to Endpoint 1 is requested
With the UploadId you divide the file in several parts of 10MB and from each one you get the pre-signed url upload using the Endpoint 2
A PUT is made with the part converted to blob to the pre-signed url obtained in Endpoint 2
When you finish uploading each part you make a last request the Endpoint 3
In the example of all this the function uploadMultipartFile
I was managed to achieve this in serverless architecture by creating a Canonical Request for each part upload using Signature Version 4. You will find the document here AWS Multipart Upload Via Presign Url
from the AWS documentation:
For request signing, multipart upload is just a series of regular requests, you initiate multipart upload, send one or more requests to upload parts, and finally complete multipart upload. You sign each request individually, there is nothing special about signing multipart upload request
So I think you should have to generate a presigned url for each part of the multipart upload :(
what is your use case? can't you execute a script from your server, and give s3 access to this server?
I am using AWS SDK in Server Side with Node.JS and having issue with uploading files as formData from client side.
On the server side I have simple route, which creates upload link, where video will be uploaded later directly from client side.
I am using S3 getSignedUrl method for generating that link with putObject, which creates PUT request for client, but causes very strange issue with formData.
Video uploaded as formData is not behaving correctly - instead of playing it S3 uploaded url downloads that video and it is also broken.
Here is simple how i configure that method on server side:
this.s3.getSignedUrl(
'putObject',
{
Bucket: '<BUCKET_NAME>',
ContentType: `${contentType}` -> video/mp4 as a rule,
Key: key,
},
(err, url) => {
if (err) {
reject(err)
} else {
resolve(url)
}
},
)
axios put request with blob is actually working, but not for formData.
axios.put(url, file, {
headers: {
'Content-Type': file.type,
},
onUploadProgress: ({ total, loaded }) => {
setProgress((loaded / total) * 100)
},
})
This is working version, but when I try to add file to formData, it is uploaded to S3, but video downloads instead of playing.
I do not have big experience in AWS, so if somebody knows how to handle that issue, I will be thankfull
After a few days of trying to upload a video to AWS, I have successfully (almost) been able to. The main problem I am seeing is when I head to my S3 bucket, the file has a Size 0 B. I was hoping to see what I might be doing wrong that is causing this to occur.
On the backend I get a presignedUrl such as:
const s3 = new AWS.S3({
accessKeyId: ACCESSKEY_ID,
secretAccessKey: SECRETKEY
});
const s3Params = {
Bucket: BUCKET_NAME,
Key: uuidv4() + '.mov',
Expires: 60 * 10,
ContentType: 'mov',
ACL: 'public-read'
};
let url = await s3.getSignedUrl('putObject', s3Params);
return { url };
Once I have the url for upload. On the frontend the way I am sending the file is:
const uploadFileToS3 = async (uri) => {
const type = video.uri.split('.').pop();
const respo = await fetch(uri, {
method: 'PUT',
body: {
url: video.uri,
type,
name: 'testing'
},
headers: {
'Content-Type': type,
'x-amz-acl': 'public-read'
}
});
const some = await JSON.stringify(respo);
It does seem to be saving something since I see it in the bucket but am unable to download or view it. Just shows an empty page and it feels like nothing (the video) possibly was uploaded to S3. Any pointers to where I might be going wrong in uploading a video to S3?
Thank you for all the help.
You can not specify an URL when you upload a file. You need 2 fetches:
the first one downloads the video from video.uri
the second uploads the video to S3: body: blob
To download a file as a blob, use response.blob(). Then you can use that to upload the file (here is an example).
i'm trying to read data from AWS S3 Bucket using JavaScript but getting
error :
"Error: Missing credentials in config"
AWS.config.update({
"region": "eu-west-1"
});
var params = { Bucket: <BucketName>, Key: "data.json" };
new AWS.S3().getObject(params, function (err, json_data) {
if (!err) {
var json = JSON.parse(new Buffer(json_data.Body).toString("utf8"));
console.log(json);
}
else
console.log(err);
});
even if i tried without AWS.config.update i'm getting this error.
any idea?
AFAIK if you wish to access a bucket which is not public then you will need to supply your AWS credentials along with the request. Here's the SDK page for building an AWS.credentials object that you put into your options when making the AWS.S3 object.
https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/Credentials.html
No example because I am not a JS dev and can't write it out of memory, sorry!
I managed to get my generated pdf uploaded to s3 from my node JS server. Pdf looks okay on my local folder but when I tried to access it from the AWS console, it indicates "Failed to load PDF document".
I have tried uploading it via the s3.upload and s3.putObject APIs, (for the putObject I also used an .on finish checker to ensure that the file has been fully loaded before sending the request). But the file in the S3 bucket is still the same (small) size, 26 bytes and cannot be loaded. Any help is greatly appreciated!!!
var pdfDoc = printer.createPdfKitDocument(inspectionReport);
var writeStream = fs.createWriteStream('pdfs/inspectionReport.pdf');
pdfDoc.pipe(writeStream);
pdfDoc.end();
writeStream.on('finish', function(){
const s3 = new aws.S3();
aws.config.loadFromPath('./modules/awsconfig.json');
var s3Params = {
Bucket: S3_BUCKET,
Key: 'insp_report_test.pdf',
Body: '/pdf/inspectionReport.pdf',
Expires: 60,
ContentType: 'application/pdf'
};
s3.putObject(s3Params, function(err,res){
if(err)
console.log(err);
else
console.log(res);
})
I realised that pdfDoc.end() must come before piping starts. Have also used a callback to ensure that s3 upload is called after pdf write is finished. See code below, hope it helps!
var pdfDoc = printer.createPdfKitDocument(inspectionReport);
pdfDoc.end();
async.parallel([
function(callback){
var writeStream = fs.createWriteStream('pdfs/inspectionReport.pdf');
pdfDoc.pipe(writeStream);
console.log('pdf write finished!');
callback();
}
], function(err){
const s3 = new aws.S3();
var s3Params = {
Bucket: S3_BUCKET,
Key: 'insp_report_test.pdf',
Body: pdfDoc,
Expires: 60,
ContentType: 'application/pdf'
};
s3.upload(s3Params, function(err,result){
if(err) console.log(err);
else console.log(result);
});
}
);