Upload file to s3 from vuejs and Django backend - django

I would like to generate a signed URL in Django, send it to frontend using Axios Ajax and then using that url upload a file directly from Vue JS to S3. In following code when user clicks on upload button - Vue method uploadButtonClicked is called which calls Django function ajaxSendPresignedUrlForS3 which generates presigned post url. This url is passed back to vue uploadButtonClicked and then vue method uploadFile is called.
So far url generation is sucessful. But on posting file to S3 bucket I get error Error: Request failed with status code 403. I have been reading around and making some modifications with the code which results in new errors like 412, 405 etc.
Django code
def ajaxSendPresignedUrlForS3(request):
input=json.loads(request.body.decode('utf-8')); print('ajaxSendPresignedUrlForS3');
S3_BUCKET = os.environ.get('S3_BUCKET')
file_name = input['file_name'][0]
file_type = input['file_type'][0]
s3 = boto3.client('s3',
aws_access_key_id=os.environ.get('AWS_ACCESS_KEY_ID'),
aws_secret_access_key=os.environ.get('AWS_SECRET_ACCESS_KEY'),
region_name='us-east-2',
config=Config(signature_version='s3v4'),
)
presigned_post = s3.generate_presigned_post(
Bucket = S3_BUCKET,
Key = file_name,
Fields = {"Content-Type": 'multipart/form-data'},
Conditions = [{"Content-Type": 'multipart/form-data'}],
ExpiresIn = 300 #seconds
)
return JsonResponse({'data': presigned_post})
Javascript Vue code
Vue method 1 :
uploadButtonClicked:function(){ //gettting presigned POST URL from django
console.log('uploadButtonClicked');
//call axios ajax to get presigned URL from django and then upload file to s3 using axios func "upLoadFile"
axios({
method: 'post',
baseURL: window.location.origin, //we need base url coz this ajax can be called from any page on timeout
url: 'main/ajaxSendPresignedUrlForS3/',
data: {
file_name: this.inputFilesName,
file_type: this.inputFilesType,
},
responseType: 'json', //server will response with this datatype
})
.then ( function (response){
data=response.data;
console.log('uploadButtonClicked succes. data =',data ); //works
this.upLoadFile(data); //upload file to S3
}.bind(this))
.catch ( function (error){
console.log('uploadButtonClicked error=',error);
});
},
Vue method 2:
upLoadFile:function(data){ //upload file directly to s3
console.log('upLoadFile')
var postData = new FormData(); //its type to JS inbuilt form
console.log('data.data.fields=',data.data.fields,'\nKeys =')
for(key in data.data.fields){
console.log(key);
postData.append(key, data.data.fields[key]);
}
postData.append('file', document.getElementById('id_inputButtonReal').files[0]);
console.log('postData=',postData)
axios({
method: 'get',
url: data.data.url+new Date().getTime(),
data: {
postData: postData,
},
// responseType: 'json', //server will response with this datatype
})
.then ( function (response){
data=response.data;
console.log('upLoadFile success');
}.bind(this))
.catch ( function (error){
console.log('upLoadFile error=',error);
});
},
I was able to upload file to s3 directly from Django though. Which probably means my python part is correct:
from boto3.s3.transfer import S3Transfer
myfile='/home/user/img1.jpg';
transfer = S3Transfer(s3); #s3 is declared in above code
transfer.upload_file(myfile, S3_BUCKET,'snake2.jpg') ; print('upload successful');
Thanks

Error explaination: I was sending data to S3 in form of dictionary of dictionary. S3 likes direct postData.
Above Django code is perfectly fine. I have consolidated Javascript code into a smaller version below for comprehension.
uploadButtonClickedforStackoverflow:function(){ //gettting presigned POST URL from django and then uploading file to S3 using that url
axios({
method: 'post',
baseURL: window.location.origin, //we need base url coz this ajax can be called from any page on timeout
url: 'main/ajaxSendPresignedUrlForS3/',
data: { file_name: 'snake.jpg' },//sending filename to django so it can build a presigned POST URL granting frontend permission to upload
responseType: 'json', //server will response with this datatype
})
.then ( function (response){
presigned_post=response.data
var postData = new FormData(); //its type to JS inbuilt form
for(key in presigned_post.fields){ //appending all keys sent by django to our JS FormData object
postData.append(key, presigned_post.fields[key]);
}
var file=document.getElementById('id_inputButtonReal').files[0]; //id_inputButtonReal is id of HTML input element
postData.append('file', file);
axios({
method: 'post',
url: presigned_post.url,
data: postData, //dont sent data as {data:postData}, instead S3 likes it just by itself
})
});
},

Related

It is possible to upload an over 5GB file into S3 via curl with presigned url? [duplicate]

Is there a way to do a multipart upload via the browser using a generated presigned URL?
Angular - Multipart Aws Pre-signed URL
Example
https://multipart-aws-presigned.stackblitz.io/
https://stackblitz.com/edit/multipart-aws-presigned?file=src/app/app.component.html
Download Backend:
https://www.dropbox.com/s/9tm8w3ujaqbo017/serverless-multipart-aws-presigned.tar.gz?dl=0
To upload large files into an S3 bucket using pre-signed url it is necessary to use multipart upload, basically splitting the file into many parts which allows parallel upload.
Here we will leave a basic example of the backend and frontend.
Backend (Serveless Typescript)
const AWSData = {
accessKeyId: 'Access Key',
secretAccessKey: 'Secret Access Key'
};
There are 3 endpoints
Endpoint 1: /start-upload
Ask S3 to start the multipart upload, the answer is an UploadId associated to each part that will be uploaded.
export const start: APIGatewayProxyHandler = async (event, _context) => {
const params = {
Bucket: event.queryStringParameters.bucket, /* Bucket name */
Key: event.queryStringParameters.fileName /* File name */
};
const s3 = new AWS.S3(AWSData);
const res = await s3.createMultipartUpload(params).promise()
return {
statusCode: 200,
headers: {
'Access-Control-Allow-Origin': '*',
'Access-Control-Allow-Credentials': true,
},
body: JSON.stringify({
data: {
uploadId: res.UploadId
}
})
};
}
Endpoint 2: /get-upload-url
Create a pre-signed URL for each part that was split for the file to be uploaded.
export const uploadUrl: APIGatewayProxyHandler = async (event, _context) => {
let params = {
Bucket: event.queryStringParameters.bucket, /* Bucket name */
Key: event.queryStringParameters.fileName, /* File name */
PartNumber: event.queryStringParameters.partNumber, /* Part to create pre-signed url */
UploadId: event.queryStringParameters.uploadId /* UploadId from Endpoint 1 response */
};
const s3 = new AWS.S3(AWSData);
const res = await s3.getSignedUrl('uploadPart', params)
return {
statusCode: 200,
headers: {
'Access-Control-Allow-Origin': '*',
'Access-Control-Allow-Credentials': true,
},
body: JSON.stringify(res)
};
}
Endpoint 3: /complete-upload
After uploading all the parts of the file it is necessary to inform that they have already been uploaded and this will make the object assemble correctly in S3.
export const completeUpload: APIGatewayProxyHandler = async (event, _context) => {
// Parse the post body
const bodyData = JSON.parse(event.body);
const s3 = new AWS.S3(AWSData);
const params: any = {
Bucket: bodyData.bucket, /* Bucket name */
Key: bodyData.fileName, /* File name */
MultipartUpload: {
Parts: bodyData.parts /* Parts uploaded */
},
UploadId: bodyData.uploadId /* UploadId from Endpoint 1 response */
}
const data = await s3.completeMultipartUpload(params).promise()
return {
statusCode: 200,
headers: {
'Access-Control-Allow-Origin': '*',
'Access-Control-Allow-Credentials': true,
// 'Access-Control-Allow-Methods': 'OPTIONS,POST',
// 'Access-Control-Allow-Headers': 'Content-Type',
},
body: JSON.stringify(data)
};
}
Frontend (Angular 9)
The file is divided into 10MB parts
Having the file, the multipart upload to Endpoint 1 is requested
With the UploadId you divide the file in several parts of 10MB and from each one you get the pre-signed url upload using the Endpoint 2
A PUT is made with the part converted to blob to the pre-signed url obtained in Endpoint 2
When you finish uploading each part you make a last request the Endpoint 3
In the example of all this the function uploadMultipartFile
I was managed to achieve this in serverless architecture by creating a Canonical Request for each part upload using Signature Version 4. You will find the document here AWS Multipart Upload Via Presign Url
from the AWS documentation:
For request signing, multipart upload is just a series of regular requests, you initiate multipart upload, send one or more requests to upload parts, and finally complete multipart upload. You sign each request individually, there is nothing special about signing multipart upload request
So I think you should have to generate a presigned url for each part of the multipart upload :(
what is your use case? can't you execute a script from your server, and give s3 access to this server?

In NestJS, downloading s3 bucket's file response is no image in PostMan Test

I'd like to download s3 bucket's image file with that image file's s3 URL.
So I received the s3 URL from client-side.
With that s3 URL, I find one file entity by it.
And, I selected key field from that file entity.
With that key value, I got a file in my s3.
To implement these process, I wrote my code like these.
// controller
#Get('download')
async downloadFile(#Query('url') url: URL, #Response() response) {
return await this.filesService.downloadFile(url, response);
}
// service
async downloadFile(url: URL, response) {
return await this.fileModel
.findOne({ url: url })
.exec()
.then((file) => {
if (!file) throw new NotFoundException('no file');
return this.s3
.getObject({
Bucket: 'attalepro-user-files',
Key: file.key,
})
.promise()
.then((data) => {
response.setHeader('Content-Type', file.type);
response.setHeader('Content-Length', file.size);
response.send(data.Body);
});
});
}
I tested this api with postman, but the postman's response body said no image icon.
What is a missing point in my code??

Error while uploading image on AWS S3 - with Axios & React-Native

I'm trying to upload an image from my Mobile App (with React-Native) on AWS S3 with a presigned URL. I'm using axios to send the request.
The problem is that even if my image is uploaded on AWS, if I download it and try to open it says it's corrupted. I tried to open with Photoshop and it works :/
Creating the formData:
const createFormData = (photo) => {
const data = new FormData();
data.append('image', {
name: photo.fileName, // a name
type: photo.type, // image/jpg
uri: photo.uri, // the uri starting with file://....
});
return data;
};
My PUT request:
const formData = createFormData(responseImage)
axios({
method: "put",
url: awsURL.data.url_thumbnail,
data: formData,
headers: { "Content-Type": "multipart/form-data" },
})
This isn't how it works.
headers: { "Content-Type": "multipart/form-data" }
The content-type multipart/form-data also contains a field called boundary separated by a delimiter. You can get more details here. The article has the details for the format of boundary.
Another example for the same.
Hope it helps!
PS: There are some articles related to parsing data for multipart/form-data that I can't find right now, which explain how to parse the data before uploading so that the data isn't corrupted.

Protect Strapi uploads folder via S3 SignedUrl

uploading files from strapi to s3 works fine.
I am trying to secure the files by using signed urls:
var params = {Bucket:process.env.AWS_BUCKET, Key: `${path}${file.hash}${file.ext}`, Expires: 3000};
var secretUrl = ''
S3.getSignedUrl('getObject', params, function (err, url) {
console.log('Signed URL: ' + url);
secretUrl = url
});
S3.upload(
{
Key: `${path}${file.hash}${file.ext}`,
Body: Buffer.from(file.buffer, 'binary'),
//ACL: 'public-read',
ContentType: file.mime,
...customParams,
},
(err, data) => {
if (err) {
return reject(err);
}
// set the bucket file url
//file.url = data.Location;
file.url = secretUrl;
console.log('FIle URL: ' + file.url);
resolve();
}
);
file.url (secretUrl) contains the correct URL which i can use in browser to retrieve the file.
But whenever reading the file form strapi admin panel no file nor tumbnail is shown.
I figured out that strapi adds a parameter to the file e.g ?2304.4005 which corrupts the get of the file to AWS. Where and how do I change that behaviour
Help is appreciated
Here is my solution to create a signed URL to secure your assets. The URL will be valid for a certain amount of time.
Create a collection type with a media field, which you want to secure. In my example the collection type is called invoice and the media field is called document.
Create an S3 bucket
Install and configure strapi-provider-upload-aws-s3 and AWS SDK for JavaScript
Customize the Strapi controller for your invoice endpoint (in this exmaple I use the core controller findOne)
const { sanitizeEntity } = require('strapi-utils');
var S3 = require('aws-sdk/clients/s3');
module.exports = {
async findOne(ctx) {
const { id } = ctx.params;
const entity = await strapi.services.invoice.findOne({ id });
// key is hashed name + file extension of your entity
const key = entity.document.hash + entity.document.ext;
// create signed url
const s3 = new S3({
endpoint: 's3.eu-central-1.amazonaws.com', // s3.region.amazonaws.com
accessKeyId: '...', // your accessKeyId
secretAccessKey: '...', // your secretAccessKey
Bucket: '...', // your bucket name
signatureVersion: 'v4',
region: 'eu-central-1' // your region
});
var params = {
Bucket:'', // your bucket name
Key: key,
Expires: 20 // expires in 20 seconds
};
var url = s3.getSignedUrl('getObject', params);
entity.document.url = url // overwrite the url with signed url
return sanitizeEntity(entity, { model: strapi.models.invoice });
},
};
It seems like although overwriting controllers and lifecycle of the collection models and strapi-plugin-content-manager to take into account the S3 signed urls, one of the Strapi UI components adds a strange hook/refs ?123.123 to the actual url that is received from the backend, resulting in the following error from AWS There were headers present in the request which were not signed when trying to see images from the CMS UI.
Screenshot with the faulty component
After digging the code & node_modules used by Strapi, it seems like you will find the following within strapi-plugin-upload/admin/src/components/CardPreview/index.js
return (
<Wrapper>
{isVideo ? (
<VideoPreview src={url} previewUrl={previewUrl} hasIcon={hasIcon} />
) : (
// Adding performance.now forces the browser no to cache the img
// https://stackoverflow.com/questions/126772/how-to-force-a-web-browser-not-to-cache-images
<Image src={`${url}${withFileCaching ? `?${cacheRef.current}` : ''}`} />
)}
</Wrapper>
);
};
CardPreview.defaultProps = {
extension: null,
hasError: false,
hasIcon: false,
previewUrl: null,
url: null,
type: '',
withFileCaching: true,
};
The default is set to true for withFileCaching, which therefore appends the const cacheRef = useRef(performance.now()); query param to the url for avoiding browser caches.
By setting it to false, or leaving just <Image src={url} /> should solve the issue of the extra query param and allow you to use S3 signed URLs previews also from Strapi UI.
This would also translate to use the docs https://strapi.io/documentation/developer-docs/latest/development/plugin-customization.html to customize the module strapi-plugin-upload in your /extensions/strapi-plugin-upload/...

Fetch API for Django POST requests

I'm trying to remove jQuery from a React/Redux/Django webapp and replace the $.ajax method with the Fetch API. I've more or less got all my GET requests working fine and I seem to be able to hit my POST requests, but I cannot seem to format my request in such a way as to actually get my POST data into the Django request.POST object. Every time I hit my /sign_in view, the request.POST object is empty. My entire app's backend is built around using Django forms (no Django templates, just React controlled components) and I would really like to not have to rewrite all my views to use request.body or request.data.
Here is all the code I can think that would be relevant, please let me know if there's more that would be helpful:
This is the curried function I use to build my full POST data and attach the CSRF token:
const setUpCsrfToken = () => {
const csrftoken = Cookies.get('csrftoken')
return function post (url, options) {
const defaults = {
'method': 'POST',
'credentials': 'include',
'headers': {
'X-CSRFToken': csrftoken,
'Content-Type': 'application/x-www-form-urlencoded'
}
}
const merged = merge(options, defaults)
return fetch(url, merged)
}
}
export const post = setUpCsrfToken()
This is the API method I use from my React app:
export const signIn = data => {
return post('/api/account/sign_in/', data)
}
The data when it is originally packaged up in the React app itself is as simple as an object with string values:
{
email: 'email#email.com',
password: 'password
}
I've looked at these questions and found them to be nominally helpful, but I can't figure out to synthesize an answer for myself that takes into account what I assume is some of the intricacies of Django:
POST Request with Fetch API?
Change a jquery ajax POST request into a fetch api POST
Convert JavaScript object into URI-encoded string
Is there a better way to convert a JSON packet into a query string?
Thanks!
You have to set the appropriate X-Requested-With header. jQuery does this under the hood.
X-Requested-With: XMLHttpRequest
So, in your example, you would want something like:
const setUpCsrfToken = () => {
const csrftoken = Cookies.get('csrftoken')
return function post (url, options) {
const defaults = {
'method': 'POST',
'credentials': 'include',
'headers': new Headers({
'X-CSRFToken': csrftoken,
'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8',
'X-Requested-With': 'XMLHttpRequest'
})
}
const merged = merge(options, defaults)
return fetch(url, merged)
}
}