Uppy - How do you upload to s3 via multipart? Using companion? - amazon-web-services

https://uppy.io/docs/aws-s3-multipart/
Uppy multipart plugin sounds like exactly what I need but I can't see how to do the backend part of things. The impression I get is that I need to setup a companion to route the upload to S3 but can't find any details on setting up the companion for this.
I can see lots of references about using Companion to fetch external content but none on the multipart S3 uploading.
I neither see anywhere inside Uppy to provide AWS credentials which makes me think Companion even more.
But there are 4 steps to complete a multipart upload and I can't see how providing one companion url will help Uppy.
Thanks in advance to anyone who can help or jog me in the right direction.

Providing Uppy a companion URL makes it so that Uppy will fire off a series of requests to the-passed-url.com/s3/multipart. You then need to configure your server to handle these requests. Your server will be where your credentials are handled for AWS.
In short when you click the upload button in Uppy, this is what happens:
Uppy sends a post request to /s3/multipart to create/initiate the multipart upload.
Using the data returned from the previous request, Uppy will send a get request to /s3/multipart/{uploadId} to generate AWS S3 pre-signed URLs to use for uploading the parts.
Uppy will then upload the parts using the pre-signed URLs from the previous request.
Finally, Uppy will send a post request to /s3/multipart/{uploadId}/complete to complete the multipart upload.
I was able to accomplish this using Laravel/Vue. I don't know what your environment is but I've posted my solution which should help, especially if your server is using PHP.
Configuring Uppy to Use Multipart Uploads with Laravel/Vue

I am sharing code snippets for AWS S3 Multipart [github]
If you add Companion to the mix, your users will be able to select files from remote sources, such as Instagram, Google Drive, and Dropbox, bypassing the client (so a 5 GB video isn’t eating into your users’ data plans), and then uploaded to the final destination. Files are removed from Companion after an upload is complete, or after a reasonable timeout. Access tokens also don’t stick around for long, for security reasons.
Setup companion server:
1: Setup s3 configuration.
Uppy automatically generates the upload URL and puts the file in the uploads directory.
s3: {
getKey: (req, filename) =>{
return `uploads/${filename}`;
},
key: 'AWS KEY',
secret: 'AWS SECRET',
bucket: 'AWS BUCKET NAME',
},
2: Support upload from a remote resource
Uppy handles everything for us. We just need to provide a secret key and token from different remote resources like Instagram, drive, etc.
example: Drive upload
Generate google key and secrete from google and add it to code
Add redirect URL for authentication
3: Run node server locally
const fs = require('fs')
const path = require('path')
const rimraf = require('rimraf')
const companion = require('#uppy/companion')
const app = require('express')()
const DATA_DIR = path.join(__dirname, 'tmp')
app.use(require('cors')({
origin: true,
credentials: true,
}))
app.use(require('cookie-parser')())
app.use(require('body-parser').json())
app.use(require('express-session')({
secret: 'hello planet',
}))
const options = {
providerOptions: {
drive: {
key: 'YOUR GOOGLE DRIVE KEY',
secret: 'YOUR GOOGLE DRIVE SECRET'
},
s3: {
getKey: (req, filename) =>{
return `uploads/${filename}`;
} ,
key: 'AWS KEY',
secret: 'AWS SECRET',
bucket: 'AWS BUCKET NAME',
},
},
server: { host: 'localhost:3020' },
filePath: DATA_DIR,
secret: 'blah blah',
debug: true,
}
try {
fs.accessSync(DATA_DIR)
} catch (err) {
fs.mkdirSync(DATA_DIR)
}
process.on('exit', () => {
rimraf.sync(DATA_DIR)
})
app.use(companion.app(options))
// handle server errors
const server = app.listen(3020, () => {
console.log('listening on port 3020')
})
companion.socket(server, options)
Setup client:
1: client HTML code:
This code will allow upload from the drive, webcam, local, etc. You can customize it to support more remote places.
Add companion URL as your above node server running URL(http://localhost:3020)
<!doctype html>
<html>
<head>
<meta charset="utf-8">
<title>Uppy</title>
<link href="https://releases.transloadit.com/uppy/v1.29.1/uppy.min.css" rel="stylesheet">
</head>
<body>
<div id="drag-drop-area"></div>
<script src="https://releases.transloadit.com/uppy/v1.29.1/uppy.min.js"></script>
<script>
Uppy.Core({
debug: false,
autoProceed: false,
restrictions: {
maxNumberOfFiles: 5,
}
}).
use(Uppy.AwsS3Multipart, {
limit: 4,
companionUrl: 'http://localhost:3020'
}).
use(Uppy.Dashboard, {
inline: true,
showProgressDetails: true,
showLinkToFileUploadResult: false,
proudlyDisplayPoweredByUppy: false,
target: '#drag-drop-area',
}).use(Uppy.GoogleDrive, { target: Uppy.Dashboard, companionUrl: 'http://localhost:3020' })
.use(Uppy.Url, { target: Uppy.Dashboard, companionUrl: 'http://localhost:3020' })
.use(Uppy.Webcam, { target: Uppy.Dashboard, companionUrl: 'http://localhost:3020' });
</script>
</body>
</html>

Related

AWS Amplify Multiple frontends single API & Backend

I have 2 amplify webapps and I want that both are using the same AWS backend. So i followed the instructions on https://docs.amplify.aws/cli/teams/multi-frontend/#workflow. App A has the full amplify backend src and App B shall use these too. So I run on App B.
amplify pull
>Amplify AppID found: df4xxxxxxx. Amplify App name is: XXXXX
>Backend environment dev found in Amplify Console app: XXXXX
Seems to work.
But when I now try to make an api call via:
AWS_API = 'api_name_on_aws';
async getUserInfosByUsername(username) {
var userInfos;
await API.get(this.AWS_API, `/users/infos/testuser`,
{
headers: {},
response: true,
body: {},
queryStringParameters: {},
})
.then((response) => {
userInfos = response;
})
.catch((error) => {
console.log(error.response);
});
return userInfos;
}
then no api request will send. (I can see within the google chrome dev console/network that no request is send).
The "request" method is just return "undefined" and thats all... On App A everything is working fine.
Did I miss something? Should I do something else that App B can use the API of APP A?

Uppy Companion doesn't work for > 5GB files with Multipart S3 uploads

Our app allow our clients large file uploads. Files are stored on AWS/S3 and we use Uppy for the upload, and dockerize it to be used under a kubernetes deployment where we can up the number of instances.
It works well, but we noticed all > 5GB uploads fail. I know uppy has a plugin for AWS multipart uploads, but even when installed during the container image creation, the result is the same.
Here's our Dockerfile. Has someone ever succeeded in uploading > 5GB files to S3 via uppy? IS there anything we're missing?
FROM node:alpine AS companion
RUN yarn global add #uppy/companion#3.0.1
RUN yarn global add #uppy/aws-s3-multipart
ARG UPPY_COMPANION_DOMAIN=[...redacted..]
ARG UPPY_AWS_BUCKET=[...redacted..]
ENV COMPANION_SECRET=[...redacted..]
ENV COMPANION_PREAUTH_SECRET=[...redacted..]
ENV COMPANION_DOMAIN=${UPPY_COMPANION_DOMAIN}
ENV COMPANION_PROTOCOL="https"
ENV COMPANION_DATADIR="COMPANION_DATA"
# ENV COMPANION_HIDE_WELCOME="true"
# ENV COMPANION_HIDE_METRICS="true"
ENV COMPANION_CLIENT_ORIGINS=[...redacted..]
ENV COMPANION_AWS_KEY=[...redacted..]
ENV COMPANION_AWS_SECRET=[...redacted..]
ENV COMPANION_AWS_BUCKET=${UPPY_AWS_BUCKET}
ENV COMPANION_AWS_REGION="us-east-2"
ENV COMPANION_AWS_USE_ACCELERATE_ENDPOINT="true"
ENV COMPANION_AWS_EXPIRES="3600"
ENV COMPANION_AWS_ACL="public-read"
# We don't need to store data for just S3 uploads, but Uppy throws unless this dir exists.
RUN mkdir COMPANION_DATA
CMD ["companion"]
EXPOSE 3020
EDIT:
I made sure I had:
uppy.use(AwsS3Multipart, {
limit: 5,
companionUrl: '<our uppy url',
})
And it still doesn't work- I see all the chunks of the 9GB file sent on the network tab but as soon as it hits 100% -- uppy throws an error "cannot post" (to our S3 url) and that's it. failure.
Has anyone ever encountered this? upload goes fine till 100%, then the last chunk gets HTTP error 413, making the entire upload fail.
Thanks!
Here I'm adding some code samples from my repository that will help you to understand the flow of using the BUSBOY package to stream the data to the S3 bucket. Also, I'm adding the reference links here for you to get the package details I'm using.
https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/clients/client-s3/index.html
https://www.npmjs.com/package/busboy
export const uploadStreamFile = async (req: Request, res: Response) => {
const busboy = new Busboy({ headers: req.headers });
const streamResponse = await busboyStream(busboy, req);
const uploadResponse = await s3FileUpload(streamResponse.data.buffer);
return res.send(uploadResponse);
};
const busboyStream = async (busboy: any, req: Request): Promise<any> {
return new Promise((resolve, reject) => {
try {
const fileData: any[] = [];
let fileBuffer: Buffer;
busboy.on('file', async (fieldName: any, file: any, fileName: any, encoding: any, mimetype: any) => {
// ! File is missing in the request
if (!fileName)
reject("File not found!");
let totalBytes: number = 0;
file.on('data', (chunk: any) => {
fileData.push(chunk);
// ! given code is only for logging purpose
// TODO will remove once project is live
totalBytes += chunk.length;
console.log('File [' + fieldName + '] got ' + chunk.length + ' bytes');
});
file.on('error', (err: any) => {
reject(err);
});
file.on('end', () => {
fileBuffer = Buffer.concat(fileData);
});
});
// ? Haa, finally file parsing wen't well
busboy.on('finish', () => {
const responseData: ResponseDto = {
status: true, message: "File parsing done", data: {
buffer: fileBuffer,
metaData
}
};
resolve(responseData)
console.log('Done parsing data! -> File uploaded');
});
req.pipe(busboy);
} catch (error) {
reject(error);
}
});
}
const s3FileUpload = async (fileData: any): Promise<ResponseDto> {
try {
const params: any = {
Bucket: <BUCKET_NAME>,
Key: <path>,
Body: fileData,
ContentType: <content_type>,
ServerSideEncryption: "AES256",
};
const command = new PutObjectCommand(params);
const uploadResponse: any = await this.S3.send(command);
return { status: true, message: "File uploaded successfully", data: uploadResponse };
} catch (error) {
const responseData = { status: false, message: "Monitor connection failed, please contact tech support!", error: error.message };
return responseData;
}
}
In the AWS S3 service in a single PUT operation, you can upload a single object up to 5 GB in size.
To upload > 5GB files to S3 you need to use the multipart upload S3 API, and also the AwsS3Multipart Uppy API.
Check your upload code to understand if you are using AWSS3Multipart correctly, setting the limit properly for example, in this case a limit between 5 and 15 is recommended.
import AwsS3Multipart from '#uppy/aws-s3-multipart'
uppy.use(AwsS3Multipart, {
limit: 5,
companionUrl: 'https://uppy-companion.myapp.net/',
})
Also, check this issue on Github Uploading a large >5GB file to S3 errors out #1945
If you're getting Error: request entity too large in your Companion server logs I fixed this in my Companion express server by increasing the body-parser limit:
app.use(bodyparser.json({ limit: '21GB', type: 'application/json' }))
This is a good working example of Uppy S3 MultiPart uploads (without this limit increased): https://github.com/jhanitesh10/uppy
I'm able to upload files up to a (self-imposed) limit of 20GB using this code.

S3 - Video, uploaded with getSignedUrl link, does not play and is downloaded in wrong format

I am using AWS SDK in Server Side with Node.JS and having issue with uploading files as formData from client side.
On the server side I have simple route, which creates upload link, where video will be uploaded later directly from client side.
I am using S3 getSignedUrl method for generating that link with putObject, which creates PUT request for client, but causes very strange issue with formData.
Video uploaded as formData is not behaving correctly - instead of playing it S3 uploaded url downloads that video and it is also broken.
Here is simple how i configure that method on server side:
this.s3.getSignedUrl(
'putObject',
{
Bucket: '<BUCKET_NAME>',
ContentType: `${contentType}` -> video/mp4 as a rule,
Key: key,
},
(err, url) => {
if (err) {
reject(err)
} else {
resolve(url)
}
},
)
axios put request with blob is actually working, but not for formData.
axios.put(url, file, {
headers: {
'Content-Type': file.type,
},
onUploadProgress: ({ total, loaded }) => {
setProgress((loaded / total) * 100)
},
})
This is working version, but when I try to add file to formData, it is uploaded to S3, but video downloads instead of playing.
I do not have big experience in AWS, so if somebody knows how to handle that issue, I will be thankfull

elastic transcode is not writing metadata in s3, AWS

I am using lambda function to transcode the video file I upload. Here is the code I am using in the lambda function.
var params = {
PipelineId: pipelineId,
Input: {
Key: inputKey
},
Outputs: [{
Key: outputKey,
PresetId: transcoderPresetID,
}],
UserMetadata : {jid : 'test', vid: v001 }
}
but when I check the metadata on the s3 object that was written by elastic transcoder, all I can see is "content-type": "video/mp4"
My log files are not showing any errors, am I missing something. Please let me know. Thank you
The UserMetadata is not used when saving an object to S3. The UserMetadata is sent as part of the job status notification as documented here:
https://docs.aws.amazon.com/elastictranscoder/latest/developerguide/notifications.html
If you wish to add custom metadata on the S3 object after transcoding, you could perform an object copy. For example:
$s3Client->copyObject(
$sourceObject,
$sourceObject,
array(
"jid" => "test",
"vid" => "v001",
)
);

Device based redirection on CloudFront serving from S3 origin

Is there a way to redirect the user to the mobile version of a web app say m.foobar.com based on the User Agent header using CloudFront?
I did read up on header caching using the user's device type using CloudFront-Is-Mobile-Viewer header. But, I can only whitelist it if I'm using a custom origin to serve my assets (ELB or an EC2 instance). In such a scenario, I could edit my server configuration to handle the redirection.
However, I'm using S3 to serve my application now and would prefer a solution within the CloudFront/S3 ecosystem.
Edit:
For S3 distributions, I DONOT have access to the CloudFront-Is-Mobile-Viewer and other CF headers.
Any help, pointers would be greatly appreciated!
Background Material: http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/header-caching.html
https://aws.amazon.com/blogs/aws/enhanced-cloudfront-customization/
Here's how I'd solve it.
Lambda#Edge Function
'use strict';
exports.handler = (event, context, callback) => {
/*
* If mobile, redirect to mobile domain
*/
const isMobileHeader = 'CloudFront-Is-Mobile-Viewer'
const request = event.Records[0].cf.request;
const headers = request.headers;
let response = event.Records[0].cf.response;
if (headers[isMobileHeader.toLowerCase()] && headers[isMobileHeader.toLowerCase()] == "true") {
response = {
status: '302',
statusDescription: 'Found',
headers: {
location: [{
key: 'Location',
value: 'http://m.foobar.com',
}],
},
};
callback(null, response);
};
CloudFront Distribution
Behaviours:
Default:
Cache Based on Selected Request Headers: Whitelist
Whitelist Headers:
- CloudFront-Is-Mobile-Viewer
Lambda Function Associations:
Event Type: Viewer Response
Lambda Function ARN: [ARN of function from Lambda#Edge Function]
Further Reading
Lambda#Edge
Lambda#Edge Example functions
Edit 1
Turns out S3 Origins, as Sanjay pointed out are limited to a select set of headers for caching.
My suggestion for this would be to change from an S3 Origin, to a Custom Origin, using S3 Static Website hosting, which we can then target as a Custom Origin.
S3 Bucket Configuration
S3 Bucket:
Properties:
Static Website Hosting: Use this bucket to host a website
Note the Endpoint name that you are given on this page, you will need it for the next step.
CloudFront Updates
Origins:
Create Origin:
Origin Domain Name: [Endpoint from above]
Origin ID: Custom-S3StaticHosting
Behaviours:
Default:
Origin: Custom-S3StaticHosting
Here is how I would solve it.
You don't need to perform a redirect for mobile apps. (Avoid redirect when possible) You can use the same url to serve desktop or mobile contents.
In your cloudfront whitelist, just whitelist CloudFront-Is-Mobile-Viewer header. That will cache the contents based on your device.
Implement Viewer Request Lambda Edge and add it to CloudFront.
Lambda Edge is to program pop or CloudFront before the request gets to server.
In the LambdaEdge, verify the User-Agent header and classify whether you want to serve mobile or desktop contents. If mobile, then you can change the origin url to serve from mobile contents, else you can change it to desktop contents or default content.
You get your http headers in the User Request LambdaEdge.
Lambda Edge Documentation:
http://docs.aws.amazon.com/lambda/latest/dg/lambda-edge.html
Sample node implementation is available on the reference page.
If you really want to perform a redirect, you can do that with viewer response and make decision-based on the device header received.
A sample implementation of viewer response is covered in this blog,
https://aws.amazon.com/blogs/aws/lambdaedge-intelligent-processing-of-http-requests-at-the-edge/
The above implementation just spits back all the headers that it received, instead of sending 200 OK, the code need to be modified for 3xx status with the redirect location.
Hope it helps.
Currently at viewer response we don't have access to CloudFront-Is-X-Viewer headers even after setting them as whitelisted and moreover status header is read-only. I've solved it by triggering at origin request:
exports.handler = (event, context, callback) => {
const name = 'cloudfront-is-mobile-viewer';
const request = event.Records[0].cf.request;
const headers = request.headers;
if (headers[name] && headers[name][0].value == "true") {
callback(null, {
status:'302',
statusDescription: 'Found',
headers: {
location: [{
key: 'Location',
value: `http://m.example.com${request.uri}`,
}]
}
})
}
callback(null, request);
};
How to add decoder with this code for nodejs, i use this code but its give me a string of character(How to decode it URi string)
Sorry for i add it to answer because i have low reputation
exports.handler = (event, context, callback) => {
const name = 'cloudfront-is-mobile-viewer';
const request = event.Records[0].cf.request;
const headers = request.headers;
if (headers[name] && headers[name][0].value == "true") {
callback(null, {
status:'302',
statusDescription: 'Found',
headers: {
location: [{
key: 'Location',
value: `http://m.example.com${request.uri}`,
}]
}
})
}
callback(null, request);
};