I have containerized my project that uploads files to S3.
Everything was working fine when I was uploading the files from my local file system.
I just mounted my container to my local file system, and then uploading stopped.
The following is the piece of function for uploading the files to the S3 bucket:
// AWS configuration
AWS.config.update({ region: 'ap-northeast-1' });
let s3 = new AWS.S3({ apiVersion: '2006-03-01' });
.
.
.
function s3uploader(uploadingVideo) {
// call S3 to retrieve upload file to specified bucket
let uploadParams = { Bucket: "my-bucket", Key: '', Body: '' };
let file = uploadingVideo;
console.log(file);
// Configure the file stream and obtain the upload parameters
let fileStream = fs.createReadStream(file);
fileStream.on('error', function (err) {
console.log('File Error', err);
});
uploadParams.Body = fileStream;
uploadParams.Key = path.basename(file);
// call S3 to retrieve upload file to specified bucket
s3.upload(uploadParams, function (err, data) {
console.log("Hello World!")
if (err) {
console.log("Error", err);
} if (data) {
console.log("Upload Success", data.Location);
}
});
}
At the moment, nothing happens when running the container. No error, not even the "Hello World!" part, so I think that s3 is not being called at the first place.
I have found a similar question here, but it wasn't helpful to my case.
I also thought of maybe installing the aws cli from dockerfile but also didn't succeed with that.
What is exactly missing here, and how to fix it?
Related
This is the current code I am using to upload a single file from local folder to s3 bucket.
try {
var filePath = "./data/1061827.png";
var s3 = new AWS.S3();
var params = {
Bucket: "**************",
Key: "folder/" + Date.now() + "_" + path.basename(filePath),
Body: fs.createReadStream(filePath),
};
s3.upload(params, function (err, data) {
if (err) {
console.log("Error", err);
}
//success
if (data) {
console.log("Uploaded in:", data.Location);
return data;
}
});
} catch (error) {
console.log(error);
return error;
}
Currently onle one file from folder data is geting uploaded but what i want is to just give the folder address(in this case data) and all files from it get uploaded into S3 Bucket.
There is no API call to upload multiple files to Amazon S3.
You would need to code a loop in your program and upload one file per API call.
Our app allow our clients large file uploads. Files are stored on AWS/S3 and we use Uppy for the upload, and dockerize it to be used under a kubernetes deployment where we can up the number of instances.
It works well, but we noticed all > 5GB uploads fail. I know uppy has a plugin for AWS multipart uploads, but even when installed during the container image creation, the result is the same.
Here's our Dockerfile. Has someone ever succeeded in uploading > 5GB files to S3 via uppy? IS there anything we're missing?
FROM node:alpine AS companion
RUN yarn global add #uppy/companion#3.0.1
RUN yarn global add #uppy/aws-s3-multipart
ARG UPPY_COMPANION_DOMAIN=[...redacted..]
ARG UPPY_AWS_BUCKET=[...redacted..]
ENV COMPANION_SECRET=[...redacted..]
ENV COMPANION_PREAUTH_SECRET=[...redacted..]
ENV COMPANION_DOMAIN=${UPPY_COMPANION_DOMAIN}
ENV COMPANION_PROTOCOL="https"
ENV COMPANION_DATADIR="COMPANION_DATA"
# ENV COMPANION_HIDE_WELCOME="true"
# ENV COMPANION_HIDE_METRICS="true"
ENV COMPANION_CLIENT_ORIGINS=[...redacted..]
ENV COMPANION_AWS_KEY=[...redacted..]
ENV COMPANION_AWS_SECRET=[...redacted..]
ENV COMPANION_AWS_BUCKET=${UPPY_AWS_BUCKET}
ENV COMPANION_AWS_REGION="us-east-2"
ENV COMPANION_AWS_USE_ACCELERATE_ENDPOINT="true"
ENV COMPANION_AWS_EXPIRES="3600"
ENV COMPANION_AWS_ACL="public-read"
# We don't need to store data for just S3 uploads, but Uppy throws unless this dir exists.
RUN mkdir COMPANION_DATA
CMD ["companion"]
EXPOSE 3020
EDIT:
I made sure I had:
uppy.use(AwsS3Multipart, {
limit: 5,
companionUrl: '<our uppy url',
})
And it still doesn't work- I see all the chunks of the 9GB file sent on the network tab but as soon as it hits 100% -- uppy throws an error "cannot post" (to our S3 url) and that's it. failure.
Has anyone ever encountered this? upload goes fine till 100%, then the last chunk gets HTTP error 413, making the entire upload fail.
Thanks!
Here I'm adding some code samples from my repository that will help you to understand the flow of using the BUSBOY package to stream the data to the S3 bucket. Also, I'm adding the reference links here for you to get the package details I'm using.
https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/clients/client-s3/index.html
https://www.npmjs.com/package/busboy
export const uploadStreamFile = async (req: Request, res: Response) => {
const busboy = new Busboy({ headers: req.headers });
const streamResponse = await busboyStream(busboy, req);
const uploadResponse = await s3FileUpload(streamResponse.data.buffer);
return res.send(uploadResponse);
};
const busboyStream = async (busboy: any, req: Request): Promise<any> {
return new Promise((resolve, reject) => {
try {
const fileData: any[] = [];
let fileBuffer: Buffer;
busboy.on('file', async (fieldName: any, file: any, fileName: any, encoding: any, mimetype: any) => {
// ! File is missing in the request
if (!fileName)
reject("File not found!");
let totalBytes: number = 0;
file.on('data', (chunk: any) => {
fileData.push(chunk);
// ! given code is only for logging purpose
// TODO will remove once project is live
totalBytes += chunk.length;
console.log('File [' + fieldName + '] got ' + chunk.length + ' bytes');
});
file.on('error', (err: any) => {
reject(err);
});
file.on('end', () => {
fileBuffer = Buffer.concat(fileData);
});
});
// ? Haa, finally file parsing wen't well
busboy.on('finish', () => {
const responseData: ResponseDto = {
status: true, message: "File parsing done", data: {
buffer: fileBuffer,
metaData
}
};
resolve(responseData)
console.log('Done parsing data! -> File uploaded');
});
req.pipe(busboy);
} catch (error) {
reject(error);
}
});
}
const s3FileUpload = async (fileData: any): Promise<ResponseDto> {
try {
const params: any = {
Bucket: <BUCKET_NAME>,
Key: <path>,
Body: fileData,
ContentType: <content_type>,
ServerSideEncryption: "AES256",
};
const command = new PutObjectCommand(params);
const uploadResponse: any = await this.S3.send(command);
return { status: true, message: "File uploaded successfully", data: uploadResponse };
} catch (error) {
const responseData = { status: false, message: "Monitor connection failed, please contact tech support!", error: error.message };
return responseData;
}
}
In the AWS S3 service in a single PUT operation, you can upload a single object up to 5 GB in size.
To upload > 5GB files to S3 you need to use the multipart upload S3 API, and also the AwsS3Multipart Uppy API.
Check your upload code to understand if you are using AWSS3Multipart correctly, setting the limit properly for example, in this case a limit between 5 and 15 is recommended.
import AwsS3Multipart from '#uppy/aws-s3-multipart'
uppy.use(AwsS3Multipart, {
limit: 5,
companionUrl: 'https://uppy-companion.myapp.net/',
})
Also, check this issue on Github Uploading a large >5GB file to S3 errors out #1945
If you're getting Error: request entity too large in your Companion server logs I fixed this in my Companion express server by increasing the body-parser limit:
app.use(bodyparser.json({ limit: '21GB', type: 'application/json' }))
This is a good working example of Uppy S3 MultiPart uploads (without this limit increased): https://github.com/jhanitesh10/uppy
I'm able to upload files up to a (self-imposed) limit of 20GB using this code.
I am trying to upload an image to S3 through graphql using the apollo-upload-client library which just give the ability to send images through a graphql query.
So the image is storying itself in the S3 bucket, but when I try to read the Location url it doesn't seems to work. When I read the url with an <img src="img_url" /> it just shows:
And when I try to manually enter the link, it just automatically downloads a strange text file with a lot of weird symbols.
This is what the upload looks like:
export async function uploadImageResolver(
_parent,
{ file }: MutationUploadImageArgs,
context: Context,
): Promise<string> {
// identify(context);
const { createReadStream, filename, mimetype } = await file;
const response = await s3
.upload({
ACL: 'public-read',
Bucket: environment.bucketName,
Body: createReadStream(),
Key: uuid(),
ContentType: mimetype,
})
.promise();
return response.Location;
}
An example of the File object looks like this:
{
filename: 'Screenshot 2021-06-15 at 13.18.10.png',
mimetype: 'image/png',
encoding: '7bit',
createReadStream: [Function: createReadStream]
}
What I am doing wrong? It returns an actual S3 link but the link itself isn't displaying any image. And I tried to upload the same image to S3 manually and it works just fine. Thanks in advance for any advice!
So after a deeper research, it seems that the problem is with the serverless framework, specially with serverless-offline. It doesn't allow transport of binary data.
So I tried to convert the createReadStream to a base64 string, but that didn't work either.
This is the try:
export async function uploadImageResolver(
_parent,
{ file }: MutationUploadImageArgs,
context: Context,
): Promise<string> {
const { createReadStream, filename, mimetype } = await file;
const response = await s3
.upload({
ACL: 'public-read',
Bucket: environment.bucketName,
Body: (await stream2buffer(createReadStream())).toString('base64'),
Key: `${uuid()}${extname(filename)}`,
ContentEncoding: 'base64',
ContentType: mimetype // image/jpg, image/png, ...
})
.promise();
return response.Location;
}
async function stream2buffer(stream: Stream): Promise<Buffer> {
return new Promise<Buffer>((resolve, reject) => {
let _buf = Array<any>();
stream.on('data', (chunk) => _buf.push(chunk));
stream.on('end', () => resolve(Buffer.concat(_buf)));
stream.on('error', (err) => reject(`error converting stream - ${err}`));
});
}
I also tried to install the serverless-apigw-binary plugin, that that didn't work either.
plugins:
- serverless-webpack
- serverless-offline
- serverless-apigw-binary
It is uploading the same corrupted image to s3.
These are some posts with the same problem, but none of them the got a solution.
https://stackoverflow.com/questions/61050997/file-uploaded-successfully-to-s3-using-serverless-api-but-it-doesnt-opencorrup
Uploading image to s3 from AWS Lambda with NodeJS results in corrupted file
UPDATE: So it is definitely not a problem with my s3.upload function or the s3 itself. It seems that the issue is getting the image to the server. I am pretty sure that is has something to do with serverless.
I've created a small function that just receives the image and load it into a local folder. And I am getting the image corrupted:
export async function uploadImageResolver(
_parent,
{ file }: MutationUploadImageArgs,
context: Context,
): Promise<string> {
// identify(context);
const { createReadStream, filename } = await file;
createReadStream().pipe(
createWriteStream(__dirname + `/../../../images/${filename}`),
);
return ''
}
UPDATE 2: I figured out that it works when deploying. So it has to be something with serverless-offline.
I am trying to return a YAML CloudFormation template file to CodePipeline so it can be passed to CloudFormation. The file is uploaded to S3, and can be accessed and read by Lambda just fine. The issue I have is turning this file into an output artifact.
I am currently able to output the raw text as an 'Output variable', which I've confirmed from the CodePipeline Execution History in the AWS Console. In this same page there is also an artifact:
However this link takes me to an error in S3, leading me to believe the artifact is never actually populated in any way.
This the Lambda function I have at the moment:
var AWS = require('aws-sdk')
var s3 = new AWS.S3();
var codepipeline = new AWS.CodePipeline();
exports.handler = (events, context) => {
const params = {
Bucket: 'myBucket',
Key: 'myFile.yml'
};
s3.getObject(params, function(err, data) {
if (err) {
// Handling
} else {
var yamlString = data.Body.toString('utf-8');
var params = {
jobId: jobId,
outputVariables: {
s3Artifact: file
}
};
codepipeline.putJobSuccessResult(params, function(err, data) {
if(err) {
context.fail(err);
} else {
context.succeed(message);
}
});
return;
}
});
}
What confuses me is that CodePipeline clearly allows me to specify an output artifact for this Lambda function, yet I cannot work out how to do so.
Any help is much appreciated.
Ultimately it seems I was going about this the wrong way. I changed the pipeline to have only an S3 source action and have it be triggered by a custom CloudWatch event for when a new image is pushed to ECR. I then manually disabled the S3 event created by CodePipeline so that only an ECR push triggers a deployment.
See more here.
i'm trying to read data from AWS S3 Bucket using JavaScript but getting
error :
"Error: Missing credentials in config"
AWS.config.update({
"region": "eu-west-1"
});
var params = { Bucket: <BucketName>, Key: "data.json" };
new AWS.S3().getObject(params, function (err, json_data) {
if (!err) {
var json = JSON.parse(new Buffer(json_data.Body).toString("utf8"));
console.log(json);
}
else
console.log(err);
});
even if i tried without AWS.config.update i'm getting this error.
any idea?
AFAIK if you wish to access a bucket which is not public then you will need to supply your AWS credentials along with the request. Here's the SDK page for building an AWS.credentials object that you put into your options when making the AWS.S3 object.
https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/Credentials.html
No example because I am not a JS dev and can't write it out of memory, sorry!