I came across a strange issue while using aws-sdk package in my app in android. We are using this in order to sign and upload pictures to S3. In iOS everything is working as expected but on android it return with this error
[MissingRequiredParameter: Missing required key 'Bucket' in params]
This is our setup
export const s3bucket = new S3({
accessKeyId: S3_ACCESS_KEY,
secretAccessKey: S3_SECRET_KEY,
Bucket: 'assets',
signatureVersion: 'v4',
});
Here we call the function
s3bucket.createBucket(e => {
// With this setup, each time the user uploads an image, will be overwritten
// To prevent this, use a different Key each time.
// This won't be needed if they're uploading their avatar, hence the filename, userAvatar.js.
const params = {
Bucket: 'assets',
Key: file.name, // type is not required
Body: base64Data,
ACL: 'public-read',
ContentEncoding: 'base64', // required
ContentType: contentType, // required. Notice the back ticks
};
s3bucket.upload(params, (err, data) => {
console.log(err, data);
if (err) {
alert('We are sorry there seem to be a problem. Please try again!');
}
if (currentIndex === -1) {
const newDocument = {
path: data.Location,
page: allDocumentPages.length + 1,
status: '',
id_type: allDocumentPages[0].id_type,
};
setAllDocumentPages([...allDocumentPages, newDocument]);
} else {
const documentToUpdate = allDocumentPages[currentIndex];
documentToUpdate.path = data.Location;
documentToUpdate.status = '';
}
setLoadingIndex(null);
});
});
Is there anything I miss?
I believe the missing 'Bucket' parameter is coming from the createBucket method, where its params object was not defined, as seen from the S3 documentation. You should re-define it as such.
export const s3bucket = new S3({
accessKeyId: S3_ACCESS_KEY,
secretAccessKey: S3_SECRET_KEY,
// Bucket: 'assets', <-- this will be ignored while defining the S3 options.
signatureVersion: 'v4',
});
// added a `params1` variable since `params` has been taken up.
var params1 = {
Bucket: 'assets'
};
s3bucket.createBucket(params1, e => {
...
};
Related
new to AWS and just not sure how to define the relevant authenitcation to get my lambda function to be able to call my graphQL endpoint for a post req. Assuming I need to put an API key somewhere in this function but just am a bit lost. Any help at all would be great. Have put the function below - created it using the amplify cli and the generategraphqlpermissions flag is set to true if thats any help narrowing it down.
import crypto from '#aws-crypto/sha256-js';
import { defaultProvider } from '#aws-sdk/credential-provider-node';
import { SignatureV4 } from '#aws-sdk/signature-v4';
import { HttpRequest } from '#aws-sdk/protocol-http';
import { default as fetch, Request } from 'node-fetch';
const GRAPHQL_ENDPOINT = <myEndpoint>;
const AWS_REGION = process.env.AWS_REGION || 'us-east-1';
const { Sha256 } = crypto;
const query = /* GraphQL */ `mutation CreateCalendarEvent($input: CreateCalendarEventInput!, $condition: ModelCalendarEventConditionInput) {
createCalendarEvent(input: $input, condition: $condition) {
__typename
id
start
end
title
actions
allDay
resizable
draggable
colour
createdAt
updatedAt
}
}`;
/**
* #type {import('#types/aws-lambda').APIGatewayProxyHandler}
*/
export const handler = async (event) => {
console.log(`EVENT: ${JSON.stringify(event)}`);
console.log(GRAPHQL_ENDPOINT);
const endpoint = new URL(GRAPHQL_ENDPOINT);
const signer = new SignatureV4({
credentials: defaultProvider(),
region: AWS_REGION,
service: 'appsync',
sha256: Sha256
});
const requestToBeSigned = new HttpRequest({
method: 'POST',
headers: {
'Content-Type': 'application/json',
host: endpoint.host
},
hostname: endpoint.host,
body: JSON.stringify({ query }),
path: endpoint.pathname
});
const signed = await signer.sign(requestToBeSigned);
const request = new Request(endpoint, signed);
let statusCode = 200;
let body;
let response;
try {
response = await fetch(request);
body = await response.json();
if (body.errors) statusCode = 400;
} catch (error) {
statusCode = 500;
body = {
errors: [
{
message: error.message
}
]
};
}
return {
statusCode,
// Uncomment below to enable CORS requests
// headers: {
// "Access-Control-Allow-Origin": "*",
// "Access-Control-Allow-Headers": "*"
// },
body: JSON.stringify(body)
};
};
WHen invoking an AWS Service from Lambda, you do not need the keys. Instead, you can give the IAM role that the Lambda function runs under the permissions to invoke that service. In your case, give the role permission to invoke app sync.
More information can be found here:
https://docs.aws.amazon.com/lambda/latest/dg/lambda-permissions.html
I'd like a user to be able to upload either JPG or PNG image to an S3 bucket.
I am using a Lambda function which allows me to only presign .jpg images for S3 and it works great for just one file type. How do I add an additional file type to presign, for example, .png images too. Do I really need to write a new Lambda where I just change the .jpg to .png or I can do it somehow in my existing code below?
const AWS = require('aws-sdk')
AWS.config.update({ region: process.env.REGION })
const s3 = new AWS.S3();
const uploadBucket = 'xxx-bucket'
exports.handler = async (event) => {
const result = await getUploadURL()
console.log('Result: ', result)
return result
};
const getUploadURL = async function() {
console.log('getUploadURL started')
let actionId = Date.now()
var s3Params = {
Bucket: uploadBucket,
Key: `${actionId}.jpg`,
ContentType: 'image/jpeg',
CacheControl: 'max-age=31104000',
ACL: 'public-read',
};
return new Promise((resolve, reject) => {
// Get signed URL
let uploadURL = s3.getSignedUrl('putObject', s3Params)
resolve({
"statusCode": 200,
"isBase64Encoded": false,
"headers": {
"Access-Control-Allow-Origin": "*"
},
"body": JSON.stringify({
"uploadURL": uploadURL,
"photoFilename": `${actionId}.jpg`
})
})
})
}
Your options are as follow :
Make a new Lambda as you suggested to handle the PNG separately
Pass parameters to your getUploadURL function through your event, something like :
exports.handler = async event => {
const { filetype } = event.body (or pathParams, query string, etc)
const result = await getUploadURL(filetype)
console.log('Result: ', result)
return result
};
const getUploadURL = async filetype => {
console.log('getUploadURL started')
let actionId = Date.now()
var s3Params = {
Bucket: uploadBucket,
Key: `${actionId}.${filetype}`,
ContentType: `image/${filetype === 'jpg'?'jpeg':'png'}`,
CacheControl: 'max-age=31104000',
ACL: 'public-read',
};
...
The call to S3.getSignedUrl() requires {Bucket: 'bucket', Key: 'key'} at a minimum for a putObject operation. So if you don't want to sacrifice the filename extension and/or the content-type attribute, those are the only options.
AWS docs
I tried to create an API for uploading & retrieving images with NestJS. Images should be stored on S3.
What I currently have:
Controller
#Post()
#UseInterceptors(FileFieldsInterceptor([
{name: 'photos', maxCount: 10},
]))
async uploadPhoto(#UploadedFiles() files): Promise<void> {
await this.s3Service.savePhotos(files.photos)
}
#Get('/:id')
#Header('content-type', 'image/jpeg')
async getPhoto(#Param() params,
#Res() res) {
const photoId = PhotoId.of(params.id)
const photoObject = await this.s3Service.getPhoto(photoId)
res.send(photoObject)
}
S3Service
async savePhotos(photos: FileUploadEntity[]): Promise<any> {
return Promise.all(photos.map(photo => {
const filePath = `${moment().format('YYYYMMDD-hhmmss')}${Math.floor(Math.random() * (1000))}.jpg`
const params = {
Body: photo.buffer,
Bucket: Constants.BUCKET_NAME,
Key: filePath,
}
return new Promise((resolve) => {
this.client.putObject(params, (err: any, data: any) => {
if (err) {
logger.error(`Photo upload failed [err=${err}]`)
ExceptionHelper.throw(ErrorCodes.SERVER_ERROR_UNCAUGHT_EXCEPTION)
}
logger.info(`Photo upload succeeded [filePath=${filePath}]`)
return resolve()
})
})
}))
}
async getPhoto(photoId: PhotoId): Promise<AWS.S3.Body> {
const object: S3.GetObjectOutput = await this.getObject(S3FileKey.of(`${Constants.S3_PHOTO_PATH}/${photoId.value}`))
.catch(() => ExceptionHelper.throw(ErrorCodes.RESOURCE_NOT_FOUND_PHOTO)) as S3.GetObjectOutput
logger.info(JSON.stringify(object.Body))
return object.Body
}
async getObject(s3FilePath: S3FileKey): Promise<S3.GetObjectOutput> {
logger.info(`Retrieving object from S3 s3FilePath=${s3FilePath.value}]`)
return this.client.getObject({
Bucket: Constants.BUCKET_NAME,
Key: s3FilePath.value
}).promise()
.catch(err => {
logger.error(`Could not retrieve object from S3 [err=${err}]`)
ExceptionHelper.throw(ErrorCodes.SERVER_ERROR_UNCAUGHT_EXCEPTION)
}) as S3.GetObjectOutput
}
The photo object actually ends up in S3, but when I download it I can't open it.
Same for the GET => can't be displayed.
What general mistake(s) I'm making here?
Not sure what values are you returning to your consumer and which values they use to get the Image again; Could you post how the actual response looks like, what is the request and verify, if the FQDN & Path match?
It seems you forgot about ACL as well, i.e. the resources you upload this way are not public-read by default.
BTW you could use aws SDK there:
import { Injectable } from '#nestjs/common'
import * as AWS from 'aws-sdk'
import { InjectConfig } from 'nestjs-config'
import { AwsConfig } from '../../config/aws.config'
import UploadedFile from '../interfaces/uploaded-file'
export const UPLOAD_WITH_ACL = 'public-read'
#Injectable()
export class ImageUploadService {
s3: AWS.S3
bucketName
cdnUrl
constructor(#InjectConfig() private readonly config) {
const awsConfig = (this.config.get('aws') || { bucket: '', secretKey: '', accessKey: '', cdnUrl: '' }) as AwsConfig // read from envs
this.bucketName = awsConfig.bucket
this.cdnUrl = awsConfig.cdnUrl
AWS.config.update({
accessKeyId: awsConfig.accessKey,
secretAccessKey: awsConfig.secretKey,
})
this.s3 = new AWS.S3()
}
upload(file: UploadedFile): Promise<string> {
return new Promise((resolve, reject) => {
const params: AWS.S3.Types.PutObjectRequest = {
Bucket: this.bucketName,
Key: `${Date.now().toString()}_${file.originalname}`,
Body: file.buffer,
ACL: UPLOAD_WITH_ACL,
}
this.s3.upload(params, (err, data: AWS.S3.ManagedUpload.SendData) => {
if (err) {
return reject(err)
}
resolve(`${this.cdnUrl}/${data.Key}`)
})
})
}
}
For anyone having the same troubles, I finally figured it out:
I enabled binary support on API Gateway (<your-gateway> Settings -> Binary Media Types -> */*) and then returned all responses from lambda base64 encoded. API Gateway will do the decode automatically before returning the response to the client.
With serverless express you can can enable the auto base64 encoding easily at the server creation:
const BINARY_MIME_TYPES = [
'application/javascript',
'application/json',
'application/octet-stream',
'application/xml',
'font/eot',
'font/opentype',
'font/otf',
'image/jpeg',
'image/png',
'image/svg+xml',
'text/comma-separated-values',
'text/css',
'text/html',
'text/javascript',
'text/plain',
'text/text',
'text/xml',
]
async function bootstrap() {
const expressServer = express()
const nestApp = await NestFactory.create(AppModule, new ExpressAdapter(expressServer))
await nestApp.init()
return serverlessExpress.createServer(expressServer, null, BINARY_MIME_TYPES)
}
In the Controller, you're now able to just return the S3 response body:
#Get('/:id')
async getPhoto(#Param() params,
#Res() res) {
const photoId = PhotoId.of(params.id)
const photoObject: S3.GetObjectOutput = await this.s3Service.getPhoto(photoId)
res
.set('Content-Type', 'image/jpeg')
.send(photoObject.Body)
}
Hope this helps somebody!
In a Lambda, I would like to sign my AppSync endpoint with aws-signature-v4 in order to use it for a mutation.
The URL generated seems to be ok but it gives me the following error when I try it:
{
"errors" : [ {
"errorType" : "InvalidSignatureException",
"message" : "The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details. etc...
} ]
}
Here is my lambda function
import { Context, Callback } from 'aws-lambda';
import { GraphQLClient } from 'graphql-request';
const v4 = require('aws-signature-v4');
export async function handle(event: any, context: Context, callback: Callback) {
context.callbackWaitsForEmptyEventLoop = false;
const url = v4.createPresignedURL(
'POST',
'xxxxxxxxxxxxxxxxx.appsync-api.eu-west-1.amazonaws.com',
'/graphql',
'appsync',
'UNSIGNED-PAYLOAD',
{
key: 'yyyyyyyyyyyyyyyyyyyy',
secret: 'zzzzzzzzzzzzzzzzzzzzz',
region: 'eu-west-1'
}
);
const mutation = `{
FAKEviewProduct(title: "Inception") {
productId
}
}`;
const client = new GraphQLClient(url, {
headers: {
'Content-Type': 'application/graphql',
action: 'GetDataSource',
version: '2017-07-25'
}
});
try {
await client.request(mutation, { productId: 'jfsjfksldjfsdkjfsl' });
} catch (err) {
console.log(err);
callback(Error());
}
callback(null, {});
}
I got my key and secret by creating a new user and Allowing him appsync:GraphQL action.
What am I doing wrong?
This is how I trigger an AppSync mutation using by making a simple HTTP-request, using axios.
const AWS = require('aws-sdk');
const axios = require('axios');
exports.handler = async (event) => {
let result.data = await updateDb(event);
return result.data;
};
function updateDb({ owner, thingName, key }){
let req = new AWS.HttpRequest('https://xxxxxxxxxxx.appsync-api.eu-central-1.amazonaws.com/graphql', 'eu-central-1');
req.method = 'POST';
req.headers.host = 'xxxxxxxxxxx.appsync-api.eu-central-1.amazonaws.com';
req.headers['Content-Type'] = 'multipart/form-data';
req.body = JSON.stringify({
"query":"mutation ($input: UpdateUsersCamsInput!) { updateUsersCams(input: $input){ latestImage uid name } }",
"variables": {
"input": {
"uid": owner,
"name": thingName,
"latestImage": key
}
}
});
let signer = new AWS.Signers.V4(req, 'appsync', true);
signer.addAuthorization(AWS.config.credentials, AWS.util.date.getDate());
return axios({
method: 'post',
url: 'https://xxxxxxxxxxx.appsync-api.eu-central-1.amazonaws.com/graphql',
data: req.body,
headers: req.headers
});
}
Make sure to give the IAM-role your Lambda function is running as, permissions for appsync:GraphQL.
Adding an answer here because I had difficulty getting the accepted answer to work and I found an issue on the AWS SDK GitHub issues that said it's not recommended to use the AWS.Signers.V4 object in production. This is how I got it to work using the popular aws4 npm module that is recommended later on in the issue linked above.
const axios = require('axios');
const aws4 = require('aws4');
const query = `
query Query {
todos {
id,
title
}
}`
const sigOptions = {
method: 'POST',
host: 'xxxxxxxxxx.appsync-api.eu-west.amazonaws.com',
region: 'eu-west-1',
path: 'graphql',
body: JSON.stringify({
query
}),
service: 'appsync'
};
const creds = {
// AWS access tokens
}
axios({
url: 'https://xxxxxxxxxx.appsync-api.eu-west/graphql',
method: 'post',
headers: aws4.sign(sigOptions, creds).headers,
data: {
query
}
}).then(res => res.data))
You don't need to construct a pre-signed URL to call an AWS AppSync endpoint. Set the authentication mode on the AppSync endpoint to AWS_IAM, grant permissions to your Lambda execution role, and then follow the steps in the "Building a JavaScript Client" tutorial to invoke a mutation or query.
I am currently trying to load images from my website to AWS S3. I have the functionality working where it uploads the image to the server but when i try to view the images they download instead of displaying. I read there is a way to set the file type so this would not happen. I am not sure how to do that. Any help would be great.
router.post('/heroes/createNewHeroes', function(req,res) {
var formidable = require('formidable'),
http = require('http'),
util = require('util');
var form = new formidable.IncomingForm();
form.parse(req, function(err, fields, files) {
console.log(fields);
console.log(files);
// Load the AWS SDK for Node.js
var AWS = require('aws-sdk');
var shortid = require('shortid');
var fs = require('fs');
var fileStream = fs.createReadStream(files.asset.path);
var newFilename = shortid.generate()+"_"+files.asset.name;
// Set your region for future requests.
AWS.config.region = 'us-west-2';
AWS.config.accessKeyId = 'access Key';
AWS.config.secretAccessKey = 'secret Key';
console.log(newFilename);
fileStream.on('error', function (err) {
if (err) { throw err; }
});
fileStream.on('open', function () {
var s3bucket = new AWS.S3({params: {Bucket: ' '}});
s3bucket.createBucket(function() {
var params = {Key: newFilename, Body: fileStream};
s3bucket.upload(params, function(err, data) {
if (err) {
console.log("Error uploading data: ", err);
} else {
console.log("Successfully uploaded data");
projectX.createHeroes(['plantTypes', 'asset', 'cost', 'energy', 'isSunProducer', 'isShooter', 'isExploding', 'sunFrequency', 'shootingFrequency', 'damage'], [fields.plantTypes, newFilename, fields.cost, fields.energy, fields.isSunProducer, fields.isShooter, fields.isExploding, fields.sunFrequency, fields.shootingFrequency, fields.damage], function(data){
res.redirect('/heroes')
});
}
});
});
});
});
});
var params = {Key: newFilename, ContentType: 'image/png', Body: fileStream};
http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#upload-property
Just put "contentType: multerS3.AUTO_CONTENT_TYPE " . It will work .
Ex:
var upload = multer({
storage: multerS3({
s3: s3,
bucket: 'some-bucket',
contentType: multerS3.AUTO_CONTENT_TYPE,
key: function (req, file, cb) {
cb(null, Date.now().toString())
}
})
})
Visit this link for more details https://github.com/badunk/multer-s3
This Helped me
storage: multerS3({
s3: s3,
bucket: "bucketname",
acl: "public-read",
contentType: multerS3.AUTO_CONTENT_TYPE,
key: function(req, file, cb) {
console.log("req.file", file);
cb(null, `${Date.now()}-${file.originalname}`);
}
})