I am new to Serverless and AWS also. So My requirement is like. I need to download a file from S3. I have tried in many ways. I read many articles but could not find a proper document for this purpose.
So What I did was, I created a boilerplate using Serverless and got all the files like handler.js, serverless.yml
I need to know the correct steps to download a file from S3.
What I tried is.
Handler.js
const AWS = require('aws-sdk');
const S3= new AWS.S3();
exports.hello = async (event, context) => {
console.log(`Hi from Node.js ${process.version} on Lambda!`)
S3.getObject({Bucket: '*******', Key: '******'}).promise().then( data =>{
return {
statusCode: 200,
body: data
})
}
Serveless.yml
service: node11
custom:
bucket: *********
provider:
name: aws
runtime: provided # set to provided
stage: dev
region: us-east-1
iamRoleStatements:
- Effect: Allow
Action:
- s3:*
Resource: "arn:aws:s3:::${self:custom.bucket}/*"
functions:
hello:
handler: handler.hello
events:
- http:
path: /
method: get
layers: # add layer
- arn:aws:lambda:us-east-1:553035198032:layer:nodejs11:3
Whatever I did, I am always getting an error like INTERNAL SERVER ERROR.
What is the proper way to get the file from S3?
Try this:
const AWS = require('aws-sdk');
const S3= new AWS.S3();
exports.hello = async (event, context) => {
try {
console.log(`Hi from Node.js ${process.version} on Lambda!`);
// Converted it to async/await syntax just to simplify.
const data = await S3.getObject({Bucket: '*******', Key: '******'}).promise();
return {
statusCode: 200,
body: JSON.stringify(data)
}
}
catch (err) {
return {
statusCode: err.statusCode || 400,
body: err.message || JSON.stringify(err.message)
}
}
}
and ensure that in serverless.yml you've set runtime: nodejs8.10 under provider.
Lambda response body must be a string as defined here.
Related
I'm trying to receive webhooks from Github using a probot application, but every single time I try this, I get a {"message":"Service Unavailable"} error.
Github sends this payload to an AWS Lambda function, and by googling (i think) this is an issue with not having enough nodes to handle the number of requests.
Either something is wrong with my code or there is an error with the configuration.
I'm using the serverless framework to upload to AWS lambda.
Here is the part where the code fails (no error messages in logs, and the bot just quits):
const yamlFile = async (context) => {
try {
console.log("trying to get yaml")
var yamlfile = await context.octokit.repos.getContent({
owner: context.payload.repository.owner.login,
repo: context.payload.repository.name,
path: ".bit/config.yml",
});
console.log(yamlfile)
} catch (e) {
console.log("Error with getting content of yaml");
console.log(e)
return null
}
console.log("got yaml, but no content yet");
yamlfile = Buffer.from(yamlfile.data.content, 'base64').toString()
console.log(yamlfile)
try {
let fileContents = yamlfile
configyml = yaml.load(fileContents);
} catch (e) {
const issueBody = context.issue({
title: "[ERROR] Please read",
body: `There was an issue parsing the config file of this course. Please contact your counselor and send them the below error.\n${e}`,
});
context.octokit.issues.create(issueBody)
console.log("ERROR: " + e);
return null
}
console.log("returining configyml")
return configyml;
}
The function yamlFile() is being called in our main function here:
let currentStep = ""
let configData = await data.yamlFile(context);
console.log(configData)
if (configData == null) {
console.log("null config data");
return
}
AWS Config
Timeout: 60 seconds
serverless.yml for Serverless framework:
service: <SERVICE NAME>
# app and org for use with dashboard.serverless.com
app: <APP NAME>
org: <ORG NAME>
frameworkVersion: "2"
useDotenv: true
provider:
name: aws
runtime: nodejs12.x
lambdaHashingVersion: 20201221
environment:
APP_ID: ${param:APP_ID}
PRIVATE_KEY: ${param:PRIVATE_KEY}
WEBHOOK_SECRET: ${param:WEBHOOK_SECRET}
NODE_ENV: production
LOG_LEVEL: debug
memorySize: 2048
functions:
webhooks:
handler: handler.webhooks
memorySize: 2048
events:
- httpApi:
path: /api/github/webhooks
method: post
I am trying to run a newly created lambda function using SAM template with run time node.js locally.
I have the following :
a) aws account with region, accessKeyId & secretAccessKey
b) aws-cli, aws-sam, docker
Running lambda function locally using sam local invoke is fine, but the problem is when i used to connect to dynamodb in my function, getting the below error.
{"message":"User: arn:aws:iam::*********:user/test.user is not authorized to perform: dynamodb:Query on resource: arn:aws:dynamodb:us-west-2:********:table/my_table with an explicit deny","code":"AccessDeniedException","time":"2020-08-30T07:45:51.678Z","requestId":"7SIBTHTKDSSDJNSTLNUSSBHDNDHBSMVJF66Q9ASUAAJG","statusCode":400,"retryable":false,"retryDelay":40.84164538820391}
I have access to aws account and has accessKeyId & secretAccessKey and able to query when trying from aws console. In vscode editor, installed aws-toolkit and added the credentials, but still i am getting the same error in local.
Can somebody help me with this issue as i am going through a difficult situation in finding the solution.
Here is my code snippet.
let response;
const AWS = require('aws-sdk');
AWS.config.update({
region: "us-west-2",
accessKeyId: '************',
secretAccessKey: '********************'
});
const dbClient = new AWS.DynamoDB.DocumentClient();
exports.lambdaHandler = async (event, context) => {
try {
let myTable = await getData(event.myId);
response = {
'statusCode': 200,
'body': JSON.stringify({
message: myTable
})
}
} catch (err) {
console.log(err);
return err;
}
return response
};
const MY_TABLE_NAME = "my_table";
const getData = async (myId) => {
const params = {
TableName: MY_TABLE_NAME ,
KeyConditionExpression: "#uid = :id",
ExpressionAttributeValues: {
':id': myId
},
ExpressionAttributeNames: {
'#uid': 'userID'
}
};
let { Count, Items } = await dbClient.query(params).promise();
if (Items.length == 0) {
return false;
} else {
var obj = {
Name: (Count > 0) ? Items[0].name : null,
MY_TABLE: MY_TABLE
};
return obj;
}
};
template.yaml
Resources:
HelloWorldFunction:
Type: AWS::Serverless::Function # More info
Properties:
CodeUri: hello-world/
Handler: app.lambdaHandler
Runtime: nodejs12.x
Policies: AmazonDynamoDBFullAccess
Events:
HelloWorld:
Type: Api # More info
Properties:
Path: /hello
Method: get
Any help would be really appreciated. Thanks in advance.
The reason why i am getting user not authorized error was beacuase i donot have programmatic access, and only authorized to access aws console access.
Thanks
I have set up a simple serverless rest api using Node JS and AWS Lambda.
The deployment is done using AWS SAM
Below is the SAM Template :
AWSTemplateFormatVersion: '2010-09-09'
Transform: 'AWS::Serverless-2016-10-31'
Description: Serverless API With SAM
Resources:
createUser:
Type: AWS::Serverless::Function
Properties:
Handler: handler.create
MemorySize: 128
Runtime: nodejs12.x
Timeout: 3
Events:
createUserApi:
Type: Api
Properties :
Path : /users
Method : post
listUsers:
Type: AWS::Serverless::Function
Properties:
Handler: handler.list
MemorySize: 128
Runtime: nodejs12.x
Timeout: 3
Events:
listUserApi:
Type: Api
Properties :
Path : /users
Method : get
This works fine but below is my observation regarding stacks created.
It creates two AWS Lambda functions instead of one.
Both contain two APIs listed as -
createUser
listUsers
Can it not contain only on Lambda function with these two handlers inside ?
handler.js file :
const connectedToDb = require('./src/db/db.js');
const User = require('./src/model/user.js');
module.exports.create = async (event,context) =>{
console.log('create function called !!');
try{
console.log('before connecting to DB ');
await connectedToDb();
console.log('after connecting to DB ');
const usr = new User(JSON.parse(event.body));
console.log('saving a user now with ',event.body);
await usr.save();
return{
statusCode : 200,
body:JSON.stringify(usr)
}
}catch(err){
return{
statusCode : err.statusCode || 500,
body : 'Cannot Create Order'
}
}
}
module.exports.list = async (event,context) => {
console.log('listing all users');
try{
await connectedToDb();
const users = await User.find({});
console.log('users are == ',users);
return {
statusCode:200,
body:JSON.stringify(users)
}
}catch(err){
return {
statusCode:err || 500,
body:JSON.stringify(err)
}
}
}
I use serverless Lambda services to transcribe from speech to text with Amazon Transcribe. My current scripts are able to transcribe file from S3 and store the result as a JSON file also in S3.
Is there a possibility to get the result directly, because I want to store it in a database (PostgreSQL in AWS RDS)?
Thank you for your hints
serverless.yml
...
provider:
name: aws
runtime: nodejs10.x
region: eu-central-1
memorySize: 128
timeout: 30
environment:
S3_AUDIO_BUCKET: ${self:service}-${opt:stage, self:provider.stage}-records
S3_TRANSCRIPTION_BUCKET: ${self:service}-${opt:stage, self:provider.stage}-transcriptions
LANGUAGE_CODE: de-DE
iamRoleStatements:
- Effect: Allow
Action:
- s3:PutObject
- s3:GetObject
Resource:
- 'arn:aws:s3:::${self:provider.environment.S3_AUDIO_BUCKET}/*'
- 'arn:aws:s3:::${self:provider.environment.S3_TRANSCRIPTION_BUCKET}/*'
- Effect: Allow
Action:
- transcribe:StartTranscriptionJob
Resource: '*'
functions:
transcribe:
handler: handler.transcribe
events:
- s3:
bucket: ${self:provider.environment.S3_AUDIO_BUCKET}
event: s3:ObjectCreated:*
createTextinput:
handler: handler.createTextinput
events:
- http:
path: textinputs
method: post
cors: true
...
resources:
Resources:
S3TranscriptionBucket:
Type: 'AWS::S3::Bucket'
Properties:
BucketName: ${self:provider.environment.S3_TRANSCRIPTION_BUCKET}
...
handler.js
const db = require('./db_connect');
const awsSdk = require('aws-sdk');
const transcribeService = new awsSdk.TranscribeService();
module.exports.transcribe = (event, context, callback) => {
const records = event.Records;
const transcribingPromises = records.map((record) => {
const recordUrl = [
'https://s3.amazonaws.com',
process.env.S3_AUDIO_BUCKET,
record.s3.object.key,
].join('/');
// create random filename to avoid conflicts in amazon transcribe jobs
function makeid(length) {
var result = '';
var characters = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789';
var charactersLength = characters.length;
for ( var i = 0; i < length; i++ ) {
result += characters.charAt(Math.floor(Math.random() * charactersLength));
}
return result;
}
const TranscriptionJobName = makeid(7);
return transcribeService.startTranscriptionJob({
LanguageCode: process.env.LANGUAGE_CODE,
Media: { MediaFileUri: recordUrl },
MediaFormat: 'wav',
TranscriptionJobName,
//MediaSampleRateHertz: 8000, // normally 8000 if you are using wav file
OutputBucketName: process.env.S3_TRANSCRIPTION_BUCKET,
}).promise();
});
Promise.all(transcribingPromises)
.then(() => {
callback(null, { message: 'Start transcription job successfully' });
})
.catch(err => callback(err, { message: 'Error start transcription job' }));
};
module.exports.createTextinput = (event, context, callback) => {
context.callbackWaitsForEmptyEventLoop = false;
const data = JSON.parse(event.body);
db.insert('textinputs', data)
.then(res => {
callback(null,{
statusCode: 200,
body: "Textinput Created! id: " + res
})
})
.catch(e => {
callback(null,{
statusCode: e.statusCode || 500,
body: "Could not create a Textinput " + e
})
})
};
I think your best option is to trigger a lambda from the s3 event when a transcription is stored, and then post the data to your database. As Dunedan mentioned, you can't go directly from transcribe to a DB.
You can add the event to a lambda via serverless like so:
storeTranscriptonInDB:
handler: index.storeTransciptInDB
events:
- s3:
bucket: ${self:provider.environment.S3_TRANSCRIPTION_BUCKET}
rules:
- suffix: .json
The s3 key for the transcript file will be event.Records[#].s3.object.key
I would loop through the records to be thorough, and for each do something like this:
const storeTransciptInDB = async (event, context, callback) => {
const records = event.Records;
for (record of event.Records) {
let key = record.s3.object.key;
let params = {
Bucket: record.s3.bucket.name,
Key: key
}
let transcriptFile = await s3.getObject(params).promise();
let transcriptObject = JSON.parse(data.Body.toString("utf-8"));
let transcriptResults = transcriptObject.results.transcripts;
let transcript = "";
transcriptResults.forEach(result => (transcript += result.transcript + " "));
// at this point you can post the transcript variable to your database
}
}
Amazon Transcribe currently only supports storing transcriptions in S3, as explained in the API definition for StartTranscriptionJob. There is one special case though: If you don't want to manage your own S3 bucket for transcriptions, you can just leave out the OutputBucketName and the transcription will be stored in an AWS-managed S3 bucket. In that case you'd get a pre-signed URL allowing you to download the transcription.
As transcribing happens asynchronously I suggest you create a second AWS Lambda function, triggered by a CloudWatch Event which gets emitted when the state of your transcription changes (as explained in Using Amazon CloudWatch Events with Amazon Transcribe) or by a S3 notification (Using AWS Lambda with Amazon S3). This AWS Lambda function can then fetch the finished transcription from S3 and store its contents in PostgreSQL.
I want some of my Lambda resources to push to an AWS IOT endpoint using aws-sdk's AWS.IotData({ endpoint: url }) function - where endpoint is a required parameter.
Right now, I am passing the endpoint URL via an environment variable to my Lambda. However, when put into a SAM/CF template, I can't find a way to retrieve my IOT endpoint URL, so that I could simply !Ref it.
Browsing through the AWS resource type reference I did not find any resource that corresponds to an IOT endpoint.
It seems like IOT endpoint can only be provisioned manually, via AWS Console (enabled / disabled), as on the screenshot below:
Any advice on how to have control over provisioning an IOT endpoint or at least reading the IOT URL from within a SAM/CF template, without scripting this with aws-cli?
For anyone interested in the solution with CloudFormation Custom Resource, I wrote a simple Lambda and a CF template that provides an IOT endpoint address to other CF stacks.
template.yaml
AWSTemplateFormatVersion: '2010-09-09'
Transform: 'AWS::Serverless-2016-10-31'
Resources:
IotEndpointProvider:
Type: 'AWS::Serverless::Function'
Properties:
FunctionName: IotEndpointProvider
Handler: iotEndpointProvider.handler
Runtime: nodejs6.10
CodeUri: .
MemorySize: 128
Timeout: 3
Policies:
- Version: '2012-10-17'
Statement:
- Effect: Allow
Action:
- iot:DescribeEndpoint
Resource:
- '*'
IotEndpoint:
Type: 'Custom::IotEndpoint'
Properties:
ServiceToken: !GetAtt IotEndpointProvider.Arn
Outputs:
IotEndpointAddress:
Value: !GetAtt IotEndpoint.IotEndpointAddress
Export:
Name: IotEndpointAddress
iotEndpointProvider.js
var aws = require("aws-sdk");
exports.handler = function(event, context) {
console.log("REQUEST RECEIVED:\n" + JSON.stringify(event));
// For Delete requests, immediately send a SUCCESS response.
if (event.RequestType == "Delete") {
sendResponse(event, context, "SUCCESS");
return;
}
const iot = new aws.Iot();
iot.describeEndpoint({}, (err, data) => {
let responseData, responseStatus;
if (err) {
responseStatus = "FAILED";
responseData = { Error: "describeEndpoint call failed" };
console.log(responseData.Error + ":\n", err);
} else {
responseStatus = "SUCCESS";
responseData = { IotEndpointAddress: data.endpointAddress };
console.log('response data: ' + JSON.stringify(responseData));
}
sendResponse(event, context, responseStatus, responseData);
});
};
// Send response to the pre-signed S3 URL
function sendResponse(event, context, responseStatus, responseData) {
var responseBody = JSON.stringify({
Status: responseStatus,
Reason: "See the details in CloudWatch Log Stream: " + context.logStreamName,
PhysicalResourceId: context.logStreamName,
StackId: event.StackId,
RequestId: event.RequestId,
LogicalResourceId: event.LogicalResourceId,
Data: responseData
});
console.log("RESPONSE BODY:\n", responseBody);
var https = require("https");
var url = require("url");
var parsedUrl = url.parse(event.ResponseURL);
var options = {
hostname: parsedUrl.hostname,
port: 443,
path: parsedUrl.path,
method: "PUT",
headers: {
"content-type": "",
"content-length": responseBody.length
}
};
console.log("SENDING RESPONSE...\n");
var request = https.request(options, function(response) {
console.log("STATUS: " + response.statusCode);
console.log("HEADERS: " + JSON.stringify(response.headers));
// Tell AWS Lambda that the function execution is done
context.done();
});
request.on("error", function(error) {
console.log("sendResponse Error:" + error);
// Tell AWS Lambda that the function execution is done
context.done();
});
// write data to request body
request.write(responseBody);
request.end();
}
I'm afraid you cannot provision an IoT endpoint, as the only API call that is related to an IoT endpoint is DescribeEndpoint.
What you can do is create a Lambda-backed CloudFormation Custom Resource. The Lambda function will execute the DescribeEndpoint call (using the AWS SDK of your choice depending on the Lambda's runtime) and return the endpoint's URL so your other CloudFormation resources can consume it.
Here's a good example on Lambda-backed Custom Resources: http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/template-custom-resources-lambda.html.