I would like to create a new api key from lambda. I have usage plan with my Gateway API, created with CF like:
MyApi:
Type: AWS::Serverless::Api
Properties:
Auth:
UsagePlan:
UsagePlanName: MyUsagePlan
CreateUsagePlan: PER_API
...
...
Using this as a reference https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/APIGateway.html
I guess the process in the lambda should be like this:
- createApiKey
- getUsagePlan
- createUsagePlanKey
In the lambda, I have MyApi id and I'm trying to fetch the api:
var apiGateway = new AWS.APIGateway({region: region});
const restApi = await new Promise((resolve, reject) => {
apiGateway.getRestApi({restApiId: MYAPI_ID}, function(err, data) {
if (err) {
console.log('getRestApi err', err, err.stack);
reject(err);
} else {
console.log('getRestApi', data);
resolve(data);
}
});
});
But this gets timed out by my lambda.
If I try to input values manually, it gets timed out as well:
const keyParams = {
keyId: 'xxxxxxxx',
keyType: 'API_KEY',
usagePlanId: 'yyyyyyyy'
};
const apiKey = await new Promise((resolve, reject) => {
apiGateway.createUsagePlanKey(keyParams, function (err, data) {
if (err) {
console.log('createUsagePlanKey err', err, err.stack);
reject(err);
} else {
console.log('createUsagePlanKey', data);
resolve(data);
}
});
});
Why do every function call to api get timed out and nothing gets printed in console.log? Is my approach ok or how should I create the new api key for a user?
Edited: Timeout for lambdas is 10 seconds and they run in VPC
It sounds like you probably haven't configured your VPC to allow your Lambda function to access resources (like the AWS API) that exist outside the VPC. First, is it really necessary to run the function inside a VPC? If not then removing it from the VPC should fix the issue.
If it is necessary to run the function in a VPC, then you will need to place your Lambda function inside a private subnet with a route to a NAT Gateway, or configure a VPC endpoint for the AWS services it needs to access.
Related
The code below works when I have added the AWS IAM role "AdministratorAccess" - But it is risky and a bit of overkill... But how do I know and find only the necessary role(s)...It is very confusing and hard to know by when I look at all the possible roles in the console?
try {
// Load the AWS SDK for Node.js
var AWS = require('aws-sdk');
// Set the region
AWS.config.update({region: 'us-east-2'});
var instanceParams = {
ImageId: 'ami-xxxxxxxxxxxx',
InstanceType: 't2.micro',
KeyName: 'xxxxxxxxxx',
SecurityGroups: ['xxxxxxxxxxxxxxx'],
MinCount: 1,
MaxCount: 1
};
// Create a promise on an EC2 service object
var instancePromise = new AWS.EC2({apiVersion: '2016-11-15'}).runInstances(instanceParams).promise();
// Handle promise's fulfilled/rejected states
instancePromise.then(
function (data) {
console.log(data);
var instanceId = data.Instances[0].InstanceId;
console.log("Created instance", instanceId);
// Add tags to the instance
var tagParams = {
Resources: [instanceId], Tags: [
{
Key: 'Name',
Value: 'SDK Sample'
}
]
};
// Create a promise on an EC2 service object
var tagPromise = new AWS.EC2({apiVersion: '2016-11-15'}).createTags(tagParams).promise();
// Handle promise's fulfilled/rejected states
tagPromise.then(
function (data) {
console.log("Instance tagged");
}).catch(
function (err) {
console.error(err, err.stack);
});
}).catch(
function (err) {
console.error(err, err.stack);
});
}
catch(e){
wl.info('Error: ' + e);
}
Firstly you can see the api's you are calling via the sdk as a hint to what permissions you need i.e ec2:RunInstance and ec2:CreateTags.
You first create a policy then select the service then attach permissions (RunInstances and CreateTags)
You then create a Role with that policy attached.
Then you can attach the role to your Lambda
I would like to scale out my aws fargate containers based on the size of the SQS queue. It appears that I can only scale based on the container's CPU or Memory usage. Is there a way to create a policy that would scale out and in based on queue size? Has anyone been able to scale based on other cloudwatch metrics?
Yes you can do this. You have to use a step scaling policy, and you need to have an alarm created already for your SQS queue depth (ApproximateNumberOfMessagesVisible).
Go to CloudWatch, create a new alarm. We'll call this alarm sqs-queue-depth-high, and have it trigger when the approximate number of messages visible is 1000.
With that done, go to ECS to the service you want to autoscale. Click Update for the service. Add a scaling policy and choose the Step Tracking variety. You'll see there's an option to create a new alarm (which only lets you choose between CPU or MemoryUtilization), or use an existing alarm.
Type sqs-queue-depth-high in the "Use existing alarm" field and press enter, you should see a green checkmark that lets you know the name is valid (i.e. the alarm exists). You'll see new dropdowns where you can adjust the step policy now.
This works for any metric alarm and ECS services. If you're going to be trying to scale this setup out, for multiple environments for example, or making it any more sophisticated than 2 steps, do yourself a favor and jump in with CloudFormation or Terraform to help manage it. Nothing is worse than having to adjust a 5-step alarm across 10 services.
AWS provides a solution for scaling based on SQS queue: https://docs.aws.amazon.com/autoscaling/ec2/userguide/as-using-sqs-queue.html
Main idea
Create a CloudWatch Custom Metric sqs-backlog-per-task using formula:
sqs-backlog-per-task = sqs-messages-number / running-task-number.
Create a Target Tracking Scaling Policy based on the backlogPerInstance metric.
Implementation details
Custom Metric
In my case all the infrastructure (Fargate, SQS, and other resources) is described in CloudFormation stack. So for calculating and logging the custom metric I decided to use AWS Lambda function which is also described in CloudFormation stack and deployed together with the entire infrastructure.
Below you can find code snippets for the AWS Lambda function for logging the following custom metrics:
sqs-backlog-per-task - used for scaling
running-task-number - used for scaling optimization and debugging
AWS Lambda function described in AWS SAM syntax in CloudFormation stack (infrastructure.yml):
CustomMetricLoggerFunction:
Type: AWS::Serverless::Function
Properties:
FunctionName: custom-metric-logger
Handler: custom-metric-logger.handler
Runtime: nodejs8.10
MemorySize: 128
Timeout: 3
Role: !GetAtt CustomMetricLoggerFunctionRole.Arn
Environment:
Variables:
ECS_CLUSTER_NAME: !Ref Cluster
ECS_SERVICE_NAME: !GetAtt Service.Name
SQS_URL: !Ref Queue
Events:
Schedule:
Type: Schedule
Properties:
Schedule: 'cron(0/1 * * * ? *)' # every one minute
AWS Lambda Javascript code for calculating and logging (custom-metric-logger.js):
var AWS = require('aws-sdk');
exports.handler = async () => {
try {
var sqsMessagesNumber = await getSqsMessagesNumber();
var runningContainersNumber = await getRunningContainersNumber();
var backlogPerInstance = sqsMessagesNumber;
if (runningContainersNumber > 0) {
backlogPerInstance = parseInt(sqsMessagesNumber / runningContainersNumber);
}
await putRunningTaskNumberMetricData(runningContainersNumber);
await putSqsBacklogPerTaskMetricData(backlogPerInstance);
return {
statusCode: 200
};
} catch (err) {
console.log(err);
return {
statusCode: 500
};
}
};
function getSqsMessagesNumber() {
return new Promise((resolve, reject) => {
var data = {
QueueUrl: process.env.SQS_URL,
AttributeNames: ['ApproximateNumberOfMessages']
};
var sqs = new AWS.SQS();
sqs.getQueueAttributes(data, (err, data) => {
if (err) {
reject(err);
} else {
resolve(parseInt(data.Attributes.ApproximateNumberOfMessages));
}
});
});
}
function getRunningContainersNumber() {
return new Promise((resolve, reject) => {
var data = {
services: [
process.env.ECS_SERVICE_NAME
],
cluster: process.env.ECS_CLUSTER_NAME
};
var ecs = new AWS.ECS();
ecs.describeServices(data, (err, data) => {
if (err) {
reject(err);
} else {
resolve(data.services[0].runningCount);
}
});
});
}
function putRunningTaskNumberMetricData(value) {
return new Promise((resolve, reject) => {
var data = {
MetricData: [{
MetricName: 'running-task-number',
Value: value,
Unit: 'Count',
Timestamp: new Date()
}],
Namespace: 'fargate-sqs-service'
};
var cloudwatch = new AWS.CloudWatch();
cloudwatch.putMetricData(data, (err, data) => {
if (err) {
reject(err);
} else {
resolve(data);
}
});
});
}
function putSqsBacklogPerTaskMetricData(value) {
return new Promise((resolve, reject) => {
var data = {
MetricData: [{
MetricName: 'sqs-backlog-per-task',
Value: value,
Unit: 'Count',
Timestamp: new Date()
}],
Namespace: 'fargate-sqs-service'
};
var cloudwatch = new AWS.CloudWatch();
cloudwatch.putMetricData(data, (err, data) => {
if (err) {
reject(err);
} else {
resolve(data);
}
});
});
}
Target Tracking Scaling Policy
Then based on the sqs-backlog-per-task metric I created Target Tracking Scaling Policy in my Cloud Formation template.
Target Tracking Scaling Policy based on the sqs-backlog-per-task metric (infrastructure.yml):
ServiceScalingPolicy:
Type: AWS::ApplicationAutoScaling::ScalingPolicy
Properties:
PolicyName: service-scaling-policy
PolicyType: TargetTrackingScaling
ScalingTargetId: !Ref ServiceScalableTarget
TargetTrackingScalingPolicyConfiguration:
ScaleInCooldown: 60
ScaleOutCooldown: 60
CustomizedMetricSpecification:
Namespace: fargate-sqs-service
MetricName: sqs-backlog-per-task
Statistic: Average
Unit: Count
TargetValue: 2000
As a result AWS Application Auto Scaling creates and manages the CloudWatch alarms that trigger the scaling policy and calculates the scaling adjustment based on the metric and the target value. The scaling policy adds or removes capacity as required to keep the metric at, or close to, the specified target value. In addition to keeping the metric close to the target value, a target tracking scaling policy also adjusts to changes in the metric due to a changing load pattern.
Update to 2021 (before maybe...)
For those who need it but in CDK
An example use case:
// Create the vpc and cluster used by the queue processing service
const vpc = new ec2.Vpc(stack, 'Vpc', { maxAzs: 2 });
const cluster = new ecs.Cluster(stack, 'FargateCluster', { vpc });
const queue = new sqs.Queue(stack, 'ProcessingQueue', {
QueueName: 'FargateEventQueue'
});
// Create the queue processing service
new QueueProcessingFargateService(stack, 'QueueProcessingFargateService', {
cluster,
image: ecs.ContainerImage.fromRegistry('amazon/amazon-ecs-sample'),
desiredTaskCount: 2,
maxScalingCapacity: 5,
queue
});
from:
https://github.com/aws/aws-cdk/blob/master/design/aws-ecs/aws-ecs-autoscaling-queue-worker.md
I wrote a blog article about exactly this topic including a docker container to run it.
The article can be found at:
https://allaboutaws.com/how-to-auto-scale-aws-ecs-containers-sqs-queue-metrics
The prebuild container is available at DockerHub:
https://hub.docker.com/r/sh39sxn/ecs-autoscaling-sqs-metrics
The files are available at GitHub:
https://github.com/sh39sxn/ecs-autoscaling-sqs-metrics
I hope it helps you.
I have lambda function for auto Ami backup is possible to execute lambda across the region for take automatic backup of all my EC2 working on account.
One lambda function execution for all ec2 across region
var aws = require('aws-sdk');
aws.config.region = 'us-east-1','ap-south-1','eu-central-1';
var ec2 = new aws.EC2();
var now = new Date();
date = now.toISOString().substring(0, 10)
hours = now.getHours()
minutes = now.getMinutes()
exports.handler = function(event, context) {
var instanceparams = {
Filters: [{
Name: 'tag:Backup',
Values: [
'yes'
]
}]
}
ec2.describeInstances(instanceparams, function(err, data) {
if (err) console.log(err, err.stack);
else {
for (var i in data.Reservations) {
for (var j in data.Reservations[i].Instances) {
instanceid = data.Reservations[i].Instances[j].InstanceId;
nametag = data.Reservations[i].Instances[j].Tags
for (var k in data.Reservations[i].Instances[j].Tags) {
if (data.Reservations[i].Instances[j].Tags[k].Key == 'Name') {
name = data.Reservations[i].Instances[j].Tags[k].Value;
}
}
console.log("Creating AMIs of the Instance: ", name);
var imageparams = {
InstanceId: instanceid,
Name: name + "_" + date + "_" + hours + "-" + minutes,
NoReboot: true
}
ec2.createImage(imageparams, function(err, data) {
if (err) console.log(err, err.stack);
else {
image = data.ImageId;
console.log(image);
var tagparams = {
Resources: [image],
Tags: [{
Key: 'DeleteOn',
Value: 'yes'
}]
};
ec2.createTags(tagparams, function(err, data) {
if (err) console.log(err, err.stack);
else console.log("Tags added to the created AMIs");
});
}
});
}
}
}
});
}
where aws.config.region is for region config..it's working for current(in which lambda deploy) region
This line:
var ec2 = new aws.EC2();
connects to the Amazon EC2 service in the region where the Lambda function is running.
You can modify it to connect to another region:
var ec2 = new AWS.EC2({apiVersion: '2006-03-01', region: 'us-west-2'});
Thus, your program could loop through a list of regions (from ec2.describeRegions), creating a new EC2 client for the given region, then running the code you already have.
See: Setting the AWS Region - AWS SDK for JavaScript
In your Lambda Role, you need to add a policy which gives the Lambda function necessary permissions to access the EC2 on different accounts, typically you can add ARN's of EC2 instances you wan't access to or you can specify "*" which gives permissions to all instances.
Also on other accounts where EC2 instances are running you need to add IAM policy which gives access to your Lambda Role, note that you need to provide Lambda role ARN,
In this way your Lambda role will have policy to access EC2 and cross account EC2 will have policy which grant's access to Lambda role.
Without this in place you might need to do heavy lifting of configuring IP's of each EC2 in each account.
Yes and you also need to point EC2 object to a region where the instance is running,
Any code (including a Lambda function) can create a client that connects to a different region.
I'm trying to create a lambda function in AWS which will create a new Stripe token:
import stripePackage from 'stripe';
const stripe = stripePackage('...');
module.exports.create = (event, context, callback) => {
stripe.tokens.create({
card: {
"number": 4242424242424242,
"exp_month": '02',
"exp_year": '22',
"cvc": '123'
}
}, (err, token) => {
if (err) {
console.log(err);
callback(null, {
statusCode: 400,
body: "error"
});
}
callback(null, {
statusCode: 200,
body: "ok"
});
console.log(token);
});
}
However, this will time out every time. I have a security group for outbound connections as follows:
Ports Destination
All 0.0.0.0/0
However the only thing I seem to be able to connect to are other AWS services. How can I open my Lambda function up to connections outside AWS?
You either need to remove the Lambda function from your VPC (if it doesn't need VPC resource access then adding it to the VPC only introduces performance issues anyway), or you need to make sure the Lambda function is in a private subnet of your VPC and that subnet has a route to a NAT Gateway.
I want some of my Lambda resources to push to an AWS IOT endpoint using aws-sdk's AWS.IotData({ endpoint: url }) function - where endpoint is a required parameter.
Right now, I am passing the endpoint URL via an environment variable to my Lambda. However, when put into a SAM/CF template, I can't find a way to retrieve my IOT endpoint URL, so that I could simply !Ref it.
Browsing through the AWS resource type reference I did not find any resource that corresponds to an IOT endpoint.
It seems like IOT endpoint can only be provisioned manually, via AWS Console (enabled / disabled), as on the screenshot below:
Any advice on how to have control over provisioning an IOT endpoint or at least reading the IOT URL from within a SAM/CF template, without scripting this with aws-cli?
For anyone interested in the solution with CloudFormation Custom Resource, I wrote a simple Lambda and a CF template that provides an IOT endpoint address to other CF stacks.
template.yaml
AWSTemplateFormatVersion: '2010-09-09'
Transform: 'AWS::Serverless-2016-10-31'
Resources:
IotEndpointProvider:
Type: 'AWS::Serverless::Function'
Properties:
FunctionName: IotEndpointProvider
Handler: iotEndpointProvider.handler
Runtime: nodejs6.10
CodeUri: .
MemorySize: 128
Timeout: 3
Policies:
- Version: '2012-10-17'
Statement:
- Effect: Allow
Action:
- iot:DescribeEndpoint
Resource:
- '*'
IotEndpoint:
Type: 'Custom::IotEndpoint'
Properties:
ServiceToken: !GetAtt IotEndpointProvider.Arn
Outputs:
IotEndpointAddress:
Value: !GetAtt IotEndpoint.IotEndpointAddress
Export:
Name: IotEndpointAddress
iotEndpointProvider.js
var aws = require("aws-sdk");
exports.handler = function(event, context) {
console.log("REQUEST RECEIVED:\n" + JSON.stringify(event));
// For Delete requests, immediately send a SUCCESS response.
if (event.RequestType == "Delete") {
sendResponse(event, context, "SUCCESS");
return;
}
const iot = new aws.Iot();
iot.describeEndpoint({}, (err, data) => {
let responseData, responseStatus;
if (err) {
responseStatus = "FAILED";
responseData = { Error: "describeEndpoint call failed" };
console.log(responseData.Error + ":\n", err);
} else {
responseStatus = "SUCCESS";
responseData = { IotEndpointAddress: data.endpointAddress };
console.log('response data: ' + JSON.stringify(responseData));
}
sendResponse(event, context, responseStatus, responseData);
});
};
// Send response to the pre-signed S3 URL
function sendResponse(event, context, responseStatus, responseData) {
var responseBody = JSON.stringify({
Status: responseStatus,
Reason: "See the details in CloudWatch Log Stream: " + context.logStreamName,
PhysicalResourceId: context.logStreamName,
StackId: event.StackId,
RequestId: event.RequestId,
LogicalResourceId: event.LogicalResourceId,
Data: responseData
});
console.log("RESPONSE BODY:\n", responseBody);
var https = require("https");
var url = require("url");
var parsedUrl = url.parse(event.ResponseURL);
var options = {
hostname: parsedUrl.hostname,
port: 443,
path: parsedUrl.path,
method: "PUT",
headers: {
"content-type": "",
"content-length": responseBody.length
}
};
console.log("SENDING RESPONSE...\n");
var request = https.request(options, function(response) {
console.log("STATUS: " + response.statusCode);
console.log("HEADERS: " + JSON.stringify(response.headers));
// Tell AWS Lambda that the function execution is done
context.done();
});
request.on("error", function(error) {
console.log("sendResponse Error:" + error);
// Tell AWS Lambda that the function execution is done
context.done();
});
// write data to request body
request.write(responseBody);
request.end();
}
I'm afraid you cannot provision an IoT endpoint, as the only API call that is related to an IoT endpoint is DescribeEndpoint.
What you can do is create a Lambda-backed CloudFormation Custom Resource. The Lambda function will execute the DescribeEndpoint call (using the AWS SDK of your choice depending on the Lambda's runtime) and return the endpoint's URL so your other CloudFormation resources can consume it.
Here's a good example on Lambda-backed Custom Resources: http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/template-custom-resources-lambda.html.