I am trying to add the lambda in the Auto-scaling target, and getting the error "No scalable resources found" while trying to fetch by tag.
Is it possible or allowed to add lambda to the auto-scaling target?
UPDATE:
I am trying to figure out how to change the provisional concurrency during non-peak hours on the application that will help to save some cost so I was exploring the option of auto-scaling
Lambda automatically scales out for incoming requests, if all existing execution contexts (lambda instances) are busy. There is basically nothing you need to do here, except maybe set the maximum allowed concurrency if you want to throttle.
As a result of that, there is no integration with AutoScaling, but you can still use an Application Load Balancer to trigger your Lambda Function if that's what you're after.
If you're building a purely serverless application, you might want to look into the API Gateway instead of the ALB integration.
Update
Since you've clarified what you want to use auto scaling for, namely changing the provisioned concurrency of the function, there are ways to build something like that. Clément Duveau has mentioned a solution in the comments that I can get behind.
You can create a Lambda Function with two CloudWatch events triggers with Cron-Expressions. One for when you want to scale out and another one for when you want to scale in.
Inside the lambda function you can use the name of the rule that triggered the function to determine if you need to do a scale out or scale in. You can then use the PutFunctionConcurrency API-call through one of the SDKs mentioned at the bottom of the documentation to adjust the concurrency as you see fit.
Update 2
spmdc has mentioned an interesting blog post using application auto scaling to achieve this, I had missed that one - you might want to check it out, looks promising.
Related
Is there a way for me to run a cloud function to update some data whenever there is lots of activity to one of my pages? sorry if this is a stupid question.
Cloud Functions does not have any triggers that automatically respond to the load on a web site. You will have to find some other way to gauge that traffic, and then perhaps invoke an HTTP trigger directly.
If you want to see the complete list of trigger types, see the documentation.
as far as I know there is no such thing as a direct trigger between a spike of events in service A and cloud function (unless the service A is itself a cloud function, that would indeed scale).
However it exists second party trigger with stackdriver that allows you to define rules that once hit would deliver an event to a pub/sub topic that a cloud function of your choice can suscribe to. Be cautious with what you intend to do with your cloud function, you may have concurency issues depending the technology of the database you want to modify. In other word, try to be as idempotent as possible.
I think you may be interested by this community tutorial.
I have 2 instances, connected to a load balancer. I would like to stop 1 instance, and start it only when a certain alarm happens, for example when the first intance has a high CPU load.
I couldn't find how to do it. in the Auto scaling group, i see i can launch a brand new instance, but that's not what i want, i want a specific instance to start.
I couldn't find how to connect an alert to an action - wake up this specific instance.
Should this be done in the load balancer configuration? i couldn't find how...
This is not really the way autoscaling is supposed to work, and hence the solution to your particular problem is a little bit more complex than simply using autoscaling to create new instances in response to metric thresholds being crossed. It may be worth asking yourself exactly why you need to do it this way and whether it could be achieved in the usual way.
In order to achieve starting (and stopping) a particular instance you'll need three pieces:
A CloudWatch Alarm triggered by the metric you need (CPUUtilization) crossing your desired threshold.
An SNS topic that is triggered by the alarm in the previous step.
A lambda function (with the correct IAM permissions) that is subscribed to the SNS topic, which sends the relevant API calls to EC2 to start or stop the instances when the notification from SNS arrives. You can find some examples of the code needed to do this eg here in node.js and here from AWS although there are probably others if you prefer another language.
Once you put all these together you should be able to respond to a change in CPU with starting and stopping particular instances.
I have a batch job that I need to run on AWS. I'm wondering what's the best service to use. The job needs to run once a day, so I think that naturally AWS Lambda with a CloudWatch Rule triggering it would do it. However, I'm starting to think that AWS Lambda is thought to be used as a service to handle requests. This AWS official library to integrate Spring-Boot is very oriented to handle HTTP requests, and when creating a lambda via AWS Console, only test cases that send an input to the lambda can be written.
Then, is this a use case for AWS Lambda? Also, these functions can run up to 15 minutes. What should I use if my job needs to run longer?
The purpose of Lambda, as compared to AWS EC2, is to simplify building smaller, on-demand applications that are responsive to events and new information.
If your batch is running within a limit of 15 minutes then you can go with a lambda function.
But if you want batch processing to be done, you should check AWS batch.
Here is nice article which demonstrates the usage of AWS batch.
If you are already using some batch framework like spring-batch, you can also take a look at ECS scheduled task with Fargate.
With ECS Fargate you can launch and stop container services that you need to run only at certain times.
Here are some related articles on Fargate event and scheduled task and Scheduled Tasks.
If you're confident that your function will only run at maximum of 15mins, AWS Lambda could be the solution. Here are the AWS Lambda limits that could help you decide on that.
Also note that lambda has cold start, it's when it will run slower at first but will eventually pick up the pace. Here are some good reads about it that could help you decide on the lambda direction, but feel free to check on any articles that could better explain at your disposal.
This one shows a brief lists that you would like to consider and the factors affecting it.
This one might have a deeper explanation of the cold start with regards to how it works internally.
What should I use if my job needs to run longer?
Depending on your infrastructure, you could maybe explore Scheduled Tasks
To reduce the cost on instances, we were looking for options.
AWS lambda seems to be a good option for us.
Its still in the preliminary stage of searching for available alternatives.
My concern is if we switch some of our applications to lambda, we will be confined to use AWS environments only , and in future it might become a boundation for a scenario , which we cant predict at the moment.
So my question is, is there a way that we can still use lambda in an environment which is not an AWS environment.
Thanks!
AWS Lambda functions are basically containers, where its lifecycle is managed by Amazon.
When you use Lambda, there are several best practices you can follow, to avoid full locking. One of the recommended practice is to separate the business logic from Lambda handler. When you separate the Lambda handler, it only works as the controller which points to the executing code.
/handler.js
/lib
/create-items
/list-items
For example, if you design a web application API this way with NodeJS in Lambda, you can later move the business logic to an ExpressJS server by moving the handler code to ExpressJS Routes.
As you can see, you will still require putting additional effort to move an application from Lambda to another environment. By properly designing, you can only reduce the efforts.
As per my knowledge,
Its AWS lambda function, so it is suppose to be deployed on AWS instances only, because they support the needed environment.
From AWS site there are couple of options ...
https://docs.aws.amazon.com/lambda/latest/dg/deploying-lambda-apps.html
I've got my web servers set up to autoscale at certain times of the day.
I can measure the load on the box using scripts executed by Consul - and this can trigger events at certain thresholds.
I want to push these two together and trigger autoscaling at certain load levels. (Assume CPU load at 75% is the threshold).
My question is: What is the mechanism to get load on a box to trigger an autoscaling group in AWS?
Assumptions:
I was not planning to use AWS Cloudwatch - but am interested if this is the solution.
I'm more interested in the autoscale triggering interface. Is it a queue or a rest endpoint?
As #mahdi said, you can easily use AWS Cloudwatch to do this.
However, if you want Consul (or anything outside the scope of an AWS "service") to do it, you can use lambda.
You would create a lambda function that scales your instance up or down (or both). Lambda can have many triggers, such as an HTTP endpoint through API Gateway. If you already have Consul set up to do it (sounds like you do since you said can trigger events at certain thresholds.) just make it issue the HTTP request to API Gateway to scale up or down.
You can create a CloudWatch alarm with a CPUUtilization metric and set it to change state when your instance has CPU utilization more than 75%. Then in the Auto Scaling Group, you use this alarm for scaling (in/out) policy. You can also control the number of instances in an Auto Scaling Group by manually (e.g. through your application running on one the instances) changing the Desired value. This documentation can be helpful.