I have a full deployment job that takes an ip of a running instance and deploys my system on it.
I currently hold an EC2 instance for automation tests that run every night, but the instance is expensive and im looking for a way to initiate it before the tests and terminate it after the test.
I looked for EC2 plugins that can help and the closest one was this but this is for making slaves and thats not what I want.
I want to be able to launch an EC2 instance, and pass its IP address to the automation tests job, then terminate that instance once done.
I started making a command line bash file for this, but this seems like too much work, and I thought maybe there is something im missing.
Your requirement is valid and amazon knows:
When you stop an instance, we shut it down. We don't charge usage for a stopped instance, or data transfer fees, but we do charge for the storage for any Amazon EBS volumes.
Reference :
- https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Stop_Start.html
Here some approaches to start/stop your instaces
Amazon EC2 HTTP API
This is an api rest and you can perform a simple http request to start or stop your instance:
Amazon EC2 API Reference
Start Instance endpoint
https://ec2.amazonaws.com/?Action=StartInstances&...
Stop Instance endpoint
https://ec2.amazonaws.com/?Action=StopInstances&...
You can invoke this api from Jenkins in many ways : simple shell execution,groovy and scripted/declarative, pipelines.
AWS CLI
start instance
stop instance
Here more about how suspend instances using aws cli:
https://docs.aws.amazon.com/autoscaling/ec2/userguide/as-suspend-resume-processes.html
Also with powershell: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Stop_Start.html
You can invoke this api from Jenkins in many ways : simple shell execution,groovy and scripted/declarative, pipelines.
AWS Instance Scheduler
In 2018, AWS launched the AWS Instance Scheduler, a new and improved scheduling solution that enables customers to schedule Amazon EC2 instances.
With this tool you can automatically start and stop the Amazon EC2 and Amazon RDS instances.
Reference :
https://aws.amazon.com/answers/infrastructure-management/instance-scheduler/
With this approach you don't need Jenkins :b
Related
I have some etl python scripts on an EC2 instance which I run in order to fetch, process and load some data in a Postgresql DB that is hosted on the same EC2 instance.
The request was to host the DB on the EC2 instance instead of using RDS in order to save money.
The scripts take 1 hour to complete (because of some heavy processing involved which consumes 90% of the time).
I would like to build a solution to automate this task.
What I would like to do is to:
Turn on the EC2 instance at a given time;
Run the scripts inside the EC2 instance;
Turn off the EC2 instance after the scripts finish their job (which means after 1 hour);
I've seen some solutions with AWS System Manager and AWS Lambda, but they all seem outdated since the AWS interface changed.
I have a service that needs to run on our own EC2 instances, since it requires some support from the kernel. My previous experience is all with containers in AWS. The application itself is distributed as a single JAR file and I'm looking for advice for how I should automate deployments. The architecture is:
An ALB in front of the ASG.
EC2 instance running a single Java application.
Any open sockets are open for an hour tops and to not cause any trouble, we have to drain the connections to the EC2 instances before performing an update, so a hard requirement is for the ALB to stop opening new connections for an hour before updating the software. The application is mission critical and ECS has had some issues last year, so I want to minimize the AWS services I depend on. While I could do what I want on my own ECS cluster with custom AMIs, I don't want to do it, since I will run a single instance of the app per host and don't need the extra layer.
My question: What is the simplest method to achieve this using CodePipeline? My understanding is that I need to use a CodeDeploy deployment step to push something to bare EC2 instances. How does draining with an ALB work in this case? We're using CloudFormation for the deployment.
You need to use codedeploy. You can find tutorial on AWS codedeploy documentation.
Codedeploy deployment lifecycle hooks for EC2.
https://docs.aws.amazon.com/codedeploy/latest/userguide/reference-appspec-file-structure-hooks.html#appspec-hooks-server
I have a Docker image containing Python code and third-party binary executables. There are only outbound network requests. The image must run hourly and each execution lasts ~3 minutes.
I can:
Use an EC2 instance and schedule hourly execution via cron
Create a CloudWatch Event/Rule to run an ECS Task Defintion hourly
Setup an Elastic Beanstalk environment and schedule hourly deployment of the image
In all of these scenarios, an EC2 instance is running 24/7 and I am being charged for extended periods of no usage.
How do I accomplish scheduling the starting of an existing EC2 instance hourly and the stopping of said instance after the completion of my docker image?
Here's one approach I can think of. It's very high-level, and omits some details, but conceptually it would work just fine. You'll also need to consider the Identity & Access Management (IAM) Roles used:
CloudWatch Event Rule to trigger the Step Function
AWS Step Function to trigger the Lambda function
AWS Lambda function to start up EC2 instances
EC2 instance polling the Step Functions service for Activity Tasks
Create a CloudWatch Event Rule to schedule a periodic task, using a cron expression
The Target of the CloudWatch Event Rule is an AWS Step Function
The AWS Step Function State Machine starts by triggering an AWS Lambda function, which starts the EC2 instance
The next step in the Step Functions State Machine invokes an Activity Task, representing the Docker container that needs to execute
The EC2 instance has a script running on it, which polls the Activity Task for work
The EC2 instance executes the Docker container, waits for it to finish, and sends a completion message to the Step Functions Activity Task
The script running on the EC2 instance shuts itself down
The AWS Step Function ends
Keep in mind that a potentially better option would be to spin up a new EC2 instance every hour, instead of simply starting and stopping the same instance. Although you might get better startup performance by starting an existing instance vs. launching a new instance, you'll also have to spend time to maintain the EC2 instance like a pet: fix issues if they crop up, or patch the operating system periodically. In today's world, it's a commonly accepted practice that infrastructure should be disposable. After all, you've already packaged up your application into a Docker container, so you most likely don't have overly specific expectations around which host that container is actually being executed on.
Another option would be to use AWS Fargate, which is designed to run Docker containers, without worrying about spinning up and managing container infrastructure.
AWS Step Functions
AWS Fargate
Blog: AWS Fargate: An Overview
Creating a CloudWatch Event Rule that triggers on a schedule
I am able to monitor a Windows instance's memory usage using custom metrics in CloudWatch.
I have followed the following blog to achieve that :
http://blog.krishnachaitanya.ch/2016/03/monitor-ec2-memory-usage-using-aws.html
Using that, I am able to monitor only one instance. I am now doing the process in every instance launched.
Can I do it at once for all instances instead of changing .json file and enabling cloud watch integration in every instance?
If the instances are already launched, you have to do it for each instance. Else you can take an AMI of the first instance, then launch other instances from that AMI and you do not have to do it for each instance.
If you have to do it manually, consider something like Ansible to do it for you. There is a bit of learning but not difficult.
BTW, adding custom metrics is straightforward for Linux instances. Monitoring Memory and Disk Metrics for Amazon EC2 Linux Instances
For Windows instance: Sending Performance Counters to CloudWatch and Logs to CloudWatch Logs Using Amazon EC2 Simple Systems Manager
If your instances have the appropriate instance profile and are running the SSM agent (which they probably are if you launched from an Amazon provided AMI), you can use SSM run command to run arbitrary powershell against an instance or a set of instances (using tags). There is even a Amazon managed SSM document called AWS-ConfigureCloudWatch that is built specifically for this use case.
See http://docs.aws.amazon.com/systems-manager/latest/userguide/run-command.html
How can I start or stop EC2 instances based on a schedule (night, weekend)? are AWS EC2 API only solution for this?
AWS does not have any built-in method to schedule instances to start or stop. You have a few options:
Run your own EC2 instance that used cron or Task Manager to run your own scripts using the AWS SDK/CLI to start and stop the instances.
Use scheduled Auto Scaling to terminate and re-launch EC2 instances for you.
Use AWS Lambda to execute a Lambda function that starts and stops your instances, using the Lambda scheduler to schedule the execution of your Lambda function.
Use a third-party service that will start and stop your EC2 instances for you.
You can try my software, CloudRoboAWSScheduler. Download it from http://cloudrobo.net/download/
Please go through the setup details mentioned in http://cloudrobo.net/setup/
It is absolutely free.
Features:
- Easy to use GUI
- START / STOP EC2 Instances
- Reduce costs by starting and stopping your EC2 instances automatically
- Create dynamic schedules using ClouRobo AWS scheduler
- Manage multiple accounts in different AWS regions - Easy to use user interface
- Installs on Windows Server 2008/2012 or Windows 7/8/10
Note: This is a beta release.
-Viswanath