I'm running through a loop with a PHP script that's on AWS instance. From my experiences with AWS, as soon as the instance is stopped, all of the code that's in the process of being executed is stopped. What I have is this:
<?php
require("vendor/autoload.php");
use Aws\Ec2\Ec2Client;
$instance_id = 'instance_id';
$creds = array('key' => 'key',
'secret' => 'secret',
'region' => 'us-west-2');
$client = Ec2Client::factory($creds);
$instance = array('InstanceIds' => array($instance_id), 'DryRun' => false);
for($i=0;$i<10;$i++) {
// Execute irrelevant code
// .....
$result = $client->stopInstances($instance);
sleep(300);
$result = $client->startInstances($instance);
}
?>
So, my question is this: Once the instance is stopped, everything that is written after that will not be executed since the instance will be stopped, right? The loop will not continue on to the next iteration, right? If so, then how could I get around that?
When you call the stopinstances api, EC2 will start shutting down your instance (and the OS inside the instance will kill running processes as part of it)
There's no guarantee exactly how long this will take, although in my experience you'll rarely get more than a couple of seconds, so that sleep(300) pretty much guarantees that the call to stopInstances is the last thing that your code will do.
There's nothing you can do about this other than not stopping the instance you are running on. To that end you can query the instance metadata service to find out what the id of the instance running your code is. You can get this data by making a request to http://169.254.169.254/latest/meta-data/instance-id
You cannot start an instance that is Stopped from the same instance. You can keep an additional (external) server either on EC2 or otherwise to control automatic shutdowns/startups.
To follow on from #TJ-'s answers...
You can check to see if the instance is stopped and then continue with your code
$client->waitUntil('InstanceStopped', array('InstanceIds' => $instanceId));
But you have to run this from a different instance than the one being terminated.
Related
Creating an instance in lambda and assigning an Elastic IP. Create instance part works, code section below is meant to wait before assigning the elastic IP. (1) What state does the instance have to be in to assign? (assuming "running") (2) Below logic never proceeds to after the loop, although I can verify the instance goes into "running" state. I verified the instance ID is good and there's only 1 instance in this case.
print ('waiting')
newresp = ec2_client.describe_instance_status(InstanceIds=newins_list,IncludeAllInstances=True)
while (newresp['InstanceStatuses'][0]['InstanceState']['Name'] != 'running'):
newresp = ec2_client.describe_instance_status(InstanceIds=newins_list,IncludeAllInstances=True)
print ('New Instance Running')
The "timeout" setting is important in lambda. This affects loops and timers. If you have for/while loops, sleep, or other functions that take time, lambda should be configured.
When I created the function the default setting was 3s execution time. The statement after the loop was never reached because the function timed out.
You can set the timeout from the function "Configuration" tab under General Configuration.
You have to remember though, pricing is based on the amount of time your code runs. I set it to 30s, and checked status at 5s intervals in the loop. Here's the final code, starting from the loop section, including the associate command:
....
while (newresp['InstanceStatuses'][0]['InstanceState']['Name'] != 'running'):
time.sleep(5)
newresp = ec2_client.describe_instance_status(InstanceIds=newins_list,IncludeAllInstances=True)
print ('Associating Elastic IP')
try:
ec2_client.associate_address(AllocationId='eipalloc-0805514a57680aaf8',InstanceId=newins,AllowReassociation=True)
print ('New Instance Running')
except ClientError as e:
print(e)
I am fairly new to AWS and would like your suggestions. The problem I would like to solve is that I want to automate the process. I have this ec2 image running ubuntu and I want to call this executable "executable_hello_world_repeat" inside the image which prints "Hello World" every second. and when calling the executable I want to add input parameters such as "executable_hello_world_repeat -n10" this would print "hello world" 10 times.
Manually I can do the following:
go to AWS management console and choose the ec2 image to start
check if the instance is running successfully
from the terminal call "executable_hello_world_repeat -n10"
it prints the "Hello World"
I want to write a program to do them all programatically. Eventually I will have a web page in React/JS and automate this process.
Thanks for reading.
When an Amazon EC2 instance is first launched, a User Data script can be provided, which is automatically executed as the root user towards the end of the boot process. You can use this script to install software, configure settings, start process, etc.
Please note that this script only runs on the first boot, because the software does not need to be installed on subsequent boots.
If you want a script to run on every boot, put it in the /var/lib/cloud/scripts/per-boot/ directory.
If you later want to trigger a script to run, then you will need some mechanism that receives this request and runs the script. A few ways you could do this are:
Run a web server on the instance and the request comes via an HTTP / REST request, or
Trigger the AWS Systems Manager Run Command that will cause a script to be run on the instance, or even multiple instances, or
Have a program or script running on the instance that is continuously polling an Amazon SQS queue. When a message is received from the queue, trigger a program/script to process the message. This is known as a "Worker" that pulls from the Queue
The EC2 instance is basically just a normal Linux instance, so you'll need to somehow get something to trigger on the instance when desired.
I have the following python code to detect whether an EC2 is really started. But it completes when "instance state" shows running.
which API function should I use to block until EC2 "status check" show "2/2 checks passed"
ec2 = boto3.resource('ec2')
instance = ec2.Instance(instanceid)
instance.wait_until_running()
It is rare that you would need to wait for the status check to pass.
When an instance enters the running state, the machine boots, loads the operating system and generally "runs".
The EC2 Status Checks are an independent process that check attributes of the virtual machine. However, your machine is normally running, and you can login to it, well before the status checks show a positive response.
If you do wish to wait for the Status Check, there are two waiters that might do this, but the documentation is unclear:
InstanceStatusOk
SystemStatusOk
Using the node aws-sdk, I check the status of an instance, and want to let a user start it if it is dead, or kill it if it is alive.
I found that there is a method called runinstances but it seems like it creates new instances, and I want to revive a live one.
Is there a way using the node sdk to start/kill an instance?
You can write the logic to check the EC2 instance status using describeInstanceStatus method which will return the instance states (InstanceState.Name)
pending
running
shutting-down
terminated
stopping
stopped
Based on the current state (running or stopped) you can toggle the instance state using either of the following method.
startInstances
stopInstances
I want to use AWS AutoScaling to scaledown a group of instances when SQS queue is short.
These instances do some heavy work that sometimes requires 5-10 minutes to complete. And I want this work to be completed before the instance termination.
I know a lot of people should have faced the same problem. Is it possible on EC2 to handle the AWS termination request and complete all my running processes before the instance is actually terminated? What is the best approach to this?
You could also use Lifecycle hooks. You would need a way to control a specific worker remotely, because AWS will select a particular instance to put in Terminating:Wait state and you need to manage that instance. You would want to take the following actions:
instruct the worker process running on the instance to not accept any more work.
wait for the worker to finish the work it already is handling
call the complete-lifecycle action.
AWS will take care of the rest for you.
ps. if you are using celery to power your workers then you can remotely ask a worker to shutdown gracefully. It won't shutdown unless it finishes with the tasks it had started executing.
Assuming you are using linux, you can create a pre-baked AMI that you use in your Launch Config attached to your Auto Scaling Group.
In the AMI you can put a script under /etc/init.d say /etc/init.d/servicesdown. This script would execute anything that you need to shutdown which would be scripts under /usr/share/services for example.
Here's kind like the gist:
servicesdown
It would always get executed when doing a graceful shutdown.
Then say on Ubuntu/Debian you would do something like this to add it to your shutdown sequence:
/usr/sbin/update-rc.d servicesdown stop 25 0 1 6 .
On CentOS/RedHat you can use the chkconfig command to add it to the right shutdown runlevel.
I stumbled onto this problem because I didn't want to terminate an instance that was doing work. Thought I'd share my findings here. There are two ways to look at this though :
I need to terminate a worker, but I only want to terminate one that's not working
I need to terminate a SPECIFIC worker and I want that specific worker to wait until it's done with the work.
If you're goal is #1, Amazon's new "Instance Protection" looks like it was designed to resolve this.
See the below link for an example, they give this code snippet as an example:
https://aws.amazon.com/blogs/aws/new-instance-protection-for-auto-scaling/
while (true)
{
SetInstanceProtection(False);
Work = GetNextWorkUnit();
SetInstanceProtection(True);
ProcessWorkUnit(Work);
SetInstanceProtection(False);
}
I haven't tested this myself, but I see API calls related to setting the protection, so it appears that this could be integrated into the EC2 Worker App code-base and then when Scaling In, instances shouldn't be terminated if they are protected (currently working).
http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/autoscaling/AmazonAutoScaling.html
As far as I know currently there is no option to terminate instance while gracefully shutdown and let process to complete work.
I suggest you to look at http://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide/as-configure-healthcheck.html.
We implemented it for resque workers while we are moving instance to unhealthy state and than downsizing AS. There is a script which checking constantly health state on each instance. Once instance moved to unhealthy state it stops all services gracefully and sending terminate signal to ec2.
Hope it helps you.