This question already has answers here:
Aws Ec2 run script program at startup
(5 answers)
Closed 1 year ago.
I am trying to run a python script on ec2 instance . The python file is residing on s3.
I am able to run manually from ec2 instance using iam role which allow access to s3 folder and files.
The question is , how can i automate the start and stop of ec2 instance whenever needed and how to invoke /pass a python file to run upon starting the ec2 instance and stop the instance once the python files completes the execution.
Thanks,
Nikhil
Your requirements seem to be:
Schedule an Amazon EC2 instance to start at a specific time every day
The instance should run a Python script after starting
When the Python script has finished running, Stop the instance
Start EC2 instance on a schedule
You can use Amazon EventBridge to trigger an AWS Lambda function on a schedule.
You can code the Lambda function to call StartInstances() on the EC2 instance to Start it.
Run a script on startup
Install a script into the /var/lib/cloud/scripts/per-boot/ directory. This script can download the Python program from S3 and then run it.
When the EC2 instance starts up, it will automatically run any script in that directory.
Stop the instance when the script is finished
At the end of the script, add the command:
shutdown -h now
This will turn off the instance and place it in the Stopped state.
(This assume that the script is running as root. If it is running as another user, it will need to use sudo shutdown -h now.)
EC2 instances use cloudinit which you can customize to run a given script on each boot. You can use use regular os tools from python to shutdown your instance (e.g. shutdown -h now).
Here another alternative could be to use lambda function instead of EC2 instance to run the python script if maximum execution time of script is less than 15 minutes. Go serverless with AWS lambda rather than EC2. just add your script code in AWS lambda and schedule lambda function from AWS event bridge to invoke it.
Related
I have a python scraper script and I wish to deploy and run it on 100 ec2 (spot?) instances at the same time. It should run them for an hour and then stop all of them. How should i do it? Is there any aws service that will allow me to orchestrate it easily?
When launching an Amazon EC2 instance (including those on Spot pricing), you can nominate the number of instances to launch. So, you can launch 100 spot instances with a single command.
One hour later, you could then issue a command to terminate the instances using a list of the instances returned by the first command.
Or, you could simply add a command in the startup script of each instance (issued via the User Data field) and waits for one hour and then issues a command to the operating system to shutdown (sudo shutdown now -h). This User Data script can also be used to install and run your scraper code on the instance.
I suggest that you experiment by launching just a single instance until you have things working the way you want.
I have followed ec2 run scripts every boot and tried the same approach for per-instance but my script still doesn't get executed.
I also followed Execute User Data After the Initial Launch of Your EC2 Instance and yet the script doesn't get executed. Content of the script:
nohup /home/ubuntu/anaconda3/envs/tensorflow_p36/bin/python3 -u medi_app.py &> nohup_medi &
Why is it proving so hard to run a script every time an instance restarts? Can someone please help out here?
If you are using an Amazon EC2 Linux instance to run a batch job on startup, simply install the script in this directory:
/var/lib/cloud/scripts/per-boot/
This will work on any system that has cloud-init installed, which also runs User Data scripts.
For more details, see: Auto-Stop EC2 instances when they finish a task - DEV Community
I have a requirement, where I need to run a shell script inside my ec2 instance when the instance state changes from stopped to running.
I used cloud watch for this requirement but the event is trigerring the shell script when the instance is in pending state but not in running state(I have used "ec2 instance state-change notification" and "running" as parameter. Target as RunCommand).
Now I am trying to use SSM association I was succeded in trigerring the shell script manually but i need to automate it (i.e., when ec2 instance status changed to running only the bash script should start running)
I think what you're looking for is how to automatically execute a script with every restart of your EC2 instance (these options do not use AWS Systems Manager).
Use user data scripts and cloud-init directives
By default, user data scripts and cloud-init directives run only
during the first boot cycle when an instance is launched. However, you
can configure your user data script and cloud-init directives with a
mime multi-part file. A mime multi-part file allows your script to
override how frequently user data is executed in the cloud-init
package. Then, the file executes the user script.
https://aws.amazon.com/premiumsupport/knowledge-center/execute-user-data-ec2/
Alternatively, you can execute a cron job after system reboot, see more info here:
https://www.cyberciti.biz/faq/linux-execute-cron-job-after-system-reboot/
I can see a generic way of starting an EC2 from lambda in Start and Stop Instances at Scheduled Intervals Using Lambda and CloudWatch.
Suppose I use that method to start an EC2, and suppose the AMI is a windows server 2019 customised to have a .bat file on the desktop, and also suppose I'm using a python lambda.
How can I execute this batch file from the lambda? (i.e. just as though someone had RDP'd into the instance and double-clicked on it)
Note: To be very clear, basically I want to start the EC2 using the method given in the AWS docs (above), and right after the instance has started, to run the batch file that will be sitting on the instance's desktop
I think you have a few concepts mixed together.
AWS Lambda functions run on the Lambda service, without having to use Amazon EC2 instances. This is what makes them "serverless".
If you have a batch file on an Amazon EC2 instance, you would presumably want to run that batch file on the EC2 instance itself, without involving Lambda (since you have got a server).
If you wish to run a script on an EC2 instance when it launches for the first time, you can provide a PowerShell or Command-Line script via the User Data field. Software on the AMI will automatically execute this script the first time that the instance starts.
This script could do all the work itself, or it could simply call another script that is stored on the disk. Some people use the script to download another script from a repository (eg Amazon S3 or GitHub) and then execute the downloaded script.
For more information, see: Running Commands on Your Windows Instance at Launch - Amazon Elastic Compute Cloud
If the Amazon EC2 instance is already running and you wish to trigger a script to execute, you can use the AWS Systems Manager Run Command. This works by having an agent on the instance which can be remotely triggered, thereby running scripts without having to login to the instance.
I'm running a instance in amazon AWS and it runs non-stop everyday. I'm using ubuntu ec2 instance which is running Apache, Mirthconnect tool and LAMP server. I want to run this instance only on particular time duration of a day. I prefer not use any additional AWS services such as cloud-watch . Is there a way we could acheive this?.
The major purpose is for using Mirthconnect fetching data from mysql database
There are 3 solutions.
AWS Data Pipeline - You can schedule the instance start/stop just like cron. It will cost you one hour of t1.micro instance for every start/stop
AWS Lambda - Define a lambda function that gets triggered at a pre defined time. Your lambda function can start/stop instances. Your cost will be very minimal or $0
Write a shell script and run it as a cron job or run it on demand. The script will have AWS CLI command to start and stop the instance.
I used Data Pipeline for a long time before moving to Lambda. Data Pipeline is very trivial. Just paste the AWS CLI commands to stop and start instances. Lambda is more involved.
I guess for that you'll need another machine which is on 24x7. On which you can write cron job in python using boto or any other language like bash.
I don't see how you start a instance in stopped state without using any other machine.
Or you can have a simple raspberry pi on at your home which does the ON-OFF work for you using AWS CLI or simple Python. How about that? ;)