I have a python scraper script and I wish to deploy and run it on 100 ec2 (spot?) instances at the same time. It should run them for an hour and then stop all of them. How should i do it? Is there any aws service that will allow me to orchestrate it easily?
When launching an Amazon EC2 instance (including those on Spot pricing), you can nominate the number of instances to launch. So, you can launch 100 spot instances with a single command.
One hour later, you could then issue a command to terminate the instances using a list of the instances returned by the first command.
Or, you could simply add a command in the startup script of each instance (issued via the User Data field) and waits for one hour and then issues a command to the operating system to shutdown (sudo shutdown now -h). This User Data script can also be used to install and run your scraper code on the instance.
I suggest that you experiment by launching just a single instance until you have things working the way you want.
Related
I have followed ec2 run scripts every boot and tried the same approach for per-instance but my script still doesn't get executed.
I also followed Execute User Data After the Initial Launch of Your EC2 Instance and yet the script doesn't get executed. Content of the script:
nohup /home/ubuntu/anaconda3/envs/tensorflow_p36/bin/python3 -u medi_app.py &> nohup_medi &
Why is it proving so hard to run a script every time an instance restarts? Can someone please help out here?
If you are using an Amazon EC2 Linux instance to run a batch job on startup, simply install the script in this directory:
/var/lib/cloud/scripts/per-boot/
This will work on any system that has cloud-init installed, which also runs User Data scripts.
For more details, see: Auto-Stop EC2 instances when they finish a task - DEV Community
I am creating a AWS EC2 launch template that includes commands within the User Data field to perform actions when the instance is first launched (package updates, install software, format EBS volumes, etc). In addition to this I also want to perform tasks on reboot or subsequent starting of the instance, such as mounting existing EBS volumes and configuring and mounting volatile SSD volumes. I see that I can use MIME-type to have code run when instance restarts here:
https://aws.amazon.com/premiumsupport/knowledge-center/execute-user-data-ec2/
So, I can clearly modify User Data after I initially launch the instance, but this is cumbersome as it likely needs manual intervention or requires waiting for the instance to have executed the initial User Data code that runs on initialization of the instance.
My question is:
Can the multi-part MIME format be configured to run code that will execute on initialization of the instance and other code that will run every time the instance restarts?
I understand that what you're trying to achieve is passing two sets of commands using Userdata. One set which will be executed on the instance creation and another set which should be executed every reboot. Please lemme know if I misunderstood it.
For the first part, you can use Userdata itself as you already know. For the commands that should run on every reboot, you can leverage rc.local .
The script /etc/rc.local is for use by the system administrator. It is traditionally executed after all the normal system services are started, at the end of the process of switching to a multiuser runlevel, etc. You might use it to start a custom service or for mounting additional Volumes.
To write into /etc/rc.local , you can download the command set from S3 and copy into the file or you can simply echo it. Example:
echo 'echo HelloWorld' >> /etc/rc.local
Hope this helps.
My question is how do you run a boot script on a windows EC2 (similar to this question but for windows rather than linux)
Also, does a 'stopped' instance being restarted count as a 'boot', or must the instance be terminated in order for the script to run. I ask because I would like the script to run whenever a lambda starts a stopped (not terminated) instance
A script can be passed in the User Data property.
If you are using a Windows AMI, and the first line of the script begins with <powershell> or <script> (for normal DOS commands), then the script will be executed the first time that the instance is started.
For details, see: Running Commands on Your Windows Instance at Launch - Amazon Elastic Compute Cloud
Such a script only runs the first time the instance is started. More accurately, it is only once per Instance ID. This means that if you make an AMI of the instance, then any new instances launched from the AMI will run the User Data script.
If you wish a script to run whenever the virtual machine is turned on, then use the capabilities of the operating system. It has nothing to do with the fact that it is an Amazon EC2 instance.
I can see a generic way of starting an EC2 from lambda in Start and Stop Instances at Scheduled Intervals Using Lambda and CloudWatch.
Suppose I use that method to start an EC2, and suppose the AMI is a windows server 2019 customised to have a .bat file on the desktop, and also suppose I'm using a python lambda.
How can I execute this batch file from the lambda? (i.e. just as though someone had RDP'd into the instance and double-clicked on it)
Note: To be very clear, basically I want to start the EC2 using the method given in the AWS docs (above), and right after the instance has started, to run the batch file that will be sitting on the instance's desktop
I think you have a few concepts mixed together.
AWS Lambda functions run on the Lambda service, without having to use Amazon EC2 instances. This is what makes them "serverless".
If you have a batch file on an Amazon EC2 instance, you would presumably want to run that batch file on the EC2 instance itself, without involving Lambda (since you have got a server).
If you wish to run a script on an EC2 instance when it launches for the first time, you can provide a PowerShell or Command-Line script via the User Data field. Software on the AMI will automatically execute this script the first time that the instance starts.
This script could do all the work itself, or it could simply call another script that is stored on the disk. Some people use the script to download another script from a repository (eg Amazon S3 or GitHub) and then execute the downloaded script.
For more information, see: Running Commands on Your Windows Instance at Launch - Amazon Elastic Compute Cloud
If the Amazon EC2 instance is already running and you wish to trigger a script to execute, you can use the AWS Systems Manager Run Command. This works by having an agent on the instance which can be remotely triggered, thereby running scripts without having to login to the instance.
What I need to do is: When a EC2 instance is launched, the lambda function or other installs the script to monitor memory and disk usage in the host.
I'm thinking in how I can do that.. Anyone can give me a idea?
You don't need a lambda. Pass your install script as user data.
See: Running Commands on Your Linux Instance at Launch
It appears that your requirement is to monitor Memory and Disk usage from an Amazon EC2 instance. I will assume that you want to monitor it via Amazon CloudWatch.
Amazon CloudWatch provides default metrics for EC2 instances including CPU utilization, network traffic and disk access. These metrics are visible from the hypervisor. However, CloudWatch cannot see 'inside' the EC2 instance, so it is necessary to run scripts from within the instance to track things like free memory and free disk space. The scripts talk to the operating system to retrieve these metrics, which is why they have to run 'within' the instance.
Some standard monitoring scripts are available for Linux instances: Monitoring Memory and Disk Metrics for Amazon EC2 Linux Instances
You can, of course, write your own scripts to send custom metrics to CloudWatch. Once installed, the scripts will run automatically when the instance is restarted.
If you wish to install these scripts (or your own scripts) on new EC2 instances, there are a couple of methods:
Install the scripts on one instance, then create an Amazon Machine Image (AMI) of that instance that contains a copy of the disk. You can then launch new instances using that AMI, and the scripts will already be installed on the new instances.
Launch the instance(s) with a User Data script to install the monitoring script. Any script passed through User Data will automatically be run the first time that the instance is started.
When you are using a scaling group you must specify a LaunchConfig.
Part of the LaunchConfig is the user-data script which is executed when the instance boots.
This can be also easily done from CloudFormation scripts if that is what you use to create the new EC2 VM.
You can find here samples of scripts.
enter link description here