I am trying to set up some easy code to run when trying to spin up an EMR for some ad hoc work I have to do, time to time.
Right now I pass the 'aws emr create-cluster' command and then find the DNS in the console, once the cluster is created to then use ssh to connect.
I'd like to skip having to open the console at all, and use the cluster ID to get the DNS value to create my SSH connection, but I am not seeing a clear command to do this with. I'm new to CLI so I imagine this is a simple task I am merely failing at figuring out myself.
In my mind the solution should be something along the lines of
aws emr create-cluster [config for cluster here] > file.txt
set DNS = aws emr describe-cluster --cluster-id file.txt -MasterPublicDnsName
ssh -i Desktop/AWS/EMRKey.pem -o ServerAliveInterval=15 hadoop#$DNS
probably will have to append 'hadoop#' to the DNS variable before passing it into a command, but I'm more curious at the moment to if the above makes any functional sense, and if so, how I can get the functionality of the describe-cluster command to output the -MasterPublicDnsName, as that is obviously just something I made up and not an actual option that I have found.
The AWS CLI has a query option that lets you query the output of a command. You'll also want to use a waiter to make sure the instance is up before you try to connect to it.
You could simply run
cluster_id="j-2RNBSZZBLXTZ0"
aws emr wait cluster-running --cluster-id $cluster_id
hostname=`aws emr describe-cluster --output text --cluster-id $cluster_id --query Cluster.MasterPublicDnsName`
ssh hadoop#$hostname
That should work!
Related
I want to test my Python library in GPU machine once a day.
I decided to use AWS EC2 for testing.
However, the fee of gpu machine is very high, so I want to stop the instance after the test ends.
Thus, I want to do the followings once a day automatically
Start EC2 instance (which is setup manually)
Execute command (test -> push logs to S3)
Stop EC2 (not remove)
How to do this?
It is very simple...
Run script on startup
To run a script automatically when the instance starts (every time it starts, not just the first time), put your script in this directory:
/var/lib/cloud/scripts/per-boot/
Stop instance when test has finished
Simply issue a shutdown command to the operating system at the end of your script:
sudo shutdown now -h
You can push script logs to custom coudwatch namespaces. Like when the process ends publish a state to cloudwatch. In cloudwatch create alarms based on the state of process, so if it has a completed state trigger an AWS lambda function that will stop instance after completion of your job.
Also if you want to start and stop on specific time you can use ec2 instance scheduler to start/stop instances. It just works like a cron job at specific intervals.
You can use the aws cli
To start an instance you would do the following
aws ec2 start-instances --instance-ids i-1234567890abcdef0
and to stop the instance you would do the following
aws ec2 stop-instances --instance-ids i-1234567890abcdef0
To execute commands inside the machine, you will need to ssh into it and run the commands that you need, then you can use the aws cli to upload files to s3
aws s3 cp test.txt s3://mybucket/test2.txt
I suggest reading the aws cli documentation, you will find most if not all what you need to automate aws commands there.
I created a shell script to start an EC2 instance -if not already running,- connect via SSH and, if you want, run a command.
https://gist.github.com/jotaelesalinas/396812f821785f76e5e36cf928777a12
You can use it in three different ways:
./ec2-start-and-ssh.sh -i <instance id> -s
will show status information about your instance: running state and private and public IP addresses.
./ec2-start-and-ssh.sh -i <instance id>
will connect and leave you inside the default shell.
./ec2-start-and-ssh.sh -i <instance id> <command>
will run whatever command you specify, e.g.:
./ec2-start-and-ssh.sh -i <instance id> ./run.sh
./ec2-start-and-ssh.sh -i <instance id> sudo poweroff
I use the last two commands to run periodic jobs minimizing billing costs.
I hope this helps!
I'm trying to get into the reporting of AWS instances within my environments, and I am trying to create a script using AWS cli to generate a report of the status of EC2 instances. I'm still a beginner, so I'm learning how all this works. I was wondering if it was possible to do with a line similar to this:
AWS EC2 describe-instances --region $REGION --query 'Reservations[].Instances[?LaunchTime>=`2015-03-01`][].{id: InstanceId, type: InstanceType, launched: LaunchTime}'
I am using the Cygwin Terminal to perform this query, and any with this would be awesome thanks!
If it's possible I am also trying to see if I could get the CPU usage at that time of the report. I am still trying to understand how all this works since I am new to the AWS API.
Firstly, you will need to install the aws-sdk. Checkout the following answer for doing this in cygwin.
Then you can configure your account using aws configure and following the prompts.
After that you should be able to run aws ec2 describe-instances.
Thanks for everyone's input! I was able to figure it out here. I'm starting to understand how the population output all comes together, but right now I'm trying to put this information into at TSV file. To get the status and launch time of the instances I used the following:
aws ec2 describe-instances --region $REGION --query "Reservations[].Instances[].[InstanceId, LaunchTime, State.Name] --output text >> Instances_In_AWS.tsv
Thank you guys for your help!
I am trying to come up with a script to automate the setting up of desired count of AutoScalingGroups based on some kind of profiles e.g., SHUTDOWN profile should set everything to zero.
We have lot of applications under single account itself. So when running below command, it gives all the resources.
aws ecs list-clusters
Is there a way to filter these by either tags or any other means? Apparently --filter is not a valid option for aws ecs or aws autoscaling commands.
I am utilizing the grep command for now.
aws ecs list-clusters | grep string1 | grep string2
Not sure that's exactly what you're asking, but if you want to play with the JSON output of these commands (or filter/transform any JSON string in general), there's no better tool than jq. Takes some time to get into, but this tool might become your best friend.
Once installed, you can issue commands such as:
aws ecs describe-clusters|jq -r '.clusters[]|{clusterName, status}'
To create a cluster name/status list from the info.
aws ecs describe-clusters|jq -r '.clusters[]|if .status == "INACTIVE" then .clusterArn else null end'
To list all the inactive clusters.
Add a delete command this way to delete all the inactive clusters (don't run it!!!):
aws ecs describe-clusters|jq -r '.clusters[]|if .status == "INACTIVE" then .clusterArn else null end'|xargs aws ecs delete-clusters --clusters
I have only one cluster at disposal, I didn't test if these commands still work with many clusters (JSON tables properly parsed), but you get the idea...
jq tutorial: https://stedolan.github.io/jq/tutorial/
Hi Guys I need to do a script which copies the ips from an aws region AMIs that I have running, after copying this IPs place them into a text file inside. If the Instances are turned off, that IP would get removed, and the text file would change real-time, on it's own automatically, I need this to run across all regions, so any Instance that I have "X" AMI running with, the script would find it, copy its IP keep it if it's running and remove it from the file if they switch to shutdown mode.
stack the IPs in a text like
55.555.555.55
66.123.545.54
.....
.....
real-time.
I've never really used aws cli and I know this is possible to do.
Use the describe-instances command in the AWS CLI. All the information that you need (AMI, instance state, IP address) will be included in the response to that command. Note that you will have to run describe-instances once for each region. (Set the --region flag when running the CLI to set the region.)
You can parse the JSON output of the CLI however you want then write the information you want to the text file.
This command uses a aws cli "describe-instances" command with a filter for only instances that are running.
This outputs a lot of data including the "PublicIp" field. The sed command strips out just the ip address from that line and the uniq removes duplicates
aws ec2 describe-instances --filters 'Name=instance-state-name,Values=running' | sed -n 's/^.*"PublicIp": "\([0-9\.]*\)\",/\1/p'| uniq
See http://docs.aws.amazon.com/cli/latest/reference/ec2/describe-instances.html for details on the aws cli describe instances command, including other filters you might want to apply
I am very new to AWS. I've got a Windows instance running and have my aws command line configured. I've read through the AWS docs but can't seem to find exactly what I'm looking for.
How do I view my current instances from the command line?
If by view your current instances, you mean list all running instances from the command line, you can call the describe-instances command:
aws ec2 describe-instances
This will list all of your current instances.
See describe-instances
As noted in the answer by Rodrigo M, you should use describe-instances to view your EC2 instances. In general, the help command is the best way to explore the CLI. Start with aws ec2 help and try the various options. You can get more details on subcommands with aws ec2 describe-instances help as well.
The output is a bit verbose and by default in JSON. This can be a bit overwhelming and hard to read without additional processing. I'd recommend getting familiar with the --query aws CLI parameter if you intend to use the CLI interactively.
In particular, I use this for a quick overview of my EC2 instances:
aws ec2 describe-instances --query 'Reservations[].Instances[].[InstanceId, Tags[?Key==`Name`] | [0].Value, State.Name, PublicDnsName]' --output table
To check one particular attribute on an instance:
aws ec2 describe-instances --query Reservations[0].Instances[0].InstanceType --output text --instance-ids <my-instance-id>
The CLI is very powerful once you get comfortable with learning the commands and managing the output. It's also helpful for learning the porgramming APIs as well, since aws CLI commands generally map one-to-one with an API call.