I need to list EBS VolumeID and the instance that it's attached to using the aws cli. This is the line I used:
aws ec2 describe-volumes --output text --query 'Volumes[*].{VolumeID:VolumeId, Instance:InstanceId}' | head -5
None vol-07210e47
None vol-743d1234
None vol-933d12d3
None vol-493c1309
None vol-1e3b145e
For some reason the instance IDs are showing as none. When the unfiltered output of the command shows that they're there:
aws ec2 describe-volumes | head -25
{
"Volumes": [
{
"AvailabilityZone": "us-east-1d",
"Attachments": [
{
"AttachTime": "2013-09-05T15:17:39.000Z",
"InstanceId": "i-c28e20ae",
"VolumeId": "vol-07210e47",
"State": "attached",
"DeleteOnTermination": false,
"Device": "/dev/sda1"
}
],
What am I doing wrong?
You're not querying into Attachments. This worked for me:
aws ec2 describe-volumes --output text --query 'Volumes[*].Attachments[].{VolumeID:VolumeId,InstanceID:InstanceId}'
This is a good link:
https://docs.aws.amazon.com/cli/latest/userguide/controlling-output.html
Related
I have a script that fetches list of instances having tag x having abc value. The count of ec2 instances returned are in hundreds, now for each instance I need to fetch 2 tag values. Not all instances will have both the tags, it could be 1 or both or none. For now I am issuing 2 calls to get the value of each tag (this is a bash shell)
market=`aws ec2 describe-tags --filters "Name=resource-id,Values=$id" "Name=key,Values=market" --query Tags[].Value --region $aws_region --output text`
service=`aws ec2 describe-tags --filters "Name=resource-id,Values=$id" "Name=key,Values=service" --query Tags[].Value --region $aws_region --output text`
Is there any way to fetch the values of both tags in a single call?
I have 4 instances like this:
i-020f43a6253e1dd25 tags:market=1
i-0a5c4b42fe3e75c15 tags:service=1
i-027ca3de0fe11f1d3 tags:market=4,service=4
i-0e77b17601f9b2fd2 tags:none
Server side filtering using --filters returns 4 matching records
% aws ec2 describe-tags --filters "Name=key,Values=market,service"
{
"Tags": [
{
"Key": "market",
"ResourceId": "i-020f43a6253e1dd25",
"ResourceType": "instance",
"Value": "1"
},
{
"Key": "market",
"ResourceId": "i-027ca3de0fe11f1d3",
"ResourceType": "instance",
"Value": "4"
},
{
"Key": "service",
"ResourceId": "i-027ca3de0fe11f1d3",
"ResourceType": "instance",
"Value": "4"
},
{
"Key": "service",
"ResourceId": "i-0a5c4b42fe3e75c15",
"ResourceType": "instance",
"Value": "1"
}
]
}
I want to export the list displayed in the Volume section of the EC2 Console into CSV (or at least text) and I am trying to use the CLI for that.
I have trouble getting the tags information right using Filter Expression:
me#home:/$ aws ec2 describe-volumes --query 'Volumes[*].Tags[?Key=='Name'].Value'
[
[],
[],
[],
//...
[]
]
Here I expected to have a list with sometimes the value of the tag Name for the columes that have it and [] for those who don't.
Just to be sure, I checked and I do have Tags with that key (not always, though):
me#home:/$ aws ec2 describe-volumes --query 'Volumes[*].Tags[*].Key'
[
[
"kubernetes.io/created-for/pvc/name",
"Team",
"env",
"kubernetes.io/created-for/pvc/namespace",
"Name",
"kubernetes.io/cluster/DW",
"Software",
"Env",
"kubernetes.io/created-for/pv/name"
],
[
"env",
"Software",
"eks:nodegroup-name",
"Team",
"eks:cluster-name"
],
[
"Name",
"Software",
"Team",
"kubernetes.io/created-for/pvc/name",
"kubernetes.io/cluster/DW",
"kubernetes.io/created-for/pv/name",
"env",
"Env",
"kubernetes.io/created-for/pvc/namespace"
],
//...
]
So, how can I get that information right ?
One thing is that you are breaking the shell quotes with the ones inside your JMESPath expression.
You should be using a literal expression, instead
aws ec2 describe-volumes --query 'Volumes[*].Tags[?Key==`Name`].Value'
or using double quotes for the shell query option value
aws ec2 describe-volumes --query "Volumes[*].Tags[?Key=='Name'].Value"
Hi I would like to pipe an instance output to start/stop ec2 instances. Here is the beginning of the code:
aws ec2 describe-instances \
--query "Reservations[*].Instances[*].{Instance:InstanceId,Subnet:SubnetId}" \
--filters "Name=tag-value,Values=<INSTANCE NAME TAG>" \
--output text | \
How do I pipe this output to AWS ec2 start-instances command in Windows?
Output format can be JSON,YAML, TEXT or Table.It depends on your requirements.
Sample command for JSON output:
$ aws iam list-users --output json
Sample output:
{
"Users": [
{
"Path": "/",
"UserName": "Admin",
"UserId": "AIDA1111111111EXAMPLE",
"Arn": "arn:aws:iam::123456789012:user/Admin",
"CreateDate": "2014-10-16T16:03:09+00:00",
"PasswordLastUsed": "2016-06-03T18:37:29+00:00"
},
{
"Path": "/backup/",
"UserName": "backup-user",
"UserId": "AIDA2222222222EXAMPLE",
"Arn": "arn:aws:iam::123456789012:user/backup/backup-user",
"CreateDate": "2019-09-17T19:30:40+00:00"
},
{
"Path": "/",
"UserName": "cli-user",
"UserId": "AIDA3333333333EXAMPLE",
"Arn": "arn:aws:iam::123456789012:user/cli-user",
"CreateDate": "2019-09-17T19:11:39+00:00"
}
]
}
Now if you want to use this output for input of another command, one easy way is to read the json file, extract the value and use that as input to other command.
Please read https://www.business.com/articles/using-powershell-with-json-data/ for some details.
I found a PowerShell solution which suits my needs better:
$InstanceId = aws ec2 describe-instances --query "Reservations[*].Instances[*].{Instance:InstanceId}" --filters "Name=tag-value,Values=<INSTANCE NAME TAG>" --output text aws ec2 start-instances --instance-ids $InstanceId
I'm a beginner in using AWS.
I just want to stop and start several EC2 instances automatically and periodically(Not reboot).
Is there any recommended way to do this?
Amazon recently (Feb 2018) released the EC2 instance scheduler tool:
The AWS Instance Scheduler is a simple AWS-provided solution that
enables customers to easily configure custom start and stop schedules
for their Amazon Elastic Compute Cloud (Amazon EC2) and Amazon
Relational Database Service (Amazon RDS) instances. The solution is
easy to deploy and can help reduce operational costs for both
development and production environments. Customers who use this
solution to run instances during regular business hours can save up to
70% compared to running those instances 24 hours a day.
I had this up and running in my account in 15 minutes; very simple to use, and practically free.
https://aws.amazon.com/answers/infrastructure-management/instance-scheduler/
AWS has a good doc explaining how you can achieve this using Lambda and Cloudwatch events. You can refer it - https://aws.amazon.com/premiumsupport/knowledge-center/start-stop-lambda-cloudwatch/
This solution can be modified to get the EC2 list dynamically, or operate on a set of instances which can be identified based on a certain tag.
Yes, you can do that using AWS Lambda. You can select the trigger in Cloudwatch which runs on Cron expressions on UTC.
Here is a related link https://aws.amazon.com/premiumsupport/knowledge-center/start-stop-lambda-cloudwatch/
Another alternative is to use awscli which is available from pip, apt-get, yum or brew, and then running aws configure with your credentials from IAM and executing the following bash script, to stop an EC2 that has been tagged with Name: Appname and Value: Appname Prod. You can use awscli to tag your instances or tag it manually from the AWS console. aws ec2 stop-instances will stop the instance and jq is used to filter the json query and fetch the correct instance id using the tags from aws ec2 describe-instances.
To verify that aws configure was successful and returns json output run aws ec2 describe-instances and your running instance id should be there in the output. Here is a sample output
{
"Reservations": [
{
"Instances": [
{
"Monitoring": {
"State": "disabled"
},
"PublicDnsName": "ec2-xxx.ap-south-1.compute.amazonaws.com",
"State": {
"Code": xx,
"Name": "running"
},
"EbsOptimized": false,
"LaunchTime": "20xx-xx-xxTxx:16:xx.000Z",
"PublicIpAddress": "xx.127.24.xxx",
"PrivateIpAddress": "xxx.31.3.xxx",
"ProductCodes": [],
"VpcId": "vpc-aaxxxxx",
"StateTransitionReason": "",
"InstanceId": "i-xxxxxxxx",
"ImageId": "ami-xxxxxxx",
"PrivateDnsName": "ip-xxxx.ap-south-1.compute.internal",
"KeyName": "node",
"SecurityGroups": [
{
"GroupName": "xxxxxx",
"GroupId": "sg-xxxx"
}
],
"ClientToken": "",
"SubnetId": "subnet-xxxx",
"InstanceType": "t2.xxxxx",
"NetworkInterfaces": [
{
"Status": "in-use",
"MacAddress": "0x:xx:xx:xx:xx:xx",
"SourceDestCheck": true,
"VpcId": "vpc-xxxxxx",
"Description": "",
"NetworkInterfaceId": "eni-xxxx",
"PrivateIpAddresses": [
{
"PrivateDnsName": "ip-xx.ap-south-1.compute.internal",
"PrivateIpAddress": "xx.31.3.xxx",
"Primary": true,
"Association": {
"PublicIp": "xx.127.24.xxx",
"PublicDnsName": "ec2-xx.ap-south-1.compute.amazonaws.com",
"IpOwnerId": "xxxxx"
}
}
],
"PrivateDnsName": "ip-xxx-31-3-xxx.ap-south-1.compute.internal",
"Attachment": {
"Status": "attached",
"DeviceIndex": 0,
"DeleteOnTermination": true,
"AttachmentId": "xxx",
"AttachTime": "20xx-xx-30Txx:16:xx.000Z"
},
"Groups": [
{
"GroupName": "xxxx",
"GroupId": "sg-xxxxx"
}
],
"Ipv6Addresses": [],
"OwnerId": "xxxx",
"PrivateIpAddress": "xx.xx.xx.xxx",
"SubnetId": "subnet-xx",
"Association": {
"PublicIp": "xx.xx.xx.xxx",
"PublicDnsName": "ec2-xx.ap-south-1.compute.amazonaws.com",
"IpOwnerId": "xxxx"
}
}
],
"SourceDestCheck": true,
"Placement": {
"Tenancy": "default",
"GroupName": "",
"AvailabilityZone": "xx"
},
"Hypervisor": "xxx",
"BlockDeviceMappings": [
{
"DeviceName": "/dev/xxx",
"Ebs": {
"Status": "attached",
"DeleteOnTermination": true,
"VolumeId": "vol-xxx",
"AttachTime": "20xxx-xx-xxTxx:16:xx.000Z"
}
}
],
"Architecture": "x86_64",
"RootDeviceType": "ebs",
"RootDeviceName": "/dev/xxx",
"VirtualizationType": "xxx",
"Tags": [
{
"Value": "xxxx centxx",
"Key": "Name"
}
],
"AmiLaunchIndex": 0
}
],
"ReservationId": "r-xxxx",
"Groups": [],
"OwnerId": "xxxxx"
}
]
}
The following bash script is stop-ec2.sh in /home/centos/cron-scripts/
(instance=$(aws ec2 describe-instances | jq '.Reservations[].Instances | select(.[].Tags[].Value | startswith("Appname Prod") ) | select(.[].Tags[].Key == "Appname") | {InstanceId: .[].InstanceId, PublicDnsName: .[].PublicDnsName, State: .[].State, LaunchTime: .[].LaunchTime, Tags: .[].Tags} | [.]' | jq -r .[].InstanceId) && aws ec2 stop-instances --instance-ids ${instance} )
Run the file using sh /home/centos/cron-scripts/stop-ec2.sh and verify that the EC2 instance gets stopped. To debug run aws ec2 describe-instances | jq '.Reservations[].Instances | select(.[].Tags[].Value | startswith("Appname Prod") ) | select(.[].Tags[].Key == "Appname") | {InstanceId: .[].InstanceId, PublicDnsName: .[].PublicDnsName, State: .[].State, LaunchTime: .[].LaunchTime, Tags: .[].Tags} | [.]' | jq -r .[].InstanceId and see that it returns the correct instance ID which has been tagged.
Then in crontab -e the following line can be added
30 14 * * * sh /home/centos/cron-scripts/stop-ec2.sh >> /tmp/stop
which will log the output to /tmp/stop. The 30 14 * * * is the UTC cron expression that you can check in https://crontab.guru/
Lambda script to stop instance:
import json
import boto3
# Enter the region your instances are in. Include only the region without specifying Availability Zone; e.g., 'us-east-1'
region = 'us-east-1'
def lambda_handler(event, context):
ec2 = boto3.client('ec2', region_name=region)
filter = [{'Name': 'tag:Name', 'Values': ['****-env']}] //give instance name here in place of ****-env
instances = ec2.describe_instances(Filters=filter)
#ec2.stop_instances(InstanceIds=instances)
stop_instance = instances.get('Reservations')[0].get('Instances')[0].get('InstanceId')
stop_instances = []
stop_instances.append(stop_instance)
ec2.stop_instances(InstanceIds=stop_instances)
Lambda script to start instance:
import json
import boto3
# Enter the region your instances are in. Include only the region without specifying Availability Zone; e.g., 'us-east-1'
region = 'us-east-1'
def lambda_handler(event, context):
ec2 = boto3.client('ec2', region_name=region)
filter = [{'Name': 'tag:Name', 'Values': ['****-env']}]
instances = ec2.describe_instances(Filters=filter)
#ec2.stop_instances(InstanceIds=instances)
start_instance = instances.get('Reservations')[0].get('Instances')[0].get('InstanceId')
start_instances = []
start_instances.append(start_instance)
ec2.start_instances(InstanceIds=start_instances)
ASG scheduler is the best and easiest option to mange the EC2 instance if you are using ASG. If not using ASG, then you can use either AWS instance scheduler CF solution or lambda with cloudwatch Cron event.
I want to run an ansible role, for all the ip address in the vpc which are running.
How to get all the ip address of running instance in given vpc
Things: I have tired:
aws ec2 describe-instances --filters "Name=vpc-id,
Values="vpc-******"" --query
"Reservations[].Instances[].PrivateIpAddresses[*]" --output text
This is returning null
The name of the parameter is PrivateIpAddress not PrivateIpAddresses as you can see from Json object
[
[
{
"Monitoring": {
"State": "disabled"
},
"PublicDnsName": "xxxx",
"RootDeviceType": "ebs",
"State": {
"Code": 16,
"Name": "running"
},
"EbsOptimized": false,
"LaunchTime": "xxx",
"PublicIpAddress": "xxx",
"PrivateIpAddress": "xxxxx",
"ProductCodes": [
....
so if you run your command as
aws ec2 describe-instances --filters "Name=vpc-id, Values="vpc-cda7c6a8"" --query "Reservations[*].Instances[*].PrivateIpAddress" --output text
you will have your expected result
it's PrivateIPAddress, not Addresses
aws ec2 describe-instances --instance-ids --query Reservations[].Instances[].PrivateIpAddress
Hope this helps