I am using the command below aws ec2 describe-instances to return list of running EC2s. The output is a TSV in 2 lines, first line being Account ID (OwnerId), and the rest of the query is displayed at the second line. I would like to make all the attributes to be flattened in one line, but due to the nature of the json output of the API calls, is there a way to manipulate the output to be in just one line?
aws ec2 describe-instances --output text --profile $account --query 'Reservations[*].[[OwnerId, Instances[*].[InstanceId, InstanceType, State.Name, Platform, Placement.AvailabilityZone, PublicIpAddress, PrivateIpAddress,[Tags[?Key==Name].Value][0][0],[Tags[?Key==Environment].Value][0][0]]]]' --filter --filters Name=instance-state-name,Values=running >> $outfile
1234567890
i-03cxxxxxxxdab t2.medium running windows ap-southeast-1a 10.0.0.0 10.10.0.10 api-abc-prod-01 PROD
desired output in one line
1234567890 i-03cxxxxxxxdab t2.medium running windows ap-southeast-1a 10.0.0.0 10.10.0.10 api-abc-prod-01 PROD
First thing, AWS cli offer to specify the output formate, so you can set the output to text then replace the new line with space. you can try
aws ec2 describe-instances --output text --profile test --query 'Reservations[*].[[OwnerId, Instances[*].[InstanceId, InstanceType, State.Name, Platform, Placement.AvailabilityZone, PublicIpAddress, PrivateIpAddress,[Tags[?Key==Name].Value][0][0],[Tags[?Key==Environment].Value][0][0]]]]' --filter --filters Name=instance-state-name,Values=running --output text | tr '\r\n' ' '
Text Output Format
The text format organizes the AWS CLI's output into tab-delimited
lines. It works well with traditional Unix text tools such as grep,
sed, and awk, as well as the text processing performed by PowerShell.
The text output format follows the basic structure shown below. The
columns are sorted alphabetically by the corresponding key names of
the underlying JSON object.
IDENTIFIER sorted-column1 sorted-column2
IDENTIFIER2 sorted-column1 sorted-column2
so you are good to go with pipe using tr '\r\n' ' '
Or you can use awk
aws ec2 describe-instances --output text --profile test --query 'Reservations[*].[[OwnerId, Instances[*].[InstanceId, InstanceType, State.Name, Platform, Placement.AvailabilityZone, PublicIpAddress, PrivateIpAddress,[Tags[?Key==Name].Value][0][0],[Tags[?Key==Environment].Value][0][0]]]]' --filter --filters Name=instance-state-name,Values=running | awk 1 ORS=' '
update:
If you want to append ownerID with each instance details then use this.
aws ec2 describe-instances --output text --query 'Reservations[*].[[OwnerId, Instances[*].[InstanceId, InstanceType, State.Name, Platform, Placement.AvailabilityZone, PublicIpAddress, PrivateIpAddress,[Tags[?Key==Name].Value][0][0],[Tags[?Key==Environment].Value][0][0]]]]' --filter --filters Name=instance-state-name,Values=running | paste -d" " - -
As recommended in the top answer in Parsing JSON with Unix tools --> you can use jq to do this https://stedolan.github.io/jq/, check the tutorial here on extracting the relevent fields... https://stedolan.github.io/jq/tutorial/
Related
I am using the aws cli to give me some output which I require in .csv format.
The below shows the commands I've put together:
aws ec2 describe-instances --output text --query 'Reservations[*].Instances[*].[VpcId, SubnetId, InstanceId, InstanceType, ImageId, State.Name, LaunchTime, Placement.AvailabilityZone, Placement.Tenancy, PrivateIpAddress, PrivateDnsName, PublicDnsName,[Tags[?Key==`Name`].Value],[Tags[?Key==`PowerData`].Value] ]' --output text | sed -E 's/\s+/,/g'
For some reason, the final two pieces of info, "Name" and "PowerData" end up on separate lines. This screws up my formatting when I open it up in a spreadsheet.
Anyone have any ideas on what might be causing that or how I may resolve it please?
The reason is TAGs return array so you need to print the only value on 0 index to display the TAGs on a single line.
You can verify this
aws ec2 describe-instances --output json --query 'Reservations[*].Instances[*].Tags[]'
output
[
{
"Key": "Name",
"Value": "demo"
},
{
"Key": "PowerData",
"Value": "demo"
}
]
so you need pipe expression
Tags[?Key==`Name`].Value| [0]
With the addition of filters, we could pass the result of one expression to another, operating on the result of a projection (or any expression).
Expression:
foo.*.bar | [0]
jmespath-pipe-expressions
So try this
aws ec2 describe-instances --output text --query 'Reservations[*].Instances[*].[VpcId, SubnetId, InstanceId, InstanceType, ImageId, State.Name, LaunchTime, Placement.AvailabilityZone, Placement.Tenancy, PrivateIpAddress, PrivateDnsName, PublicDnsName,Tags[?Key==`Name`].Value| [0],Tags[?Key==`PowerData`].Value | [0] ]' --output text | sed -E 's/\s+/,/g'
I need to audit a large number of AWS accounts to determine which EC2 instances are missing the SSM agent. Then I need have all those instances and their tags outputted.
Running aws ssm describe-instance-information lists all the instances that have the agent installed and are running, but it doesn't list instances that are missing the agent or systems that might be turned off.
#!/bin/bash
for instance in $(aws ec2 describe-instances --query 'Reservations[*].Instances[*].[InstanceId]' --output text )
do
managed=$(aws ssm describe-instance-information --filters "Key=InstanceIds,Values=$instance" --query 'InstanceInformationList[*].[AssociationStatus]' --output text)
if [[ "$managed" != "Success" ]]; then
managed="Not Managed";
fi
aws ec2 describe-instances --instance-id $instance --output text --query 'Reservations[*].Instances[*].[InstanceId, Placement.AvailabilityZone, [Tags[?Key==`Name`].Value] [0][0], [Tags[?Key==`App`].Value] [0][0], [Tags[?Key==`Product`].Value] [0][0], [Tags[?Key==`Team`].Value] [0][0] ]'
echo "$managed"
done
Save and make the script executable, then run
script.sh > file.tsv
And finally import it into excel
This will print a list of all your instances with "success" printed beneath the ones which are managed.
for instance in $(aws ec2 describe-instances --query 'Reservations[*].Instances[*].[InstanceId]' --output text )
do;
managed=$(aws ssm describe-instance-information --filters "Key=InstanceIds,Values=$instance" --query 'InstanceInformationList[*].[AssociationStatus]' --output text)
echo "$instance $managed";
done
To add a simple but not well-formatted set of tags, replace the echo line with
if [[ "$managed" != "Success" ]]; then
managed="Fail";
fi
echo "$instance $managed"
aws --profile GC-Staging ec2 describe-instances --instance-id $instance --query 'Reservations[*].Instances[*].[Tags[*].Value]' --output text
I am looking forward to call \File.txt using below AWS CLI , if you have exp please share it.
aws ec2 describe-instances --filters Name=network-interface.group-name,Values='\file.txt' --query 'Reservations[].Instances[].[Placement.AvailabilityZone, State.Name, InstanceId,Tags[?Key==Name].Value|[0]]'
File.txt includes Security Group Names.
Here is a way you can do it, I'm assuming Security Group Name is on a separate line.
Command:
$(cat file.txt|sed 'N;s/\n/,/')
Input: file.txt:
cat file.txt
security-group-name-1
security-group-name-2
security-group-name-3
Output:
security-group-name-1,security-group-name-2
Full command:
aws ec2 describe-instances --filters Name=network-interface.group-name,Values=$(cat file.txt|sed 'N;s/\n/,/') --query 'Reservations[].Instances[].[Placement.AvailabilityZone, State.Name, InstanceId,Tags[?Key==Name].Value|[0]]'
With this command I am getting the snapshot id but i want to get the name associated with this id [TAGS]
aws ec2 describe-snapshots --owner-ids ********** --output text | awk /vol-6ac16d63/ | grep -y "2014-02-01" | awk "/vol-6ac16d63/{print \$5}"
try describe-tags:
aws ec2 describe-tags --filters Name=resource-id,Values=snap-XXXXXXX Name=key,Values=Name --query "Tags[*].Value" --output text
The aws cli has a --query option, which allows you to select only some information.
For an example, I am interested in getting just the Security group name from ec2 describe-instances.
If I run:
aws ec2 describe-instances --output text --query 'Reservations[*].Instances[*].[InstanceId,InstanceType,SecurityGroups]
my output looks like:
i-xxxxxxx m1.type [{u'GroupName': 'groupName', u'GroupId': 'sg-xxxxx'}]
I can also access elements of the list using an index:
aws ec2 describe-instances --output text --query 'Reservations[*].Instances[*].[InstanceId,InstanceType,Tags[0].Value,Tags[0].Name]
Is it possible to query tags so that instead of Tag[0] I search for a Tag where the name is specified?
As of 1.3.0, you can now query that information like this:
--query 'Reservations[*].Instances[*].Tags[?Key==`<keyname>`].Value[]'
So where you have this:
"Tags" : [
{
"Value" : "webserver01",
"Key" : "InstanceName"
},
you'd want to do this:
aws ec2 describe-instances --query 'Reservations[*].Instances[*].Tags[?Key==`InstanceName`].Value[]'
What you probably want to use is the --filters option:
aws ec2 describe-instances --output text --filters "Name=tag-key, Values=SecurityGroups, Name=tag-value, Values=Foo" --region us-east-1
You can change the filters around to "query" for the exact field you're looking for.
checkout this slideshare from the Atlanta AWS meetup group's talk on the new AWS CLI for more examples
This way works for me: (this only works in version 1.3.0 and above)
aws ec2 describe-instances --query 'Reservations[*].Instances[*].[InstanceId, Tags[?Key==`Name`].Value[*]]'
select security_groups from aws.aws_ec2_instance;
> select security_groups from aws.aws_ec2_instance limit 1;
+---------------------------------------------------------------------------------------------------------------------------------+
| security_groups |
+---------------------------------------------------------------------------------------------------------------------------------+
| [{"GroupId":"sg-xxxx","GroupName":"xxxx"},{"GroupId":"sg-xxxxxx","GroupName":"xxxx"}] |
+---------------------------------------------------------------------------------------------------------------------------------+
This will just list out the security groups for your instances.
You can also use
select security_groups from aws.aws_ec2_instance where instance_id = 'i-xxxxxx';