Need arn values from aws configservice - amazon-web-services

I am running
aws configservice get-compliance-details-by-config-rule --config-rule-name required-tags --compliance-types NON_COMPLIANT
with output
{
"EvaluationResults": [
{
"EvaluationResultIdentifier": {
"EvaluationResultQualifier": {
"ConfigRuleName": "required-tags",
"ResourceType": "AWS::ACM::Certificate",
"ResourceId": "arn:aws:acm:us-east-1:***:certificate/d9863cca-9e7b-460b-b9f8-bee23e8fb607"
},
"OrderingTimestamp": "2022-08-10T12:46:18.247000+05:30"
},
"ComplianceType": "NON_COMPLIANT",
"ResultRecordedTime": "2022-08-10T13:12:00.037000+05:30",
"ConfigRuleInvokedTime": "2022-08-10T13:11:59.841000+05:30"
},
{
"EvaluationResultIdentifier": {
"EvaluationResultQualifier": {
"ConfigRuleName": "required-tags",
"ResourceType": "AWS::EC2::Instance",
"ResourceId": "i-069c8d8c72ae8db8c"
},
"OrderingTimestamp": "2022-08-10T12:46:18.784000+05:30"
},
"ComplianceType": "NON_COMPLIANT",
"ResultRecordedTime": "2022-08-10T13:11:54.648000+05:30",
"ConfigRuleInvokedTime": "2022-08-10T13:11:54.449000+05:30
I need arn names of all the resources under the rule
and need to run
aws tag-resources
--resource-arn-list <value>
--tags <value>
[--cli-input-json <value>]
[--generate-cli-skeleton <value>]
THe problem is the the aws configservice get-compliance-details-by-config-rule command doen't list the arn of the resources and i need arn of each resource to be tagged.
What can i do?

You can construct Amazon EC2 instances ARNs using aws ec2 describe-instances command line. Please note that I used jq for that so you will need it if you want to use this method.
You can aws ec2 describe-instances with a filter by instance-id. In this case the instance-id corresponds to the ResourceId in your output. It should return the all data regarding the specified instance.
Your command line should look like aws ec2 describe-instances --region eu-west-3 --instance-id i-abd123.
Then you can format the result using jq.
aws ec2 describe-instances --region us-east-1 --instance-id i-abc123 | jq -r '.Reservations[] | .OwnerId as $OwnerId | ( .Instances[] | { "ARN": "arn:aws:ec2:\(.Placement.AvailabilityZone[:-1]):\($OwnerId):instance/\(.InstanceId)"} )' | jq -s .
The output should look like this:
[
{
"ARN": "arn:aws:ec2:us-east-1:***:instance/i-abc123"
}
]

Related

AWS CLI query and filter for Step Functions

I would like to get Step Function ARN using AWS CLI by name with wildcard strig or get Step Function ARN by Step Function name
Here is an example:
aws stepfunctions list-state-machines --region us-east-1
I got this:
{
"stateMachines": [
{
"stateMachineArn": "arn:aws:states:us-east-1:012345678912:stateMachine:firstStepFunc",
"name": "firstStepFunc",
"type": "STANDARD",
"creationDate": "2022-12-01T14:43:09.577000+01:00"
}
]
}
I tried this one:
aws stepfunctions list-state-machines --query 'stateMachines[*].stateMachineArn' --region us-east-1 --output text
And get expected result:
arn:aws:states:us-east-1:012345678912:stateMachine:firstStepFunc
But if Step Functions will be more than one, it won't work.
I need something like that, but I have no idea how to write query in proper way:
aws stepfunctions list-state-machines --query 'stateMachines[*].stateMachineArn[?stateMachineArn==`*`]' --region us-east-1
aws stepfunctions list-state-machines --query 'stateMachines[*].stateMachineArn[?name==`*`]' --region us-east-1
Thanks in advance!
You could use contains functions for this, for example:
aws stepfunctions list-state-machines --query 'stateMachines[?contains(name,`dev`)]|[*].stateMachineArn' --region us-east-1 --output text
The expression above returns the ARN of all stepfunctions which have the dev keyword in their name. If you want to get only one (the first one, for example), you can do the following:
aws stepfunctions list-state-machines --query 'stateMachines[?contains(name,`dev`)]|[0].stateMachineArn' --region us-east-1 --output text

Getting VPC-ID for specific VPC

after the VPC is created, how I can grep only 1 VPC-ID from specific VPC by aws ec2 describe-vpcs, so that VPC ID can be passed inside the script for the next step, I know I can see it manually from that command or from AWS console,
for example:
aws ec2 describe-vpcs --vpc-ids |grep VpcId
"VpcId": "vpc-00a0338c2f671a77c",
"VpcId": "vpc-0b3697513d5987516",
"VpcId": "vpc-061e25f5f78877798",
it gives me all of them, or:
aws ec2 describe-vpcs --vpc-ids |grep -i ansible
"Value": "ANSIBLE_VPC",
but I need only to get the VPC-ID for that specific VPC from command.
If you just issued a create-vpc command, then the VPC ID of that VPC would have been returned in response to that command:
Output:
{
"Vpc": {
"CidrBlock": "10.0.0.0/16",
"DhcpOptionsId": "dopt-5EXAMPLE",
"State": "pending",
"VpcId": "vpc-0a60eb65b4EXAMPLE", <-- This is the VPC ID
"OwnerId": "123456789012",
"InstanceTenancy": "default",
"Ipv6CidrBlockAssociationSet": [],
"CidrBlockAssociationSet": [
{
"AssociationId": "vpc-cidr-assoc-07501b79ecEXAMPLE",
"CidrBlock": "10.0.0.0/16",
"CidrBlockState": {
"State": "associated"
}
}
],
"IsDefault": false,
"Tags": []
}
}
Thus, you could create the VPC and store its ID like this:
$ ID=`aws ec2 create-vpc --cidr-block 10.0.0.0/16 --query Vpc.VpcId --output text`
$ echo $ID
vpc-0fb4d08f9d6501e94
If, instead, you are seeking the VPC ID for a VPC given its Name tag, you could use:
$ ID=`aws ec2 describe-vpcs --filter Name=tag:Name,Values=ANSIBLE_VPC --query Vpcs[].VpcId --output text`
$ echo $ID
vpc-0fb4d08f9d6501e94
You can use just the aws cli for this, with filters and query:
aws ec2 describe-vpcs --filters Name=tag:Name,Values=ANSIBLE_VPC --query "Vpcs[].VpcId" --output text"
Or you can use a mix of the --filters command with grep to accomplish your task:
aws ec2 describe-vpcs --filters Name=tag:Name,Values=ANSIBLE_VPC | grep VpcId | grep -oh "vpc-\w*"

AWS CLI EMR get Master node Instance ID and tag it

I want to automate the running of a cluster and can use tags to get attributes of an EC2 instance like its instance-id.
The documentation on https://docs.aws.amazon.com/cli/latest/reference/emr/create-cluster.html states that
--tags (list)
A list of tags to associate with a cluster, which apply to each Amazon
EC2 instance in the cluster. Tags are key-value pairs that consist of
a required key string with a maximum of 128 characters, and an
optional value string with a maximum of 256 characters.
You can specify tags in key=value format or you can add a tag without
a value using only the key name, for example key . Use a space to
separate multiple tags.
So this applies tags to every EC2 instance including the master and slaves. How do I discern which instance is the master node?
Additional Info:
I am using the following command to get attributes from aws cli based on tags where you can replace the "Name" and "Prod" with your tags key-value pairs respectively.
aws ec2 describe-instances | jq '.Reservations[].Instances | select(.[].Tags[].Value | startswith("Prod") ) | select(.[].Tags[].Key == "Name") | {InstanceId: .[].InstanceId, PublicDnsName: .[].PublicDnsName, State: .[].State, LaunchTime: .[].LaunchTime, Tags: .[].Tags} | [.]' | jq .[].InstanceId
As you noted when you create an EMR cluster, the tags are the same for all nodes (Master, Slave, Task).
You will find that this process using the AWS CLI to be complicated. My recomendation is to review the examples below and then write a Python program to do this.
Process to add your own tags to the EC2 instances.
STEP 1: List your EMR Clusters:
aws emr list-clusters
This will output JSON:
{
"Clusters": [
{
"Id": "j-ABCDEFGHIJKLM",
"Name": "'MyCluster'",
"Status": {
"State": "WAITING",
"StateChangeReason": {
"Message": "Cluster ready after last step completed."
},
"Timeline": {
"CreationDateTime": 1536626095.303,
"ReadyDateTime": 1536626568.482
}
},
"NormalizedInstanceHours": 0
}
]
}
STEP 2: Make a note of the Cluster ID from the JSON:
"Id": "j-ABCDEFGHIJKLM",
STEP 3: Describe your EMR Cluster:
aws emr describe-cluster --cluster-id j-ABCDEFGHIJKLM
This will output JSON (I have truncated this output to just the MASTER section):
{
"Cluster": {
"Id": "j-ABCDEFGHIJKLM",
"Name": "'Test01'",
....
"InstanceGroups": [
{
"Id": "ig-2EHOYXFABCDEF",
"Name": "Master Instance Group",
"Market": "ON_DEMAND",
"InstanceGroupType": "MASTER",
"InstanceType": "m3.xlarge",
"RequestedInstanceCount": 1,
"RunningInstanceCount": 1,
"Status": {
"State": "RUNNING",
"StateChangeReason": {
"Message": ""
},
"Timeline": {
"CreationDateTime": 1536626095.316,
"ReadyDateTime": 1536626533.886
}
},
"Configurations": [],
"EbsBlockDevices": [],
"ShrinkPolicy": {}
},
....
]
}
}
STEP 4: InstanceGroups is an array. Find the entry where InstanceGroupType is MASTER. Make note of the Id.
"Id": "ig-2EHOYXFABCDEF",
STEP 5: List your cluster instances:
aws emr list-instances --cluster-id j-ABCDEFGHIJKLM
This will output JSON (I have truncated the output):
{
"Instances": [
....
{
"Id": "ci-31LGK4KIECHNY",
"Ec2InstanceId": "i-0524ec45912345678",
"PublicDnsName": "ec2-52-123-201-221.us-west-2.compute.amazonaws.com",
"PublicIpAddress": "52.123.201.221",
"PrivateDnsName": "ip-172-31-41-111.us-west-2.compute.internal",
"PrivateIpAddress": "172.31.41.111",
"Status": {
"State": "RUNNING",
"StateChangeReason": {},
"Timeline": {
"CreationDateTime": 1536626164.073,
"ReadyDateTime": 1536626533.886
}
},
"InstanceGroupId": "ig-2EHOYXFABCDEF",
"Market": "ON_DEMAND",
"InstanceType": "m3.xlarge",
"EbsVolumes": []
}
]
}
STEP 6: Find the matching InstanceGroupId ig-2EHOYXFABCDEF. This will give you the EC2 Instance ID for the MASTER: "Ec2InstanceId": "i-0524ec45912345678"
Step 7: Tag your EC2 instance:
aws ec2 create-tags --resources i-0524ec45912345678 --tags Key=EMR,Value=MASTER
The above steps might be simpler with CLI Filters and / or jq, but this should be enough information so that you know how to find and tag the EMR Master Instance.
Below can be used to directly get instance Id
aws emr list-instances --cluster-id ${aws_emr_cluster.cluster.id} --instance-
group-id ${aws_emr_cluster.cluster.master_instance_group.0.id} --query
'Instances[*].Ec2InstanceId' --output text
In an enviroinment where you does not have the aws cli, you can cat the following file:
cat /mnt/var/lib/info/job-flow.json
An example of the content is the following one:
{
"jobFlowId": "j-0000X0X0X00XX",
"jobFlowCreationInstant": 1579512208006,
"instanceCount": 2,
"masterInstanceId": "i-00x0xx0000xxx0x00",
"masterPrivateDnsName": "localhost",
"masterInstanceType": "m5.xlarge",
"slaveInstanceType": "m5.xlarge",
"hadoopVersion": "2.8.5",
"instanceGroups": [
{
"instanceGroupId": "ig-0XX00XX0X0XXX",
"instanceGroupName": "Master - 1",
"instanceRole": "Master",
"marketType": "OnDemand",
"instanceType": "m5.xlarge",
"requestedInstanceCount": 1
},
{
"instanceGroupId": "ig-000X0XXXXXXX",
"instanceGroupName": "Core - 2",
"instanceRole": "Core",
"marketType": "OnDemand",
"instanceType": "m5.xlarge",
"requestedInstanceCount": 1
}
]
NOTE: i've omitted the ID of the jobs using 0 where a number is expected and X where a ltter is expected.
You can do this programmatically in 3 lines of code, without having to copy-paste any of the specific information:
# get cluster id
CLUSTER_ID=$(aws emr list-clusters --active --query "Clusters[0].Id" --output text)
# get instance id
INSTANCE_ID=$(aws emr list-instances --cluster-id $CLUSTER_ID --instance-group-types MASTER --query "Instances[0].Ec2InstanceId" --output text)
# tag
aws ec2 create-tags --resources $INSTANCE_ID --tags Key=EMR,Value=MASTER
Below example is for Instance Fleet, it saves Cluster ID, Instance Fleet ID and Master IP as environment variables.
Replace cluster name "My-Cluster" to the actual cluster name.
export CLUSTER_ID=$(aws emr list-clusters --active --query 'Clusters[?Name==`My-Cluster`].Id' --output text)
export INSTANCE_FLEET=$(aws emr describe-cluster --cluster-id $CLUSTER_ID | jq -r '.[].InstanceFleets | .[] | select(.InstanceFleetType=="MASTER") | .Id')
export PRIVATE_IP=aws emr list-instances --cluster-id $CLUSTER_ID --instance-fleet-id $INSTANCE_FLEET --query 'Instances[*].PrivateIpAddress' --output text
"Cleanest" way:
aws emr list-clusters --active
Search for Master cluster ID (j-xxxxxxxxxxx), then
aws emr list-instances --region {your_region} --instance-group-types MASTER --cluster-id j-xxxxxxxxxxxxx
Immediately filters out the master instance(s) with it's information using --instance-group-types MASTER flag.
For tagging refers to the other answers with aws {resource} create-tags and --tag flag.

AWS-cli ec2 describe instances

i am digging on aws cli and from past 5 hours i struggling with 2 cli commands
i should get InstanceId, Name(this is a tag value) and specific tag value by providing its key ( including not tags given i.e NULL)
I should get InstanceId, Name and specific tag value by providing its key ( excluding NULL tags)
i got 50% of the answer for the 1 Question & 2 Question 0%
My cli command:
aws ec2 describe-instances --query 'jsondata[ ].Instances[
].[InstanceId, [Tags[?keys='Name'].Value] [0][0]' --output table`
Ex: {
"Jsondata" : [
{ "Instances" : "i-xxxxxx",
"Tags":[
{ "valve":" testserver",
"key": "server"
},
{ "valve":" elb",
"key": "Name"
}
]
},
{ "Instances" : "i-yyyyyy",
"Tags":[
{ "valve": " ",
"key": " "
},
{ "valve":" elb2",
"key": "Name"
}
]
}
]`
Thanks in advance. Please help me i need to sleep
To describe all instances with Tag "NAME" Use:
aws ec2 describe-instances --filters "Name=tag-key,Values=Name"
or
This Gives InstanceId with Particular Tag "Name"
aws ec2 describe-instances --query 'Reservations[].Instances[].[InstanceId, Tags[?Key==`Name`].Value[0]]'
or
This Gives InstanceId with Particular Tag "Name" and Value of Tag
aws ec2 describe-instances --query 'Reservations[].Instances[].[InstanceId, Tags[?Key==`Name`], Tags[?Key==`Name`].Value[]]'
To describe all instances with Tag "Purpose" and its value as "test" Use:
aws ec2 describe-instances --filters "Name=tag:Purpose,Values=test"
If you already know the Instance id:
aws ec2 describe-instances --instance-ids i-1234567890abcdef0
To find every instance which doesn't contain a tag named "Purpose":
aws ec2 describe-instances | jq '.Reservations[].Instances[] | select(contains({Tags: [{Key: "Purpose"} ]}) | not)'
To filter against the value of the tag, instead of the name of the tag:
aws ec2 describe-instances | jq '.Reservations[].Instances[] | select(contains({Tags: [{Key: "Name"}, {Value: "testbox1"}]}) | not)'
To find every instance which doesn't contain a tag:
aws ec2 describe-instances | jq '.Reservations[].Instances[] | select(contains({Tags: [{Key: ""}, {Value: ""}]}) | not)'

How to list only the instance id and related tags

I using the CLI tools and I want to list only the instance ID and related tags of an instance. The command that I am using is ec2-describe instances. I have tried certain filters but nothing is working out.
Can anybody help ??
error :
./Instance_Audit.sh: line 24: $: command not found
./Instance_Audit.sh: line 25: syntax error near unexpected token `do'
./Instance_Audit.sh: line 25: ` do echo $ID ; echo $(aws ec2 describe-instances --query "Reservations[].Instances[?InstanceId==\`$ID\`].Tags[]") done'
Using the aws cli, I can produce a few different alternatives of the tag output:
aws ec2 describe-instances --output text --query 'Reservations[*].Instances[*].[InstanceId,Tags[]]'
Result: i-xxxxx [{u'Value': 'key1value', u'Key': 'key1name'}, {u'Value': 'key2value', u'Key': 'key2name'}]
aws ec2 describe-instances --output text --query 'Reservations[*].Instances[*].[InstanceId,Tags[].*]'
Result: i-xxxxx [['key1value', 'key1name'], ['key2value', 'key2name']]
aws ec2 describe-instances --output text --query 'Reservations[*].Instances[*].[InstanceId,Tags[].Key,Tags[].Value]'
Result: i-xxxx ['key1name', 'key2name'] ['key1value' 'key2value']
Is any of those formats what you're looking for?
The AWS CLI command
aws ec2 describe-instances --region us-west-2 --query 'Reservations[].Instances[].[InstanceId,Tags[]]' --output text
produces these results:
i-789c55c3
Name Instance1
aws:cloudformation:logical-id Instance1
aws:cloudformation:stack-id arn:aws:cloudformation:us-west-2:012345678901:stack/test10-test10-foobarfoobar/a6e545a0-af52-11e4-a0be-50d5020578e0
aws:cloudformation:stack-name test10-test10-foobarfoobar
i-77c108cc
Name Instance2
aws:cloudformation:logical-id Instance2
aws:cloudformation:stack-id arn:aws:cloudformation:us-west-2:012345678901:stack/test10-test10-foobarfoobar/a6e545a0-af52-11e4-a0be-50d5020578e0
aws:cloudformation:stack-name test10-test10-foobarfoobar
If that is not in the format you are looking for, can you provide an example of they format you are expecting?
If you want to only display the instance id and the tags associated to that instance, you can use something like :
$ for ID in $(aws ec2 describe-instances --region eu-west-1 --query "Reservations[].Instances[].InstanceId" --output text); do echo $ID ; echo $(aws ec2 describe-instances --region eu-west-1 --query "Reservations[].Instances[?InstanceId==\`$ID\`].Tags[]") done
i-60****2a
[ [ { "Value": "SB", "Key": "Name" } ] ]
i-1a****56
[ [ { "Value": "Course and Schedule on Apache 2.4", "Key": "Name" } ] ]
i-88eff3cb
[ [ { "Value": "secret.com", "Key": "Name" } ] ]
i-e7****a5
[ [ { "Value": "2014.03 + Docker", "Key": "Name" } ] ]
I could not find a way to do that with only one AWS CLI call. Should someone come up with a shorter solution, I would love to hear from you.
If you want to filter to certain tag key/value only, you can edit the aws ec2 describe-instances to add a --filter option.
For example :
aws ec2 describe-instances --region eu-west-1 --filter "Name=tag:Name,Values=SB"
--Seb
simple answer:
aws ec2 describe-instances \
--query 'Reservations[*].Instances[*].[InstanceId,Tags[*]]'
not so simple answer .. the same principle could be applied to both instances and vpc's
# how-to get all the tags per instance-id with aws ec2 for prd env
while read -r i ; do \
aws ec2 describe-tags --filters "Name=resource-id,Values=$i" \
--profile prd; done < <(aws ec2 describe-instances \
--query 'Reservations[*].Instances[*].InstanceId' --profile prd)
# how-to get all the tags per virtual-private-cloud-id
while read -r v; do aws ec2 describe-tags \
--filters "Name=resource-id,Values=$v" ; \
done < <(aws ec2 describe-vpcs \
--query 'Vpcs[*].VpcId'|perl -ne 's/\t/\n/g;print')