is that possible to get aws cloudfront dist id by tag via awscli or aws sdk for powershell. I could only get only ID by ARN number of resource
aws cloudfront list-tags-for-resource --resource XXX
{
"Tags": {
"Items": [
{
"Value": "TEST_APP",
"Key": "CLIENT_APP"
}
]
}
}
UPDATE
Solved via
cloudfrontdistids=$(aws cloudfront list-distributions | jq -r ".DistributionList.Items[].ARN")
for dist in $cloudfrontdistids
do
if [ $(aws cloudfront list-tags-for-resource --resource $dist | jq -r ".Tags.Items[].Value") == $VALUE ]
then
CLOUDFRONT_DISTRIBUTION_ID=${dist:(-14)}
fi
done
The answer provided by the original poster seems to be almost correct.
But the CLOUDFRONT_DISTRIBUTION_ID is not always 14 characters, so in order to avoid pulling aditional characters use this instead:
CLOUDFRONT_DISTRIBUTION_ID=${dist##*/}
In bash, it removes a prefix pattern. Here, it's basically giving you everything after the last path separator /, by greedily removing the prefix */
Related
I need to remove a list of outdated Ami's in AWS. But first I require to verifying are there any place still using those Ami's. Is there an efficient way to find it out.
First, you need to define "Outdated AMIs", it can be the creation date or something else. But here is the script that will list Instance ID along with AMI details that used by particular instance in particular region. By default it will look into default region.
#Get list of EC2 instance
echo "Getting EC2"
EC2_LIST=$(aws ec2 describe-instances --query 'Reservations[].Instances[].{InstanceId:InstanceId}' --output text | tr '\n' ' ')
#Get list of AMI used by ec2
echo "Getting AMI"
LIST_AMI_ID=$(aws ec2 describe-instances --query 'Reservations[].Instances[].{ImageId:ImageId,InstanceId:InstanceId}' --output text | tr '\n' ' ')
EC2_LIST_ARRAY=($EC2_LIST)
LIST_AMI_ID_ARRAY=($LIST_AMI_ID)
for index in ${!LIST_AMI_ID_ARRAY[*]}; do
echo "Get details for AMI ${LIST_AMI_ID_ARRAY[$index]}"
#Get details of AMI
AMI_DETAILS=$(aws ec2 describe-images --image-ids ${LIST_AMI_ID_ARRAY[$index]} --query 'Images[].{CreationDate:CreationDate,Tags:Tags[]}')
echo "Instance having ID ${EC2_LIST_ARRAY[$index]} using AMI ID ${LIST_AMI_ID_ARRAY[$index]} Details: $AMI_DETAILS"
done
output
Get details for AMI ami-0219162cf838b3455
Instance having ID i-0ceb0dfa197fd7455 using AMI ID ami-0219162cf838b3455 Details: [
{
"CreationDate": "2019-10-22T05:17:46.000Z",
"Tags": [
{
"Key": "Base_AMI_Name",
"Value": "ubuntu/images/hvm-ssd/ubuntu-bionic-18.04-amd64-server-20191021"
},
{
"Key": "OS_Version",
"Value": "Ubuntu"
},
{
"Key": "Release",
"Value": "Latest"
},
{
"Key": "Name",
"Value": "postgres"
},
{
"Key": "Scope",
"Value": "database"
}
]
}
]
aws-cli-cheatsheet
For what it's worth, I built a small tool for this purpose as well -- a bit of JavaScript wrapped around AWS API calls.
This basically tries to find all the AMIs in use in your account and looks at both AMIs by their age and also the newly introduced deprecated flag:
https://github.com/geoffreywiseman/oldamis
https://www.codiform.com/blog/oldamis/
It's pretty simplistic currently, but I'm willing to entertain feature requests. ;)
I am trying to get the values of the Name tag of AWS EC2 instances using jq.
aws ec2 describe-instances | jq -r '.Reservations[].Instances[] | .Tags[] | select (.Key == "Name")'
But I am getting this error:
jq: error: Name/0 is not defined at <top-level>, line 1:
.Reservations[].Instances[] | .Tags[] | select (.Key == Name)
jq: 1 compile error
This is the json I'm trying to process:
{
"Reservations": [{
"Instances": [{
"AmiLaunchIndex": 0,
"ImageId": "ami-00c3c949f325a4149",
"InstanceId": "i-0c17052ee1c7113e5",
"Architecture": "x86_64",
"Tags": [{
"Key": "Name",
"Value": "bastion001"
},
{
"Key": "environment",
"Value": "stg-us-east"
}
]
}]
}]
}
How can I get the value of the Name tag from EC2 instances?
You don't need extra tools like jq to query the output. AWS CLI has JMESPath
built-in to help you do that.
aws ec2 describe-instances --query 'Reservations[*].Instances[*].Tags[?Key == `Name`].Value'
You could do something like this, this will get what you want from ec2 in a comma separated list
jq -r '.Reservations[].Instances[] \ | ((.Tags // empty) | from_entries) as $tags | [($tags.Name), ($tags.environment), .ImageId, .InstanceId, .AmiLaunchIndex] | #csv'
The .Tags // empty will ignore those with tags that do not exist if you are wondering.
Hope this helps, it Is correct you do not need jq to get these details, but this is how you do it with :)
There is a mismatch between the jq command-line expression shown in the Q and the error message. I would suggest that, at least until you have sorted things out, you put your jq program in a file, and invoke jq with the -f command-line option.
I would like to edit/update my CloudFront distribution with awscli.
I'm using latest cli version:
aws-cli/1.11.56 Python/2.7.10 Darwin/16.4.0 botocore/1.5.19
To use cloudfront features in awscli you need to add this to your aws config file:
[preview]
cloudfront = true
I'm getting config of the distribution that I'd like to modify:
aws cloudfront get-distribution-config --id FOO_BAR_ID > cf_config.json
Looks like it worked as expected. Config looks ok for me. Now I'm trying to reconfigure my CF distribution with the same config.
aws cloudfront update-distribution --distribution-config file://cf_config.json --id FOO_BAR_ID
and I'm getting:
Parameter validation failed:
Missing required parameter in DistributionConfig: "CallerReference"
Missing required parameter in DistributionConfig: "Origins"
Missing required parameter in DistributionConfig: "DefaultCacheBehavior"
Missing required parameter in DistributionConfig: "Comment"
Missing required parameter in DistributionConfig: "Enabled"
Unknown parameter in DistributionConfig: "ETag", must be one of: CallerReference, Aliases, DefaultRootObject, Origins, DefaultCacheBehavior, CacheBehaviors, CustomErrorResponses, Comment, Logging, PriceClass, Enabled, ViewerCertificate, Restrictions, WebACLId, HttpVersion, IsIPV6Enabled
Unknown parameter in DistributionConfig: "DistributionConfig", must be one of: CallerReference, Aliases, DefaultRootObject, Origins, DefaultCacheBehavior, CacheBehaviors, CustomErrorResponses, Comment, Logging, PriceClass, Enabled, ViewerCertificate, Restrictions, WebACLId, HttpVersion, IsIPV6Enabled
What is the right way to reconfigure CF using awscli?
#usterk's answer is correct, but it took me another 3 hours to get to the script that I needed. Here, I am sharing it.
My case: CI/CD using S3/CloudFront with manual artifact versioning
I am hosting a static website (SSG) in S3, and I want it to be served by CloudFront. The website gets frequent updates in terms of its code (not just the content) and I want to store all the versions of the website in S3 (just like all the artifacts or docker images) and update CloudFront to point to a new version, right after a new version is pushed to S3.
I know that there is "file versioning" in S3, but this old-school format for keeping all versions of the assets helps with analyzing the assets as well as easy roll-backs.
My configs
After building the assets (JS, CSS, etc), the new files are uploaded to S3 in a folder like s3://<mybucket-name>/artifacts/<version-id>
In CloudFront I have a Distribution for www website. Route53 for www.domain.com is pointing to it.
In that Distribution I have several Origins (e.g. one to send /api path to ELB.)
The Origin that matter here is www which has its OriginPath pointing to /artifacts/<version-id>.
Workflow
After S3 sync is done via AWS CLI, I need to update CloudFront's configs for that www Origin's OriginPath value to point to the new path in S3.
I also need to initiate an invalidation on the Distribution so CloudFront picks up the new files internally (between S3 and it)
The Task
As #usterk and #BrianLeishman pointed out, the only CLI command for this job is update-distribution which per the documentation, requires the ENTIRE CONFIGURATION of the distribution to REPLACE it. So, there is no command to partially update just one field in the config.
To achieve this, one must first get the current distribution-config, then extract the "DistributionConfig" component, then update the fields it takes, and finally, put it back in the proper format with a proper verification token.
Note that what the "update" command needs is a "subset" of what "get" gives back. So parsing JSON via jq is inevitable.
The Bash Script
The following script that I came up with, does the job for me:
# 0) You need to set the followings for your case
CLOUDFRONT_DISTRIBUTION_ID="EABCDEF12345ABCD"
NEW_ORIGIN_PATH="/art/0.0.9"
CLOUDFRONT_ORIGIN_ID="E1A2B3C4D5E6F"
DIST_CONFIG_OLD_FILENAME="dist-config.json" # a temp file, which will be removed later
DIST_CONFIG_NEW_FILENAME="dist-config2.json" # a temp file, which will be removed later
# 1) Get the current config, entirely, and put it in a file
aws cloudfront get-distribution --id $CLOUDFRONT_DISTRIBUTION_ID > $DIST_CONFIG_OLD_FILENAME
# 2) Extract the Etag which we need this later for update
Etag=`cat $DIST_CONFIG_OLD_FILENAME | jq '.ETag' | tr -d \"`
# 3) Modify the config as wished, for me I used `jq` extensively to update the "OriginPath" of the desired "originId"
cat $DIST_CONFIG_OLD_FILENAME | jq \
--arg targetOriginId $CLOUDFRONT_ORIGIN_ID \
--arg newOriginPath $NEW_ORIGIN_PATH \
'.Distribution.DistributionConfig | .Origins.Items = (.Origins.Items | map(if (.Id == $targetOriginId) then (.OriginPath = $newOriginPath) else . end))' \
> $DIST_CONFIG_NEW_FILENAME
# 4) Update the distribution with the new file
aws cloudfront update-distribution --id $CLOUDFRONT_DISTRIBUTION_ID \
--distribution-config "file://${DIST_CONFIG_NEW_FILENAME}" \
--if-match $Etag \
> /dev/null
# 5) Invalidate the distribution to pick up the changes
aws cloudfront create-invalidation --distribution-id $CLOUDFRONT_DISTRIBUTION_ID --paths "/*"
# 6) Clean up
rm -f $DIST_CONFIG_OLD_FILENAME $DIST_CONFIG_NEW_FILENAME
Final Note: IAM Access
The user that performs these needs IAM access to the Get, Invalidate, and Update actions on the Distribution in CloudFront. Here is the Policy that gives that:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"cloudfront:GetDistribution",
"cloudfront:UpdateDistribution",
"cloudfront:CreateInvalidation"
],
"Resource": "arn:aws:cloudfront::<ACCOUNT_ID>:distribution/<DISTRIBUTION_ID>
}
]
}
You have to edit cf_config.json before using it with update-distribution and remove
{
"ETag": "ETag_Value",
"DistributionConfig":
from the beginning of the file and last
}
from the end of file.
Then use this command with the right id and ETag value that was removed from cf_config.json
aws cloudfront update-distribution --distribution-config file://cf_config.json --id FOO_BAR_ID --if-match ETag_Value
aws cloudfront get-distribution-config Generates extra information you cannot use in update-distribution
so you have to get only DistributionConfig from generated json and modify it.
using jq
aws cloudfront get-distribution --id <CLOUDFRONT_DISTRIBUTION_ID> | jq .Distribution.DistributionConfig > config.json
also you will need Etag separately you can get it by:
ETAG=`aws cloudfront get-distribution --id <CLOUDFRONT_DISTRIBUTION_ID> | jq -r .ETag`
and then use it in update-distribution. e.g.
if you saved config in config.json:
aws cloudfront update-distribution --id <CLOUDFRONT_DISTRIBUTION_ID> --distribution-config "file://config.json" --if-match $ETAG > /dev/null
Example bash script to change Origin:
https://gist.github.com/ahmed-abdelazim/d5aa4dea6ecb5dbbff94ce1f5c1f32ff?fbclid=IwAR1QL1CujiCEyd5cDLoEuocOuXNfstxV9Ev6ndO9IorHVsx0EMroBVnimNg
#!/bin/bash
CLOUDFRONT_DISTRIBUTION_ID=E2C3RNL2F4MRMQ
NEW_ORIGIN="origin2-zaid.s3.us-west-2.amazonaws.com"
ETAG=`aws cloudfront get-distribution --id $CLOUDFRONT_DISTRIBUTION_ID | jq -r .ETag`
aws cloudfront get-distribution --id $CLOUDFRONT_DISTRIBUTION_ID | \
jq --arg NEW_ORIGIN "$NEW_ORIGIN" '.Distribution.DistributionConfig.Origins.Items[0].Id=$NEW_ORIGIN' | \
jq --arg NEW_ORIGIN "$NEW_ORIGIN" '.Distribution.DistributionConfig.Origins.Items[0].DomainName=$NEW_ORIGIN' | \
jq --arg NEW_ORIGIN "$NEW_ORIGIN" '.Distribution.DistributionConfig.DefaultCacheBehavior.TargetOriginId=$NEW_ORIGIN' | \
jq .Distribution.DistributionConfig > config.json
aws cloudfront update-distribution --id $CLOUDFRONT_DISTRIBUTION_ID --distribution-config "file://config.json" --if-match $ETAG > /dev/null
aws cloudfront create-invalidation --distribution-id $CLOUDFRONT_DISTRIBUTION_ID --paths "/*"
rm config.json
I've got the following from describe-instances:
{
"Reservations": [
{
"Instances": [
{
"PublicDnsName": "ec2-xxxxx.amazonaws.com",
"Tags": [
{
"Key": "Name",
"Value": "yyyyy"
},
{
"Key": "budget_cluster",
"Value": "zzzzz"
},
{
"Key": "poc",
"Value": "aaaaaaa"
}
]
}
]
}
]
}
For each instance, I would like to extract the PublicDnsName and the value of the "budget_cluster" tag key. How to do this either with ec2 describe-instances or with jq ?
Modifying Frédéric's answer:
aws ec2 describe-instances --output text --query \
'Reservations[].Instances[].[PublicDnsName, Tags[?Key==`budget_cluster`].Value | [0]]'
Would produce:
ec2-xxxxx.amazonaws.com zzzzz
ec2-bbbbb.amazonaws.com yyyyy
I've changed the output to text, which removes as much formatting as possible and selected the individual tag value with | [0] since there will only ever be one per instance anyway. Finally, I removed the [] at the end so that the resulting list isn't flattened. That way in text output each entry will be on its own line.
You can also make this more robust by only selecting instances that actually have that tag. You could do so with further modifications to the --query parameter, but it is better in this case to use the --filters parameter since it does service-side filtering. Specifically you want the tag-key filter: --filters "Name=tag-key,Values=budget_cluster"
aws ec2 describe-instances --output text \
--filters "Name=tag-key,Values=budget_cluster" --query \
'Reservations[].Instances[?Tags[?Key==`budget_cluster`]].[PublicDnsName, Tags[?Key==`budget_cluster`].Value | [0]]'
Would still produce:
ec2-xxxxx.amazonaws.com zzzzz
ec2-bbbbb.amazonaws.com yyyyy
But over the wire you would only be getting the instances you care about, thus saving money on bandwidth.
Using jq 1.5 or later, the simplest approach is to use from_entries.
After a minimal fix of the illustrative input, the following invocation:
$ jq '.Reservations[]
| .Instances[]
| [.PublicDnsName, (.Tags|from_entries|.budget_cluster)]' input.json
produces:
[
"ec2-xxxxx.amazonaws.com",
"zzzzz"
]
If you do not have jq 1.5 or later, the following should work:
$ jq1.4 '.Reservations[]
| .Instances[]
| [.PublicDnsName, (.Tags[]|select(.Key=="budget_cluster") | .Value)]' input.json
The answer of #peak is great and I keep learning from him on the jq part but again you can achieve quite a lot from the AWS CLI
aws ec2 describe-instances --query \
'Reservations[].Instances[].{PublicDnsName:PublicDnsName, Budget:Tags[?Key==`budget_cluster`].Value}'
would produce
[
{
"PublicDnsName": "ec2-xxxxx.amazonaws.com",
"Budget": [
"zzzz",
]
}
]
If you do not want to make a new strict JSon out of it, just take the values
aws ec2 describe-instances --query \
'Reservations[].Instances[].[PublicDnsName, Tags[?Key==`budget_cluster`].Value][]'
will produce
[
"ec2-xxxxx.amazonaws.com",
[
"zzzz"
]
]
In AWS IAM is there a way, either by scripting or in the web console, to find which existing policies contain a given action?
For example, I want to allow role myRole to have access to the DescribeTags action on all of my EC2 instances. I know I can create my own policy with an appropriate rule, but would like to use an existing Amazon policy if such a thing exists.
This is an old post, but it may help someone... Despite what others have said, you can do this. It just requires a bit of scripting.
You can do the following with the AWS CLI.
Get the ARNs of the policies and store in the policies_arn array.
mapfile -t policies_arn < <(aws iam list-policies --query 'Policies[*].[Arn]' --output text)
Get the VersionIds for each policy and store in the policies_ver array.
mapfile -t policies_ver < <(aws iam list-policies --query 'Policies[*].[DefaultVersionId]' --output text)
Use a for loop to loop through each policy and store the policy document in policies.txt
for (( i=0; i<${#policies_arn[#]}; i++ )); do echo ${policies_arn[i]} >> policies.txt && aws iam get-policy-version --policy-arn ${policies_arn[i]} --version-id ${policies_ver[i]} --output json >> policies.txt; done
Open up policies.txt in a text editor and search for your action.
Note: Depending on your CLI configuration, you may or may not need the --output text parameter. However, the output must be text (not JSON) in order for this to work.
From there, you can turn that into a .sh shell script easily enough.
Sample Output:
arn:aws:iam::123456789012:policy/DenyAllAccess
{
"PolicyVersion": {
"CreateDate": "2016-12-06T18:40:51Z",
"VersionId": "v1",
"Document": {
"Statement": [
{
"Action": "*",
"Effect": "Deny",
"Resource": "*"
}
],
"Version": "2012-10-17"
},
"IsDefaultVersion": true
}
}
Cheers!
To elaborate on my understanding of https://stackoverflow.com/a/43128660/447862 the important thing is to get the JSON from the default version of each policy. Since my machine does not have mapfile I opted to use Python.
import boto3, json, sys
iam = boto3.client('iam')
policy_arn = sys.argv[1]
policy_name = policy_arn.split('/')[-1]
version = iam.get_policy(PolicyArn=policy_arn)['Policy']['DefaultVersionId']
policy_version = iam.get_policy_version(PolicyArn=policy_arn, VersionId=version)
policy_document = policy_version['PolicyVersion']['Document']
with open(f"{policy_name}.json", 'w') as outfile:
outfile.write(json.dumps(policy_document, indent=2))
outfile.write('\n')
Saving this as aws-iam-policy-dump.py, now I can write each policy document to its own file and search as much as I want.
aws iam list-policies --query 'Policies[*].Arn' --output text > policy-arns.txt
for arn in $(cat policy-arns.txt); do
python aws-iam-policy-dump.py $arn
done
This could probably go faster by doing everything in Python, but this hybrid approach is good enough for me.
aws iam list-policies --query 'Policies[*].[Arn, DefaultVersionId]' | jq -rc '.[] | join(" ")' | xargs -l bash -c 'aws iam get-policy-version --policy-arn=$0 --version-id=$1 --query="{\"$0\": PolicyVersion.Document.Statement[*].[Action, NotAction][][]}"' | jq -c | grep YOUR_POLICY_ACTION_HERE
This basically captures all the Arns and Versions in the single call
Then joins the output into a single space-separated string
This string is then sent to another bash process via xargs where the ARN and VersionId are passed in as separate parameters to get-policy-version
The result of this is combined with the ARN into a single line
Lines are grepped for your pleasure with the search action you're looking for.
I came across this question looking for a way to find existing policies that may contain specific actions. The command will run in O(n) time based on the number of policies since a separate call is made for every policy you have.
I couldn't figure out how to get .contains() working properly in the last --query to allow aws to filter rather than grep.