I would like to edit/update my CloudFront distribution with awscli.
I'm using latest cli version:
aws-cli/1.11.56 Python/2.7.10 Darwin/16.4.0 botocore/1.5.19
To use cloudfront features in awscli you need to add this to your aws config file:
[preview]
cloudfront = true
I'm getting config of the distribution that I'd like to modify:
aws cloudfront get-distribution-config --id FOO_BAR_ID > cf_config.json
Looks like it worked as expected. Config looks ok for me. Now I'm trying to reconfigure my CF distribution with the same config.
aws cloudfront update-distribution --distribution-config file://cf_config.json --id FOO_BAR_ID
and I'm getting:
Parameter validation failed:
Missing required parameter in DistributionConfig: "CallerReference"
Missing required parameter in DistributionConfig: "Origins"
Missing required parameter in DistributionConfig: "DefaultCacheBehavior"
Missing required parameter in DistributionConfig: "Comment"
Missing required parameter in DistributionConfig: "Enabled"
Unknown parameter in DistributionConfig: "ETag", must be one of: CallerReference, Aliases, DefaultRootObject, Origins, DefaultCacheBehavior, CacheBehaviors, CustomErrorResponses, Comment, Logging, PriceClass, Enabled, ViewerCertificate, Restrictions, WebACLId, HttpVersion, IsIPV6Enabled
Unknown parameter in DistributionConfig: "DistributionConfig", must be one of: CallerReference, Aliases, DefaultRootObject, Origins, DefaultCacheBehavior, CacheBehaviors, CustomErrorResponses, Comment, Logging, PriceClass, Enabled, ViewerCertificate, Restrictions, WebACLId, HttpVersion, IsIPV6Enabled
What is the right way to reconfigure CF using awscli?
#usterk's answer is correct, but it took me another 3 hours to get to the script that I needed. Here, I am sharing it.
My case: CI/CD using S3/CloudFront with manual artifact versioning
I am hosting a static website (SSG) in S3, and I want it to be served by CloudFront. The website gets frequent updates in terms of its code (not just the content) and I want to store all the versions of the website in S3 (just like all the artifacts or docker images) and update CloudFront to point to a new version, right after a new version is pushed to S3.
I know that there is "file versioning" in S3, but this old-school format for keeping all versions of the assets helps with analyzing the assets as well as easy roll-backs.
My configs
After building the assets (JS, CSS, etc), the new files are uploaded to S3 in a folder like s3://<mybucket-name>/artifacts/<version-id>
In CloudFront I have a Distribution for www website. Route53 for www.domain.com is pointing to it.
In that Distribution I have several Origins (e.g. one to send /api path to ELB.)
The Origin that matter here is www which has its OriginPath pointing to /artifacts/<version-id>.
Workflow
After S3 sync is done via AWS CLI, I need to update CloudFront's configs for that www Origin's OriginPath value to point to the new path in S3.
I also need to initiate an invalidation on the Distribution so CloudFront picks up the new files internally (between S3 and it)
The Task
As #usterk and #BrianLeishman pointed out, the only CLI command for this job is update-distribution which per the documentation, requires the ENTIRE CONFIGURATION of the distribution to REPLACE it. So, there is no command to partially update just one field in the config.
To achieve this, one must first get the current distribution-config, then extract the "DistributionConfig" component, then update the fields it takes, and finally, put it back in the proper format with a proper verification token.
Note that what the "update" command needs is a "subset" of what "get" gives back. So parsing JSON via jq is inevitable.
The Bash Script
The following script that I came up with, does the job for me:
# 0) You need to set the followings for your case
CLOUDFRONT_DISTRIBUTION_ID="EABCDEF12345ABCD"
NEW_ORIGIN_PATH="/art/0.0.9"
CLOUDFRONT_ORIGIN_ID="E1A2B3C4D5E6F"
DIST_CONFIG_OLD_FILENAME="dist-config.json" # a temp file, which will be removed later
DIST_CONFIG_NEW_FILENAME="dist-config2.json" # a temp file, which will be removed later
# 1) Get the current config, entirely, and put it in a file
aws cloudfront get-distribution --id $CLOUDFRONT_DISTRIBUTION_ID > $DIST_CONFIG_OLD_FILENAME
# 2) Extract the Etag which we need this later for update
Etag=`cat $DIST_CONFIG_OLD_FILENAME | jq '.ETag' | tr -d \"`
# 3) Modify the config as wished, for me I used `jq` extensively to update the "OriginPath" of the desired "originId"
cat $DIST_CONFIG_OLD_FILENAME | jq \
--arg targetOriginId $CLOUDFRONT_ORIGIN_ID \
--arg newOriginPath $NEW_ORIGIN_PATH \
'.Distribution.DistributionConfig | .Origins.Items = (.Origins.Items | map(if (.Id == $targetOriginId) then (.OriginPath = $newOriginPath) else . end))' \
> $DIST_CONFIG_NEW_FILENAME
# 4) Update the distribution with the new file
aws cloudfront update-distribution --id $CLOUDFRONT_DISTRIBUTION_ID \
--distribution-config "file://${DIST_CONFIG_NEW_FILENAME}" \
--if-match $Etag \
> /dev/null
# 5) Invalidate the distribution to pick up the changes
aws cloudfront create-invalidation --distribution-id $CLOUDFRONT_DISTRIBUTION_ID --paths "/*"
# 6) Clean up
rm -f $DIST_CONFIG_OLD_FILENAME $DIST_CONFIG_NEW_FILENAME
Final Note: IAM Access
The user that performs these needs IAM access to the Get, Invalidate, and Update actions on the Distribution in CloudFront. Here is the Policy that gives that:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"cloudfront:GetDistribution",
"cloudfront:UpdateDistribution",
"cloudfront:CreateInvalidation"
],
"Resource": "arn:aws:cloudfront::<ACCOUNT_ID>:distribution/<DISTRIBUTION_ID>
}
]
}
You have to edit cf_config.json before using it with update-distribution and remove
{
"ETag": "ETag_Value",
"DistributionConfig":
from the beginning of the file and last
}
from the end of file.
Then use this command with the right id and ETag value that was removed from cf_config.json
aws cloudfront update-distribution --distribution-config file://cf_config.json --id FOO_BAR_ID --if-match ETag_Value
aws cloudfront get-distribution-config Generates extra information you cannot use in update-distribution
so you have to get only DistributionConfig from generated json and modify it.
using jq
aws cloudfront get-distribution --id <CLOUDFRONT_DISTRIBUTION_ID> | jq .Distribution.DistributionConfig > config.json
also you will need Etag separately you can get it by:
ETAG=`aws cloudfront get-distribution --id <CLOUDFRONT_DISTRIBUTION_ID> | jq -r .ETag`
and then use it in update-distribution. e.g.
if you saved config in config.json:
aws cloudfront update-distribution --id <CLOUDFRONT_DISTRIBUTION_ID> --distribution-config "file://config.json" --if-match $ETAG > /dev/null
Example bash script to change Origin:
https://gist.github.com/ahmed-abdelazim/d5aa4dea6ecb5dbbff94ce1f5c1f32ff?fbclid=IwAR1QL1CujiCEyd5cDLoEuocOuXNfstxV9Ev6ndO9IorHVsx0EMroBVnimNg
#!/bin/bash
CLOUDFRONT_DISTRIBUTION_ID=E2C3RNL2F4MRMQ
NEW_ORIGIN="origin2-zaid.s3.us-west-2.amazonaws.com"
ETAG=`aws cloudfront get-distribution --id $CLOUDFRONT_DISTRIBUTION_ID | jq -r .ETag`
aws cloudfront get-distribution --id $CLOUDFRONT_DISTRIBUTION_ID | \
jq --arg NEW_ORIGIN "$NEW_ORIGIN" '.Distribution.DistributionConfig.Origins.Items[0].Id=$NEW_ORIGIN' | \
jq --arg NEW_ORIGIN "$NEW_ORIGIN" '.Distribution.DistributionConfig.Origins.Items[0].DomainName=$NEW_ORIGIN' | \
jq --arg NEW_ORIGIN "$NEW_ORIGIN" '.Distribution.DistributionConfig.DefaultCacheBehavior.TargetOriginId=$NEW_ORIGIN' | \
jq .Distribution.DistributionConfig > config.json
aws cloudfront update-distribution --id $CLOUDFRONT_DISTRIBUTION_ID --distribution-config "file://config.json" --if-match $ETAG > /dev/null
aws cloudfront create-invalidation --distribution-id $CLOUDFRONT_DISTRIBUTION_ID --paths "/*"
rm config.json
Related
I'm trying to delete an empty folder structure using the cli and noting seems to be working. The root folder is /+7000/ and I'm positive it's because of the "+"
My rm commands are working on other folders without special characters and the cli isn't returning an error. How would I build this script to recognize this folder and get rid of it?
Test Scripts
(%2B is '+' in hex)
>aws s3 ls
2022-07-13 10:29:36 0 %2B7000/
>
>
//Attempts
> aws s3 rm s3://namespace/ --exclude "*" --include "%2B7000/"
> aws s3 rm s3://namespace/ --exclude "*" --include "*7000/"
> aws s3 rm s3://namespace/ --exclude "*" --include "[+]7000/"
> aws s3 rm s3://namespace/ --exclude "*" --include "'+7000'/"
> aws s3 rm s3://namespace/"\+7000/"
> aws s3 rm s3://namespace/%2B7000/
delete: s3://namespace/%2B7000/
Most attempts return a successful deletion but the folder is still there.
Output from aws s3api list-objects --bucket namespace
{
"Key": "+7000/",
"LastModified": "2022-07-13T14:29:36.884Z",
"ETag": "\"100\"",
"Size": 0,
"StorageClass": "STANDARD",
"Owner": {
"DisplayName": "vmsstg",
"ID": "1-2-3-4"
}
}
If aws s3 rm isn't working, you can try the 'lower-level' API call:
aws s3api delete-object --bucket namespace --key %2B7000/
Given that you said that %2B is plus, and based on your comment, you can use:
aws s3api delete-object --bucket namespace --key "+7000/"
I guess the plus sign got translated into 'URL Encode' syntax somewhere along the way.
Another approach is to use an AWS SDK (eg boto3 for Python) to retrieve the Key and then delete the object by passing back the exact value. This would avoid any encoding in the process.
is that possible to get aws cloudfront dist id by tag via awscli or aws sdk for powershell. I could only get only ID by ARN number of resource
aws cloudfront list-tags-for-resource --resource XXX
{
"Tags": {
"Items": [
{
"Value": "TEST_APP",
"Key": "CLIENT_APP"
}
]
}
}
UPDATE
Solved via
cloudfrontdistids=$(aws cloudfront list-distributions | jq -r ".DistributionList.Items[].ARN")
for dist in $cloudfrontdistids
do
if [ $(aws cloudfront list-tags-for-resource --resource $dist | jq -r ".Tags.Items[].Value") == $VALUE ]
then
CLOUDFRONT_DISTRIBUTION_ID=${dist:(-14)}
fi
done
The answer provided by the original poster seems to be almost correct.
But the CLOUDFRONT_DISTRIBUTION_ID is not always 14 characters, so in order to avoid pulling aditional characters use this instead:
CLOUDFRONT_DISTRIBUTION_ID=${dist##*/}
In bash, it removes a prefix pattern. Here, it's basically giving you everything after the last path separator /, by greedily removing the prefix */
In AWS IAM is there a way, either by scripting or in the web console, to find which existing policies contain a given action?
For example, I want to allow role myRole to have access to the DescribeTags action on all of my EC2 instances. I know I can create my own policy with an appropriate rule, but would like to use an existing Amazon policy if such a thing exists.
This is an old post, but it may help someone... Despite what others have said, you can do this. It just requires a bit of scripting.
You can do the following with the AWS CLI.
Get the ARNs of the policies and store in the policies_arn array.
mapfile -t policies_arn < <(aws iam list-policies --query 'Policies[*].[Arn]' --output text)
Get the VersionIds for each policy and store in the policies_ver array.
mapfile -t policies_ver < <(aws iam list-policies --query 'Policies[*].[DefaultVersionId]' --output text)
Use a for loop to loop through each policy and store the policy document in policies.txt
for (( i=0; i<${#policies_arn[#]}; i++ )); do echo ${policies_arn[i]} >> policies.txt && aws iam get-policy-version --policy-arn ${policies_arn[i]} --version-id ${policies_ver[i]} --output json >> policies.txt; done
Open up policies.txt in a text editor and search for your action.
Note: Depending on your CLI configuration, you may or may not need the --output text parameter. However, the output must be text (not JSON) in order for this to work.
From there, you can turn that into a .sh shell script easily enough.
Sample Output:
arn:aws:iam::123456789012:policy/DenyAllAccess
{
"PolicyVersion": {
"CreateDate": "2016-12-06T18:40:51Z",
"VersionId": "v1",
"Document": {
"Statement": [
{
"Action": "*",
"Effect": "Deny",
"Resource": "*"
}
],
"Version": "2012-10-17"
},
"IsDefaultVersion": true
}
}
Cheers!
To elaborate on my understanding of https://stackoverflow.com/a/43128660/447862 the important thing is to get the JSON from the default version of each policy. Since my machine does not have mapfile I opted to use Python.
import boto3, json, sys
iam = boto3.client('iam')
policy_arn = sys.argv[1]
policy_name = policy_arn.split('/')[-1]
version = iam.get_policy(PolicyArn=policy_arn)['Policy']['DefaultVersionId']
policy_version = iam.get_policy_version(PolicyArn=policy_arn, VersionId=version)
policy_document = policy_version['PolicyVersion']['Document']
with open(f"{policy_name}.json", 'w') as outfile:
outfile.write(json.dumps(policy_document, indent=2))
outfile.write('\n')
Saving this as aws-iam-policy-dump.py, now I can write each policy document to its own file and search as much as I want.
aws iam list-policies --query 'Policies[*].Arn' --output text > policy-arns.txt
for arn in $(cat policy-arns.txt); do
python aws-iam-policy-dump.py $arn
done
This could probably go faster by doing everything in Python, but this hybrid approach is good enough for me.
aws iam list-policies --query 'Policies[*].[Arn, DefaultVersionId]' | jq -rc '.[] | join(" ")' | xargs -l bash -c 'aws iam get-policy-version --policy-arn=$0 --version-id=$1 --query="{\"$0\": PolicyVersion.Document.Statement[*].[Action, NotAction][][]}"' | jq -c | grep YOUR_POLICY_ACTION_HERE
This basically captures all the Arns and Versions in the single call
Then joins the output into a single space-separated string
This string is then sent to another bash process via xargs where the ARN and VersionId are passed in as separate parameters to get-policy-version
The result of this is combined with the ARN into a single line
Lines are grepped for your pleasure with the search action you're looking for.
I came across this question looking for a way to find existing policies that may contain specific actions. The command will run in O(n) time based on the number of policies since a separate call is made for every policy you have.
I couldn't figure out how to get .contains() working properly in the last --query to allow aws to filter rather than grep.
I'm trying to make my CloudFront hosted blog redirect /feed/atom/index.html to /index.xml. I have the following script that is supposed to set up redirect headers for me:
#!/bin/sh
redirect() aws s3api copy-object \
--copy-source blog.afoolishmanifesto.com$1 \
--bucket blog.afoolishmanifesto.com --key $1 \
--metadata x-amz-website-redirect-location=$2 \
--metadata-directive REPLACE
redirect /feed/atom/index.html /index.xml
After running the script I get the following output:
{
"CopyObjectResult": {
"LastModified": "2016-03-27T07:26:03.000Z",
"ETag": "\"40c27e3a5ea160c6695d7f34de8b4dea\""
}
}
And when I refresh the object in the AWS console view of S3 I do not see a Website Redirect Location (or x-amz-website-redirect-location) piece of metadata for the object in question. What can I do to ensure that the redirect is configured correctly?
Note: I have tried specifying the metadata as JSON and as far as I can tell it made no difference.
UPDATE: I have left the above question the same, as it still applies to metadata, but if you are trying to create a redirect with aws s3api you should use the --website-redirect-location option, not --metadata.
It is not able to create a key /feed/atom/index.html in the bucket, so no metadata attribute was not created. Instead you should create feed/atom/index.html. I'll modify it like:
#!/bin/sh
redirect() aws s3api copy-object \
--copy-source blog.afoolishmanifesto.com/$1 \
--bucket blog.afoolishmanifesto.com --key $1 \
--metadata x-amz-website-redirect-location=$2 \
--metadata-directive REPLACE
redirect feed/atom/index.html /index.xml
In my solution, notice / in --copy-source and the first argument to redirect script missing the leading /
How can I convert the following command and use s3cmd instead?
aws s3 sync ./test s3://test --content-encoding "gzip" --content-type "text/html" --cache-control "max-age=0, no-cache" --exclude "*" --include "index.html" --profile stage
I want to set the content-encoding, cache-control, content-type, include and exclude tags and profile as I did above.