I have created the database with ManageMasterUserPassword=True. But I couldn't get the Secret Arn from the describe-db-instances command.
aws rds describe-db-instances --db-instance-identifier database-1 --query DBInstances[*].[MasterUsername,MasterUserSecret]
[
[
"postgres",
null
]
]
I have created the database even from console,
Still I am facing the same error.
But I can clearly see the database ARN in SecretsManager and Secrets ARN in database.
Make sure you have more recent awscli, the support for this feature was added in 2.9.10:
https://raw.githubusercontent.com/aws/aws-cli/v2/CHANGELOG.rst
2.9.10
...
* api-change:``rds``: Add support for managing master user password in AWS Secrets Manager for the DBInstance and DBCluster.
...
With this, the output (only when the feature is enabled on the db instance) will contain:
$ aws rds describe-db-instances --db-instance-identifier database-1 --region=us-east-1
...
"MasterUserSecret": {
"SecretArn": "arn:aws:secretsmanager:us-east-1:776665554444:secret:rds!db-88888888-82e1-4a59-8c35-888888888888-SyXcpL",
"SecretStatus": "active",
"KmsKeyId": "arn:aws:kms:us-east-1:888888888888:key/88888888-c6c4-43da-a4a4-888888888888"
},
You can get the actual values with (for example):
$ secret_arn=$(aws rds describe-db-instances --db-instance-identifier database-1 --region=us-east-1 --query DBInstances[*].[MasterUserSecret.SecretArn] --output text)
$ aws secretsmanager get-secret-value --secret-id ${secret_arn} --region us-east-1 --query SecretString --output text
{"username":"admin","password":"SVxxxxxxxxxxxxxxxxxxxxxxxY7gwkD"}
I'm trying to see usage of each API key in my usage plan. However when I run
aws apigateway get-usage --usage-plan-id ***** --start-date 2021-01-18 --end-date 2021-01-24 --profile prod --region ap-south-1 >> week04.txt
It returns usage of only few keys. When I try to do that from the AWS management console from here
It gives me the usage of the same few keys. I have to manually click every single key and generate the report which is very tedious. What am I doing wrong? Please help.
We can combine get-usage-plan-keys and get-usage with xargs to extract for every api key at once.
aws apigateway get-usage-plan-keys --usage-plan-id xx44ww | jq -r ".items[].id" | xargs -I {} aws apigateway get-usage --usage-plan-id xg4j0w --key-id {} --start-date 2021-01-01 --end-date 2021-01-24 --no-paginate > output.json
Breaking it down for better readability:
Getting usage plan keys:
aws apigateway get-usage-plan-keys --usage-plan-id xx44ww
Extracting plan keys
| jq -r ".items[].id"
loop for every api key id with xargs
aws apigateway get-usage --usage-plan-id xg4j0w --key-id {} --start-date 2021-01-01 --end-date 2021-01-24 --no-paginate
I am trying to retrieve all the parameters under a specific path from the AWS Parameter store using the command below:
aws ssm get-parameters-by-path --path some-path --no-paginate
This returns me a JSON with a lot of fields I do not need. How can I use the --query to just retrieve the name and the value?
Any documentation on how can I use the --query parameter? I have tried passing jq query strings, but that doesn't work.
You need to extract the fields from Parameters(Array) and later select the fields you want to get using {key:value} syntax:
aws ssm get-parameters-by-path --path %PATH% --no-paginate --region %REGION% --query "Parameters[].{Key:Name,Val:Value}" --output json
Output Json:
[
{
"Key": "/test/amit",
"Val": "test1"
},
{
"Key": "/test/amit1",
"Val": "test2"
}
]
Or in case you want the output in text, change --output to text.
Output Text:
/test/amit test1
/test/amit1 test2
More info about Controlling Command Output from the AWS CLI.
Is there anyway to just nuke / remove all items in AWS Parameters Store?
All the command line I found are to remove it either one by one or remove it given a list of names.
I also tried using
aws ssm delete-parameters --cli-input-json test.json
with test.json file looks like this
{
"Names": [
"test1",
"test2"
]
}
still does not work..
Ideally if I can use --query and use it as is, that'd be great.
I'm using --query like so
aws ssm get-parameters-by-path --path / --max-items 2 --query 'Parameters[*].[Name]'
When you need to delete all parameters by path in AWS Systems Manager Parameter Store and there are more than 10 parameters you have to deal with pagination.
Otherwise, an the command will fail with the error:
An error occurred (ValidationException) when calling the DeleteParameters operation: 1 validation error detected: Value '[/config/application/prop1, ...]' at 'names' failed to satisfy constraint: Member must have length less than or equal to 10
The following Bash script using AWS CLI pagination options deletes any number of parameters from AWS SSM Parameter Store by path:
#!/bin/bash
path=/config/application_dev/
while : ; do
aws ssm delete-parameters --names $(aws ssm get-parameters-by-path --path "$path" --query "Parameters[*].Name" --output text --max-items 10 $starting_token | grep -v None)
next_token=$(aws ssm get-parameters-by-path --path "$path" --query NextToken --output text --max-items 10 | grep -v None)
if [ -z "$next_token" ]; then
starting_token=""
break
else
starting_token="--starting-token $next_token"
fi
done
You can combine get-parameters-by-path with delete-parameters:
aws ssm delete-parameters --names `aws ssm get-parameters-by-path --path / --query Parameters[].Name --output text`
I tested it by creating two parameters, then running the above command. It successfully deleted by parameters.
try this and execute multiple times
aws ssm delete-parameters --names `aws ssm get-parameters-by-path --path / --recursive --query Parameters[].Name --output text --max-items 9`
Adding to the above. I had to delete around 400 params from the parameter store. Ran the below in command line and it did it! (Change 45 in for loop to whatever number you like);
for ((n=0;n<**45**;n++)); do
aws ssm delete-parameters --names `aws ssm get-parameters-by-path --path / --recursive --query Parameters[].Name --output text --max-items 9`
done
This is my one line solution for this:
$ for key in $(aws ssm get-parameters-by-path --path "/" --recursive | jq -r '.Parameters[] | .Name' | tr '\r\n' ' '); do aws ssm delete-parameter --name ${key}; done
NOTE: Be careful if you copy & paste this as it will remove everything under "/"
Amazon recently added the wonderful feature of tagging EC2 instances with key-value pairs to make management of large numbers of VMs a bit easier.
Is there some way to query these tags in the same way as some of the other user-set data? For example:
$ curl http://169.254.169.254/latest/meta-data/placement/availability-zone
us-east-1d
Is there some similar way to query the tags?
The following bash script returns the Name of your current ec2 instance (the value of the "Name" tag). Modify TAG_NAME to your specific case.
TAG_NAME="Name"
INSTANCE_ID="`wget -qO- http://instance-data/latest/meta-data/instance-id`"
REGION="`wget -qO- http://instance-data/latest/meta-data/placement/availability-zone | sed -e 's:\([0-9][0-9]*\)[a-z]*\$:\\1:'`"
TAG_VALUE="`aws ec2 describe-tags --filters "Name=resource-id,Values=$INSTANCE_ID" "Name=key,Values=$TAG_NAME" --region $REGION --output=text | cut -f5`"
To install the aws cli
sudo apt-get install python-pip -y
sudo pip install awscli
In case you use IAM instead of explicit credentials, use these IAM permissions:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [ "ec2:DescribeTags"],
"Resource": ["*"]
}
]
}
Once you've got ec2-metadata and ec2-describe-tags installed (as mentioned in Ranieri's answer above), here's an example shell command to get the "name" of the current instance, assuming you have a "Name=Foo" tag on it.
Assumes EC2_PRIVATE_KEY and EC2_CERT environment variables are set.
ec2-describe-tags \
--filter "resource-type=instance" \
--filter "resource-id=$(ec2-metadata -i | cut -d ' ' -f2)" \
--filter "key=Name" | cut -f5
This returns Foo.
You can use a combination of the AWS metadata tool (to retrieve your instance ID) and the new Tag API to retrieve the tags for the current instance.
You can add this script to your cloud-init user data to download EC2 tags to a local file:
#!/bin/sh
INSTANCE_ID=`wget -qO- http://instance-data/latest/meta-data/instance-id`
REGION=`wget -qO- http://instance-data/latest/meta-data/placement/availability-zone | sed 's/.$//'`
aws ec2 describe-tags --region $REGION --filter "Name=resource-id,Values=$INSTANCE_ID" --output=text | sed -r 's/TAGS\t(.*)\t.*\t.*\t(.*)/\1="\2"/' > /etc/ec2-tags
You need the AWS CLI tools installed on your system: you can either install them with a packages section in a cloud-config file before the script, use an AMI that already includes them, or add an apt or yum command at the beginning of the script.
In order to access EC2 tags you need a policy like this one in your instance's IAM role:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Stmt1409309287000",
"Effect": "Allow",
"Action": [
"ec2:DescribeTags"
],
"Resource": [
"*"
]
}
]
}
The instance's EC2 tags will available in /etc/ec2-tags in this format:
FOO="Bar"
Name="EC2 tags with cloud-init"
You can include the file as-is in a shell script using . /etc/ec2-tags, for example:
#!/bin/sh
. /etc/ec2-tags
echo $Name
The tags are downloaded during instance initialization, so they will not reflect subsequent changes.
The script and IAM policy are based on itaifrenkel's answer.
If you are not in the default availability zone the results from overthink would return empty.
ec2-describe-tags \
--region \
$(curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone | sed -e "s/.$//") \
--filter \
resource-id=$(curl --silent http://169.254.169.254/latest/meta-data/instance-id)
If you want to add a filter to get a specific tag (elasticbeanstalk:environment-name in my case) then you can do this.
ec2-describe-tags \
--region \
$(curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone | sed -e "s/.$//") \
--filter \
resource-id=$(curl --silent http://169.254.169.254/latest/meta-data/instance-id) \
--filter \
key=elasticbeanstalk:environment-name | cut -f5
And to get only the value for the tag that I filtered on, we pipe to cut and get the fifth field.
ec2-describe-tags \
--region \
$(curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone | sed -e "s/.$//") \
--filter \
resource-id=$(curl --silent http://169.254.169.254/latest/meta-data/instance-id) \
--filter \
key=elasticbeanstalk:environment-name | cut -f5
You can alternatively use the describe-instances cli call rather than describe-tags:
This example shows how to get the value of tag 'my-tag-name' for the instance:
aws ec2 describe-instances \
--instance-id $(curl -s http://169.254.169.254/latest/meta-data/instance-id) \
--query "Reservations[*].Instances[*].Tags[?Key=='my-tag-name'].Value" \
--region ap-southeast-2 --output text
Change the region to suit your local circumstances. This may be useful where your instance has the describe-instances privilege but not describe-tags in the instance profile policy
I have pieced together the following that is hopefully simpler and cleaner than some of the existing answers and uses only the AWS CLI and no additional tools.
This code example shows how to get the value of tag 'myTag' for the current EC2 instance:
Using describe-tags:
export AWS_DEFAULT_REGION=us-east-1
instance_id=$(curl -s http://169.254.169.254/latest/meta-data/instance-id)
aws ec2 describe-tags \
--filters "Name=resource-id,Values=$instance_id" 'Name=key,Values=myTag' \
--query 'Tags[].Value' --output text
Or, alternatively, using describe-instances:
aws ec2 describe-instances --instance-id $instance_id \
--query 'Reservations[].Instances[].Tags[?Key==`myTag`].Value' --output text
For Python:
from boto import utils, ec2
from os import environ
# import keys from os.env or use default (not secure)
aws_access_key_id = environ.get('AWS_ACCESS_KEY_ID', failobj='XXXXXXXXXXX')
aws_secret_access_key = environ.get('AWS_SECRET_ACCESS_KEY', failobj='XXXXXXXXXXXXXXXXXXXXX')
#load metadata , if = {} we are on localhost
# http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AESDG-chapter-instancedata.html
instance_metadata = utils.get_instance_metadata(timeout=0.5, num_retries=1)
region = instance_metadata['placement']['availability-zone'][:-1]
instance_id = instance_metadata['instance-id']
conn = ec2.connect_to_region(region, aws_access_key_id=aws_access_key_id, aws_secret_access_key=aws_secret_access_key)
# get tag status for our instance_id using filters
# http://docs.aws.amazon.com/AWSEC2/latest/CommandLineReference/ApiReference-cmd-DescribeTags.html
tags = conn.get_all_tags(filters={'resource-id': instance_id, 'key': 'status'})
if tags:
instance_status = tags[0].value
else:
instance_status = None
logging.error('no status tag for '+region+' '+instance_id)
A variation on some of the answers above but this is how I got the value of a specific tag from the user-data script on an instance
REGION=$(curl http://instance-data/latest/meta-data/placement/availability-zone | sed 's/.$//')
INSTANCE_ID=$(curl -s http://instance-data/latest/meta-data/instance-id)
TAG_VALUE=$(aws ec2 describe-tags --region $REGION --filters "Name=resource-id,Values=$INSTANCE_ID" "Name=key,Values='<TAG_NAME_HERE>'" | jq -r '.Tags[].Value')
Starting January 2022, this should be also available directly via ec2 metadata api (if enabled).
curl http://169.254.169.254/latest/meta-data/tags/instance
https://aws.amazon.com/about-aws/whats-new/2022/01/instance-tags-amazon-ec2-instance-metadata-service/
Using the AWS 'user data' and 'meta data' APIs its possible to write a script which wraps puppet to start a puppet run with a custom cert name.
First start an aws instance with custom user data: 'role:webserver'
#!/bin/bash
# Find the name from the user data passed in on instance creation
USER=$(curl -s "http://169.254.169.254/latest/user-data")
IFS=':' read -ra UDATA <<< "$USER"
# Find the instance ID from the meta data api
ID=$(curl -s "http://169.254.169.254/latest/meta-data/instance-id")
CERTNAME=${UDATA[1]}.$ID.aws
echo "Running Puppet for certname: " $CERTNAME
puppet agent -t --certname=$CERTNAME
This calls puppet with a certname like 'webserver.i-hfg453.aws' you can then create a node manifest called 'webserver' and puppets 'fuzzy node matching' will mean it is used to provision all webservers.
This example assumes you build on a base image with puppet installed etc.
Benefits:
1) You don't have to pass round your credentials
2) You can be as granular as you like with the role configs.
Jq + ec2metadata makes it a little nicer. I'm using cf and have access to the region. Otherwise you can grab it in bash.
aws ec2 describe-tags --region $REGION \
--filters "Name=resource-id,Values=`ec2metadata --instance-id`" | jq --raw-output \
'.Tags[] | select(.Key=="TAG_NAME") | .Value'
No jq.
aws ec2 describe-tags --region us-west-2 \
--filters "Name=resource-id,Values=`ec2-metadata --instance-id | cut -d " " -f 2`" \
--query 'Tags[?Key==`Name`].Value' \
--output text
Download and run a standalone executable to do that.
Sometimes one cannot install awscli that depends on python. docker might be out of the picture too.
Here is my implementation in golang:
https://github.com/hmalphettes/go-ec2-describe-tags
The Metadata tool seems to no longer be available, but that was an unnecessary dependency anyway.
Follow the AWS documentation to have the instance's profile grant it the "ec2:DescribeTags" action in a policy, restricting the target resources as much as you wish. (If you need a profile for another reason then you'll need to merge policies into a new profile-linked role).
Then:
aws --region $(curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone | sed -e 's/.$//') ec2 describe-tags --filters Name=resource-type,Values=instance Name=resource-id,Values=$(curl http://169.254.169.254/latest/meta-data/instance-id) Name=key,Values=Name |
perl -nwe 'print "$1\n" if /"Value": "([^"]+)/;'
Well there are lots of good answers here but none quite worked for me exactly out of the box, I think the CLI has been updated since some of them and I do like using the CLI. The following single command works out of the box for me in 2021 (as long as the instance's IAM role is allowed to describe-tags).
aws ec2 describe-tags \
--region "$(ec2-metadata -z | cut -d' ' -f2 | sed 's/.$//')" \
--filters "Name=resource-id,Values=$(ec2-metadata --instance-id | cut -d " " -f 2)" \
--query 'Tags[?Key==`Name`].Value' \
--output text
AWS has recently announced support for instance tags in Instance Metadata Service: https://aws.amazon.com/about-aws/whats-new/2022/01/instance-tags-amazon-ec2-instance-metadata-service/
If you have have the tag metadata option enabled for an instance, you can simply do
$ TOKEN=`curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 900"`
$ curl -H "X-aws-ec2-metadata-token: $TOKEN" -v http://169.254.169.254/latest/meta-data/tags/instance
It is possible to get Instance tags from within the instance via metadata.
First, allow access to tags in instance metadata as explained here
Then, run this command for IMDSv1, Refer
curl http://169.254.169.254/latest/meta-data/tags/instance/Name
or this command for IMDSv2
TOKEN=`curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"` \
&& curl -H "X-aws-ec2-metadata-token: $TOKEN" -v http://169.254.169.254/latest/meta-data/tags/instance
Install AWS CLI:
curl "https://s3.amazonaws.com/aws-cli/awscli-bundle.zip" -o "awscli-bundle.zip"
sudo apt-get install unzip
unzip awscli-bundle.zip
sudo ./awscli-bundle/install -i /usr/local/aws -b /usr/local/bin/aws
Get the tags for the current instance:
aws ec2 describe-tags --filters "Name=resource-id,Values=`ec2metadata --instance-id`"
Outputs:
{
"Tags": [
{
"ResourceType": "instance",
"ResourceId": "i-6a7e559d",
"Value": "Webserver",
"Key": "Name"
}
]
}
Use a bit of perl to extract the tags:
aws ec2 describe-tags --filters \
"Name=resource-id,Values=`ec2metadata --instance-id`" | \
perl -ne 'print "$1\n" if /\"Value\": \"(.*?)\"/'
Returns:
Webserver
For those crazy enough to use Fish shell on EC2, here's a handy snippet for your /home/ec2-user/.config/fish/config.fish. The hostdata command now will list all your tags as well as the public IP and hostname.
set -x INSTANCE_ID (wget -qO- http://instance-data/latest/meta-data/instance-id)
set -x REGION (wget -qO- http://instance-data/latest/meta-data/placement/availability-zone | sed 's/.$//')
function hostdata
aws ec2 describe-tags --region $REGION --filter "Name=resource-id,Values=$INSTANCE_ID" --output=text | sed -r 's/TAGS\t(.*)\t.*\t.*\t(.*)/\1="\2"/'
ec2-metadata | grep public-hostname
ec2-metadata | grep public-ipv4
end