The question is, how to easily fetch sensitive information from AWS Secret Manager within Bash scripts?To get the response form aws cli command it's quite straightforward:
json_value=$(aws secretsmanager get-secret-value --secret-id "$1")
The problem is, the response is returned in json format, and it will take some space to deserialize and parse all the parameters. Is there any easy way to do it?
If you have stored the secrets as simple strings, you can retrieve them using
aws secretsmanager get-secret-value --secret-id "$SECRET_ID" --query "SecretString" --output text
I know it's Q&A, just wanted to share with you very handy bash function to get all the information in a very convenient way(python on instance required).
# Usage Ex. exportSecrets <Secrets-Name> <Key-Name-1> <Key-Name-2>...
exportSecrets() {
local json_value;
json_value=$(aws secretsmanager get-secret-value --secret-id "$1")
echo "------->"
printf "Secrets RESULT. Json: \n%s\n" "$json_value"
shift; local json_keys=("$#")
fetchJson() {
python - "$json_value" "$json_keys" <<EOF
import json, sys
secrets = json.loads(json.loads(
sys.argv[1])['SecretString']
)
ans = []
for k in sys.argv[2].split(' '):
ans.append(secrets[k])
print(' '.join(ans))
EOF
}
SECRETS=$(fetchJson)
echo "------->"
printf "Resolved Secrets: \n%s\n" "$SECRETS"
}
Now with above, you can simple call the function with params and get back exported variable with response in list for next usage.
exportSecrets "YOUR-KEY-STORAGE" "KEY-NAME-1" "KEY-NAME-2"
local key1=$(echo $SECRETS | cut -d' ' -f1)
echo $key1
local key2=$(echo $SECRETS | cut -d' ' -f2)
echo $key2
Related
I want to return only the current AWS username using AWS CLI. I'm on Windows 11. I think there's a way to do it using a regex but I can't figure out how. I think I need to use a pipe along with a regex but there's no related examples on the JMESPath website. I want to have something like "only return the text after 'user/' ".
Here's what I have so far:
aws sts get-caller-identity --output text --query 'Arn'
which returns `"arn:aws:iam::999999009999:user/joe.smith"
I just want to return "joe.smith".
jmes does not support splitting a string nor matching a substring in a string, so you'll have to resort to a native command like:
> (aws sts get-caller-identity --output text --query 'Arn').Split("/")[-1]
or use something like jq :
$ aws sts get-caller-identity --output json | jq '.Arn | split("/")[-1]' -r
I am trying to read credentials from assume role like AcccessKeyID and store in a variable but getting error:
My code and error is:
jq -r '".Credentials.AccessKeyId"' mysession.json | awk '"{print "set","AWS_ACCESS_KEY_ID="$0}"' > variables
jq: error: syntax error, unexpected INVALID_CHARACTER, expecting $end (Windows cmd shell quoting issues?) at , line 1:
'".Credentials.AccessKeyId"'
jq: 1 compile error
awk: '"{print
awk: ^ invalid char ''' in expression
Please suggest me how to achieve this activity in windows CMD .I have installed jq and awk in windows.
aws sts assume-role --role-arn role_arn --role-session-name session_name > mysession.json
$ak = jq -r ".Credentials.AccessKeyId" mysession.json
$sk = jq -r ".Credentials.SecretAccessKey" mysession.json
$tk = jq -r ".Credentials.SessionToken" mysession.json
Write-Host "Acccess Key ID:" $ak
Write-Host "Secret Acccess Key:" $sk
Write-Host "Session Token:" $tk
Powershell
$source_profile = "default"
$region = "ap-southeast-2"
$role_arn = "arn:aws:iam::account_id:role/role-test"
$target_profile = "test"
$target_profile_path = "$HOME\.aws\credentials"
$session_name = "test"
# Assume Role
$Response = (Use-STSRole -Region $region -RoleArn $role_arn -RoleSessionName $session_name -ProfileName $source_profile).Credentials
# Export Crendentail as environment variable
$env:AWS_ACCESS_KEY_ID=$Response.AccessKeyId
$env:AWS_SECRET_ACCESS_KEY=$Response.SecretAccessKey
$env:AWS_SESSION_TOKEN=$Response.SessionToken
# Create Profile with Credentials
Set-AWSCredential -StoreAs $target_profile -ProfileLocation $target_profile_path -AccessKey $Response.AccessKeyId -SecretKey $Response.SecretAccessKey -SessionToken $Response.SessionToken
# Print expiration time
Write-Host("Credentials will expire at: " + $Response.Expiration)
AWS Assume Role Script
How can I parse an assumed role's credentials in powershell and set them as a variable in a script?
On the jq site it mentions syntax adjustments for Windows:
"when using the Windows command shell (cmd.exe) it's best to use
double quotes around your jq program when given on the command-line
(instead of the -f program-file option), but then double-quotes in the
jq program need backslash escaping."
So, instead of
jq -r '".Credentials.AccessKeyId"' mysession.json
You'll need to escape double quotes, then change single quotes to double.
jq -r "\".Credentials.AccessKeyId\"" mysession.json
I can get the details with
$ aws lambda get-function --function-name random_number
{
"Configuration": {
"FunctionName": "random_number",
"FunctionArn": "arn:aws:lambda:us-east-2:193693970645:function:random_number",
"Runtime": "ruby2.5",
"Role": "arn:aws:iam::193693970645:role/service-role/random_number-role-8cy8a1a7",
...
But how can get just a couple of fields like function name ?
I tried:
$ aws lambda get-function --function-name random_number --query "Configuration[*].[FunctionName]"
but I get null
Your overall approach is correct, you just need to adjust the query:
$ aws lambda get-function --function-name random_number \
--query "Configuration.FunctionName" --output text
I also added a parameter to convert the result to text, which makes processing a bit easier.
Here is a simple awk (standard Linux gnu awk) script that does the trick: Extract the values of quoted field #3, only for line having /FunctionName/.
awk 'BEGIN {FPAT="\"[^\"]+"}/FunctionName/{print substr($3,2)}'
Piped with your initial command:
$ aws lambda get-function --function-name random_number | awk 'BEGIN {FPAT="\"[^\"]+"}/FunctionName/{print substr($3,2)}'
One way to achieve that is by using jq.
therefore, the output must be JSON.
From the docs :
jq is like sed for JSON data - you can use it to slice and filter and
map and transform structured data with the same ease that sed, awk,
grep and friends let you play with text.
Usage example :
aws lambda get-function --function-name test --output json | jq -r '.Configuration.FunctionName'
Use get-function-configuration as in the following:
aws lambda get-function-configuration --function-name MyFunction --query "[FunctionName]"
I have a Jenkins 2.0 job where I require the user to select the list of servers to execute the job against via a Jenkins "Active Choices Reactive Parameter". These servers which the job will execute against are AWS EC2 instances. Instead of hard-coding the available servers in the "Active Choices Reactive Parameter", I'd like to query the AWS CLI to get a list of servers.
A few notes:
I've assigned the Jenkins 2.0 EC2 an IAM role which has sufficient privileges to query AWS via the CLI.
The AWS CLI is installed on the Jenkins EC2.
The "Active Choices Reactive Parameter" will return a list of checkboxes if I hardcode values in a Groovy script, such as:
return ["10.1.1.1", "10.1.1.2", 10.1.1.3"]
I know my awk commands can be improved, I'm not yet sure how, but my primary goal is to get the list of servers dynamically loaded in Jenkins.
I can run the following command directly on the EC2 instance which is hosting Jenkins:
aws ec2 describe-instances --region us-east-2 --filters
"Name=tag:Env,Values=qa" --query
"Reservations[*].Instances[*].PrivateIpAddress" | grep -o
'\"[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\"' | awk
{'printf $0 ", "'} | awk {'print "[" $0'} | awk {'gsub(/^[ \t]+|[
\t]+$/, ""); print'} | awk {'print substr ($0, 1, length($0)-1)'} |
awk {'print $0 "]"'}
This will return the following, which is in the format expected by the "Active Choices Reactive Parameter":
["10.1.1.1", "10.1.1.2", 10.1.1.3"]
So, in the "Script" textarea of the "Active Choices Reactive Parameter", I have the following script. The problem is that my server list is never populated. I've tried numerous variations of this script without luck. Can someone please tell me where I've went wrong and what I can do to correct this script so that my list of server IP addresses is dynamically loaded into a Jenkins job?
def standardOut = new StringBuffer(), standardErr = new StringBuffer()
def command = $/
aws ec2 describe-instances --region us-east-2 --filters "Name=tag:Env,Values=qaint" --query "Reservations[*].Instances[*].PrivateIpAddress" |
grep -o '\"[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\"' |
awk {'printf $0 ", "'} |
awk {'print "[" $0'} |
awk {'gsub(/^[ \t]+|[ \t]+$/, ""); print'} |
awk {'print substr ($0, 1, length($0)-1)'} |
awk {'print $0 "]"'}
/$
def proc = command.execute()
proc.consumeProcessOutput(standardOut, standardErr)
proc.waitForOrKill(1000)
return standardOut
I tried to execute your script and the standardErr had some errors, Looks like groovy didn't like the double quotes in the AWS CLI. Here is a cleaner way to do without using awk
def command = 'aws ec2 describe-instances \
--filters Name=tag:Name,Values=Test \
--query Reservations[*].Instances[*].PrivateIpAddress \
--output text'
def proc = command.execute()
proc.waitFor()
def output = proc.in.text
def exitcode= proc.exitValue()
def error = proc.err.text
if (error) {
println "Std Err: ${error}"
println "Process exit code: ${exitcode}"
return exitcode
}
//println output.split()
return output.split()
This script works with Jenkins Active Choices Parameter, and returns the list of IP addresses:
def aws_cmd = 'aws ec2 describe-instances \
--filters Name=instance-state-name,Values=running \
Name=tag:env,Values=dev \
--query Reservations[].Instances[].PrivateIpAddress[] \
--region us-east-2 \
--output text'
def aws_cmd_output = aws_cmd.execute()
// probably is required if execution takes long
//aws_cmd_output.waitFor()
def ip_list = aws_cmd_output.text.tokenize()
return ip_list
I am getting an extra None in aws-cli (version 1.11.160) with --query parameter and --output text when fetching the first element of the query output.
See the examples below.
$ aws kms list-aliases --query "Aliases[?contains(AliasName,'alias/foo')].TargetKeyId|[0]" --output text
a3a1f9d8-a4de-4d0e-803e-137d633df24a
None
$ aws kms list-aliases --query "Aliases[?contains(AliasName,'alias/foo-bar')].TargetKeyId|[0]" --output text
None
None
As far as I know this was working till yesterday but from today onwards this extra None comes in and killing our ansible tasks.
Anyone experienced anything similar?
Thanks
I started having this issue in the past few days too. In my case I was querying exports from a cfn stack.
My solution was (since I'll only ever get one result from the query) to change | [0].Value to .Value, which works with --output text.
Some examples:
$ aws cloudformation list-exports --query 'Exports[?Name==`kms-key-arn`] | []'
[
{
"ExportingStackId": "arn:aws:cloudformation:ap-southeast-2:111122223333:stack/stack-name/83ea7f30-ba0b-11e8-8b7d-50fae957fc4a",
"Name": "kms-key-arn",
"Value": "arn:aws:kms:ap-southeast-2:111122223333:key/a13a4bad-672e-45a3-99c2-c646a9470ffa"
}
]
$ aws cloudformation list-exports --query 'Exports[?Name==`kms-key-arn`] | [].Value'
[
"arn:aws:kms:ap-southeast-2:111122223333:key/a13a4bad-672e-45a3-99c2-c646a9470ffa"
]
$ aws cloudformation list-exports --query 'Exports[?Name==`kms-key-arn`] | [].Value' --output text
arn:aws:kms:ap-southeast-2:111122223333:key/a13a4bad-672e-45a3-99c2-c646a9470ffa
aws cloudformation list-exports --query 'Exports[?Name==`kms-key-arn`] | [0].Value' --output text
arn:aws:kms:ap-southeast-2:111122223333:key/a13a4bad-672e-45a3-99c2-c646a9470ffa
None
I'm no closer to finding out why it's happening, but it disproves #LHWizard's theory, or at least indicates there are conditions where that explanation isn't sufficient.
The best explanation is that not every match for your query statement has a TargetKeyId. On my account, there are several Aliases that only have AliasArn and AliasName key/value pairs. The None comes from a null value for TargetKeyId, in other words.
I came across the same issue when listing step functions. I consider it to be a bug. I don't like solutions that ignore the first or last element, expecting it will always be None at that position - at some stage the issue will get fixed and your workaround has introduced a nasty bug.
So, in my case, I did this as a safe workaround (adapt to your needs):
#!/usr/bin/env bash
arn="<step function arn goes here>"
arns=()
for arn in $(aws stepfunctions list-executions --state-machine-arn "$arn" --max-items 50 --query 'executions[].executionArn' --output text); do
[[ $arn == 'None' ]] || arns+=("$arn")
done
# process execution arns
for arn in "${arns[#]}"; do
echo "$arn" # or whatever
done
Supposing you need only the first value:
Replace --output text with --output json and you could parsed with jq
Therefore, you'll have something like
Ps. the -r option with jq is to remove the quotes around the response
aws kms list-aliases --query "Aliases[?contains(AliasName,'alias/foo')].TargetKeyId|[0]" --output | jq -r '.'