I am trying to read credentials from assume role like AcccessKeyID and store in a variable but getting error:
My code and error is:
jq -r '".Credentials.AccessKeyId"' mysession.json | awk '"{print "set","AWS_ACCESS_KEY_ID="$0}"' > variables
jq: error: syntax error, unexpected INVALID_CHARACTER, expecting $end (Windows cmd shell quoting issues?) at , line 1:
'".Credentials.AccessKeyId"'
jq: 1 compile error
awk: '"{print
awk: ^ invalid char ''' in expression
Please suggest me how to achieve this activity in windows CMD .I have installed jq and awk in windows.
aws sts assume-role --role-arn role_arn --role-session-name session_name > mysession.json
$ak = jq -r ".Credentials.AccessKeyId" mysession.json
$sk = jq -r ".Credentials.SecretAccessKey" mysession.json
$tk = jq -r ".Credentials.SessionToken" mysession.json
Write-Host "Acccess Key ID:" $ak
Write-Host "Secret Acccess Key:" $sk
Write-Host "Session Token:" $tk
Powershell
$source_profile = "default"
$region = "ap-southeast-2"
$role_arn = "arn:aws:iam::account_id:role/role-test"
$target_profile = "test"
$target_profile_path = "$HOME\.aws\credentials"
$session_name = "test"
# Assume Role
$Response = (Use-STSRole -Region $region -RoleArn $role_arn -RoleSessionName $session_name -ProfileName $source_profile).Credentials
# Export Crendentail as environment variable
$env:AWS_ACCESS_KEY_ID=$Response.AccessKeyId
$env:AWS_SECRET_ACCESS_KEY=$Response.SecretAccessKey
$env:AWS_SESSION_TOKEN=$Response.SessionToken
# Create Profile with Credentials
Set-AWSCredential -StoreAs $target_profile -ProfileLocation $target_profile_path -AccessKey $Response.AccessKeyId -SecretKey $Response.SecretAccessKey -SessionToken $Response.SessionToken
# Print expiration time
Write-Host("Credentials will expire at: " + $Response.Expiration)
AWS Assume Role Script
How can I parse an assumed role's credentials in powershell and set them as a variable in a script?
On the jq site it mentions syntax adjustments for Windows:
"when using the Windows command shell (cmd.exe) it's best to use
double quotes around your jq program when given on the command-line
(instead of the -f program-file option), but then double-quotes in the
jq program need backslash escaping."
So, instead of
jq -r '".Credentials.AccessKeyId"' mysession.json
You'll need to escape double quotes, then change single quotes to double.
jq -r "\".Credentials.AccessKeyId\"" mysession.json
Related
The question is, how to easily fetch sensitive information from AWS Secret Manager within Bash scripts?To get the response form aws cli command it's quite straightforward:
json_value=$(aws secretsmanager get-secret-value --secret-id "$1")
The problem is, the response is returned in json format, and it will take some space to deserialize and parse all the parameters. Is there any easy way to do it?
If you have stored the secrets as simple strings, you can retrieve them using
aws secretsmanager get-secret-value --secret-id "$SECRET_ID" --query "SecretString" --output text
I know it's Q&A, just wanted to share with you very handy bash function to get all the information in a very convenient way(python on instance required).
# Usage Ex. exportSecrets <Secrets-Name> <Key-Name-1> <Key-Name-2>...
exportSecrets() {
local json_value;
json_value=$(aws secretsmanager get-secret-value --secret-id "$1")
echo "------->"
printf "Secrets RESULT. Json: \n%s\n" "$json_value"
shift; local json_keys=("$#")
fetchJson() {
python - "$json_value" "$json_keys" <<EOF
import json, sys
secrets = json.loads(json.loads(
sys.argv[1])['SecretString']
)
ans = []
for k in sys.argv[2].split(' '):
ans.append(secrets[k])
print(' '.join(ans))
EOF
}
SECRETS=$(fetchJson)
echo "------->"
printf "Resolved Secrets: \n%s\n" "$SECRETS"
}
Now with above, you can simple call the function with params and get back exported variable with response in list for next usage.
exportSecrets "YOUR-KEY-STORAGE" "KEY-NAME-1" "KEY-NAME-2"
local key1=$(echo $SECRETS | cut -d' ' -f1)
echo $key1
local key2=$(echo $SECRETS | cut -d' ' -f2)
echo $key2
I am trying to assume an AWS role within a CI/CD pipeline, hence I have to write a script to change the role via a script. Below is the script to do that, and I used source <script>.sh to replace the existing AWS access & secret keys, and add the session key.
I checked that the 3 env variables are there by echoing them in the terminal.
#!/bin/bash
output="/tmp/assume-role-output.json"
aws sts assume-role --role-arn "arn:aws:iam::<account-id>:role/<rolename>" --role-session-name AWSCLI-Session > $output
AccessKeyId=$(cat $output | jq '.Credentials''.AccessKeyId')
SecretAccessKey=$(cat $output | jq '.Credentials''.SecretAccessKey')
SessionToken=$(cat $output | jq '.Credentials''.SessionToken')
export AWS_ACCESS_KEY_ID=$AccessKeyId
export AWS_SECRET_ACCESS_KEY=$SecretAccessKey
export AWS_SESSION_TOKEN=$SessionToken
However, when I tried running a simple aws command to list ECR images aws ecr list-images --registry-id <id> --repository-name <name>, it gave the following error message.
An error occurred (UnrecognizedClientException) when calling the ListImages operation:
The security token included in the request is invalid.
I tried manually setting the AWS keys and token in the terminal, and surprisingly the ecr list command works.
export AWS_ACCESS_KEY_ID="XXX"
export AWS_SECRET_ACCESS_KEY="XXX"
export AWS_SESSION_TOKEN="XXX"
Does anyone know what is wrong with my script?
This is a one-liner without using a file..
OUT=$(aws sts assume-role --role-arn arn:aws:iam::<YOUR_ACCOUNT>:role/<YOUR_ROLENAME> --role-session-name aaa);\
export AWS_ACCESS_KEY_ID=$(echo $OUT | jq -r '.Credentials''.AccessKeyId');\
export AWS_SECRET_ACCESS_KEY=$(echo $OUT | jq -r '.Credentials''.SecretAccessKey');\
export AWS_SESSION_TOKEN=$(echo $OUT | jq -r '.Credentials''.SessionToken');
Might be useful..
Print it to use as bash export on another terminal
printf "export AWS_ACCESS_KEY_ID=\"%s\"\\n" $AWS_ACCESS_KEY_ID;\
printf "export AWS_SECRET_ACCESS_KEY=\"%s\"\\n" $AWS_SECRET_ACCESS_KEY;\
printf "export AWS_SESSION_TOKEN=\"%s\"\\n\\n\\n" $AWS_SESSION_TOKEN;
Print it to use in JSON context
Useful for launch.json on vs code
printf "\"AWS_ACCESS_KEY_ID\":\"$AWS_ACCESS_KEY_ID\",\\n";\
printf "\"AWS_SECRET_ACCESS_KEY\":\"$AWS_SECRET_ACCESS_KEY\",\\n";\
printf "\"AWS_SESSION_TOKEN\":\"$AWS_SESSION_TOKEN\"\\n";
Update
Here is the powershell version
$OUT = aws sts assume-role --role-arn arn:aws:iam::<YOUR_ACCOUNT>:role/<YOUR_ROLENAME> --role-session-name aaa
$JSON_OUT = ConvertFrom-Json "$OUT"
$ACCESS_KEY=$JSON_OUT.Credentials.AccessKeyId
$SECRET_KEY=$JSON_OUT.Credentials.SecretAccessKey
$SESSION_TOKEN=$JSON_OUT.Credentials.SessionToken
"Paste these env variables to your terminal to assume the role"
-join ("`n", '$Env:AWS_ACCESS_KEY_ID="', "$ACCESS_KEY", '"')
-join ('$Env:AWS_SECRET_ACCESS_KEY="', "$SECRET_KEY", '"')
-join ('$Env:AWS_SESSION_TOKEN="', "$SESSION_TOKEN", '"')
If you use jq the way you do, your export values will contain quotation marks, e.g.
"ASIASZHPM3IXQXXOXFOY"
rather then:
ASIASZHPM3IXQXXOXFOY
To avoid this, you have to add -r flag to jq:
AccessKeyId=$(cat $output | jq -r '.Credentials''.AccessKeyId')
SecretAccessKey=$(cat $output | jq -r '.Credentials''.SecretAccessKey')
SessionToken=$(cat $output | jq -r '.Credentials''.SessionToken')
Adding to #carmel's answer
Here's a function that you can add to bashrc/zshrc to set the enviroment variables automatically.
function assume-role() {
OUT=$(aws sts assume-role --role-arn $1 --role-session-name $2);\
export AWS_ACCESS_KEY_ID=$(echo $OUT | jq -r '.Credentials''.AccessKeyId');\
export AWS_SECRET_ACCESS_KEY=$(echo $OUT | jq -r '.Credentials''.SecretAccessKey');\
export AWS_SESSION_TOKEN=$(echo $OUT | jq -r '.Credentials''.SessionToken');
}
This can then be used as:
$ assume-role <role-arn> <role-session-name>
Adding the answer I needed, because I did not have jq available in the container I was using, in case it helps someone out.
You can also use cut to parse the output:
OUT=$(aws sts assume-role --role-arn "arn:aws:iam::<account-id>:role/<role-name>" --role-session-name <session-name>)
export AWS_ACCESS_KEY_ID=$(echo $OUT | cut -d '"' -f 6 )
export AWS_SECRET_ACCESS_KEY=$(echo $OUT | cut -d '"' -f 10 )
export AWS_SESSION_TOKEN=$(echo $OUT | cut -d '"' -f 14 )
Single command to get the result
eval $(aws sts assume-role --role-arn arn:aws:iam::${AWSAccountId}:role/role-name --role-session-name awscli-session | jq -r '.Credentials | "export AWS_ACCESS_KEY_ID=\(.AccessKeyId)\nexport AWS_SECRET_ACCESS_KEY=\(.SecretAccessKey)\nexport AWS_SESSION_TOKEN=\(.SessionToken)\n"')
I have a Jenkins 2.0 job where I require the user to select the list of servers to execute the job against via a Jenkins "Active Choices Reactive Parameter". These servers which the job will execute against are AWS EC2 instances. Instead of hard-coding the available servers in the "Active Choices Reactive Parameter", I'd like to query the AWS CLI to get a list of servers.
A few notes:
I've assigned the Jenkins 2.0 EC2 an IAM role which has sufficient privileges to query AWS via the CLI.
The AWS CLI is installed on the Jenkins EC2.
The "Active Choices Reactive Parameter" will return a list of checkboxes if I hardcode values in a Groovy script, such as:
return ["10.1.1.1", "10.1.1.2", 10.1.1.3"]
I know my awk commands can be improved, I'm not yet sure how, but my primary goal is to get the list of servers dynamically loaded in Jenkins.
I can run the following command directly on the EC2 instance which is hosting Jenkins:
aws ec2 describe-instances --region us-east-2 --filters
"Name=tag:Env,Values=qa" --query
"Reservations[*].Instances[*].PrivateIpAddress" | grep -o
'\"[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\"' | awk
{'printf $0 ", "'} | awk {'print "[" $0'} | awk {'gsub(/^[ \t]+|[
\t]+$/, ""); print'} | awk {'print substr ($0, 1, length($0)-1)'} |
awk {'print $0 "]"'}
This will return the following, which is in the format expected by the "Active Choices Reactive Parameter":
["10.1.1.1", "10.1.1.2", 10.1.1.3"]
So, in the "Script" textarea of the "Active Choices Reactive Parameter", I have the following script. The problem is that my server list is never populated. I've tried numerous variations of this script without luck. Can someone please tell me where I've went wrong and what I can do to correct this script so that my list of server IP addresses is dynamically loaded into a Jenkins job?
def standardOut = new StringBuffer(), standardErr = new StringBuffer()
def command = $/
aws ec2 describe-instances --region us-east-2 --filters "Name=tag:Env,Values=qaint" --query "Reservations[*].Instances[*].PrivateIpAddress" |
grep -o '\"[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\"' |
awk {'printf $0 ", "'} |
awk {'print "[" $0'} |
awk {'gsub(/^[ \t]+|[ \t]+$/, ""); print'} |
awk {'print substr ($0, 1, length($0)-1)'} |
awk {'print $0 "]"'}
/$
def proc = command.execute()
proc.consumeProcessOutput(standardOut, standardErr)
proc.waitForOrKill(1000)
return standardOut
I tried to execute your script and the standardErr had some errors, Looks like groovy didn't like the double quotes in the AWS CLI. Here is a cleaner way to do without using awk
def command = 'aws ec2 describe-instances \
--filters Name=tag:Name,Values=Test \
--query Reservations[*].Instances[*].PrivateIpAddress \
--output text'
def proc = command.execute()
proc.waitFor()
def output = proc.in.text
def exitcode= proc.exitValue()
def error = proc.err.text
if (error) {
println "Std Err: ${error}"
println "Process exit code: ${exitcode}"
return exitcode
}
//println output.split()
return output.split()
This script works with Jenkins Active Choices Parameter, and returns the list of IP addresses:
def aws_cmd = 'aws ec2 describe-instances \
--filters Name=instance-state-name,Values=running \
Name=tag:env,Values=dev \
--query Reservations[].Instances[].PrivateIpAddress[] \
--region us-east-2 \
--output text'
def aws_cmd_output = aws_cmd.execute()
// probably is required if execution takes long
//aws_cmd_output.waitFor()
def ip_list = aws_cmd_output.text.tokenize()
return ip_list
In order to delete a log stream from a log group using the CLI command , individual log stream names are required .
Is there a way to delete all log streams belonging to a log group using a single command?
You can achieve this through using --query to target the results of describe-log-streams. This allows you to loop through and delete the results.
aws logs describe-log-streams --log-group-name $LOG_GROUP_NAME --query 'logStreams[*].logStreamName' --output table | awk '{print $2}' | grep -v ^$ | while read x; do aws logs delete-log-stream --log-group-name $LOG_GROUP_NAME --log-stream-name $x; done
You can use --query to target all or specific groups or streams.
Delete streams from a specific month
aws logs describe-log-streams --log-group-name $LOG_GROUP --query 'logStreams[?starts_with(logStreamName,`2017/07`)].logStreamName' --output table | awk '{print $2}' | grep -v ^$ | while read x; do aws logs delete-log-stream --log-group-name $LOG_GROUP --log-stream-name $x; done
Delete All log groups - Warning, it deletes EVERYTHING!
aws logs describe-log-groups --query 'logGroups[*].logGroupName' --output table | awk '{print $2}' | grep -v ^$ | while read x; do aws logs delete-log-group --log-group-name $x; done
Clearing specific log groups
aws logs describe-log-groups --query 'logGroups[?starts_with(logGroupName,`$LOG_GROUP_NAME`)].logGroupName' --output table | awk '{print $2}' | grep -v ^$ | while read x; do aws logs delete-log-group --log-group-name $x; done
Credit
Implemented script with command from #Stephen's answer. The script shows summary before deletion and tracks progress of deletion.
#!/usr/bin/env bash
LOG_GROUP_NAME=${1:?log group name is not set}
echo Getting stream names...
LOG_STREAMS=$(
aws logs describe-log-streams \
--log-group-name ${LOG_GROUP_NAME} \
--query 'logStreams[*].logStreamName' \
--output table |
awk '{print $2}' |
grep -v ^$ |
grep -v DescribeLogStreams
)
echo These streams will be deleted:
printf "${LOG_STREAMS}\n"
echo Total $(wc -l <<<"${LOG_STREAMS}") streams
echo
while true; do
read -p "Prceed? " yn
case $yn in
[Yy]*) break ;;
[Nn]*) exit ;;
*) echo "Please answer yes or no." ;;
esac
done
for name in ${LOG_STREAMS}; do
printf "Delete stream ${name}... "
aws logs delete-log-stream --log-group-name ${LOG_GROUP_NAME} --log-stream-name ${name} && echo OK || echo Fail
done
Github link
To delete all log streams associated with a specific log group, run the following command, replacing NAME_OF_LOG_GROUP with your group:
aws logs describe-log-streams --log-group-name NAME_OF_LOG_GROUP --output text | awk '{print $7}' | while read x;
do aws logs delete-log-stream --log-group-name NAME_OF_LOG_GROUP --log-stream-name $x
done
Here is Script to delete all logs in a log group using python. Just change the logGroupName to match your logGroup.
import boto3
client = boto3.client('logs')
response = client.describe_log_streams(
logGroupName='/aws/batch/job'
)
def delete_stream(stream):
delete_response = client.delete_log_stream(
logGroupName='/aws/batch/job',
logStreamName=stream['logStreamName']
)
print(delete_response)
results = map(lambda x: delete_stream(x), response['logStreams'])
Based on #german-lashevich's answer
If you have thousands of log streams, you will needed to parallelize.
#!/usr/bin/env bash
LOG_GROUP_NAME=${1:?log group name is not set}
echo Getting stream names...
LOG_STREAMS=$(
aws logs describe-log-streams \
--log-group-name ${LOG_GROUP_NAME} \
--query 'logStreams[*].logStreamName' \
--output table |
awk '{print $2}' |
grep -v ^$ |
grep -v DescribeLogStreams
)
echo These streams will be deleted:
printf "${LOG_STREAMS}\n"
echo Total $(wc -l <<<"${LOG_STREAMS}") streams
echo
while true; do
read -p "Prceed? " yn
case $yn in
[Yy]*) break ;;
[Nn]*) exit ;;
*) echo "Please answer yes or no." ;;
esac
done
step() {
local name=$1
printf "Delete stream ${name}... "
aws logs delete-log-stream --log-group-name ${LOG_GROUP_NAME} --log-stream-name ${name} && echo OK || echo Fail
}
N=20
for name in ${LOG_STREAMS}; do ((i=i%N)); ((i++==0)) && wait ; step "$name" & done
This cannot be done using a single aws Cli command. Hence we achieved this using a script where we first retrieved all the log streams of a log group and then deleted them in a loop.
For Windows users this powershell script could be usefull, to remove all the log streams in a log group:
#Set your log group name
$log_group_name = "/production/log-group-name"
aws logs describe-log-streams --log-group-name $log_group_name --query logStreams --output json | ConvertFrom-json | ForEach-Object {$_.logStreamName} | ForEach-Object {
aws logs delete-log-stream --log-group-name $log_group_name --log-stream-name $_
Write-Host ($_ + " -> deleted") -ForegroundColor Green
}
Just save it as your_script_name.ps1 and execute it in powershell.
An alternative version using Powershell CLI on Windows, launch powershell command line and use:
$LOG_GROUP_NAME="cloud-watch-group-name";
$LOG_STREAM_NAMEP="cloud-watch-log-stream-name";
Set-DefaultAWSRegion -Region us-your-regions;
Set-AWSCredential -AccessKey ACCESSKEYEXAMPLE -SecretKey sEcReTKey/EXamPLE/xxxddddEXAMPLEKEY -StoreAs MyProfileName
Get-CWLLogStream -loggroupname $LOG_GROUP_NAME -logstreamnameprefix $LOG_GROUP_NAMEP | Remove-CWLLogStream -LogGroupName $LOG_GROUP_NAME;
You may use -Force parameter on the Remove-CWLogStream Cmdlet in case you don´t want to confirm one by one.
References
https://docs.aws.amazon.com/powershell/latest/reference/Index.html
The others have already described how you can paginate through all the log streams and delete them one by one.
I would like to offer two alternative ways that have (more or less) the same effect, but don't require you to loop through all the log streams.
Deleting the log group, then re-creating it has the desired effect: All the log streams of the log group will be deleted.
delete-log-group
followed by:
create-log-group
CAVEAT: Deleting a log group can have unintended consequences. For example, subscriptions and the retention policy will be deleted as well, and those have to be restored too when the log group is re-created.
Another workaround is to set a 1 day retention period.
put-retention-policy
It won't have an immediate effect, you will have to wait ca. a day, but after that all the old data will be deleted. The name of the old streams and their meta data (last event time, creation time, etc.) will remain though, but you won't be charged for that (as far as I can tell based on my own bill).
So it is not exactly what you asked for. However, probably the most important reason why one would want to delete all the log streams is to delete the logged data (to reduce costs, or for compliance reasons), and this approach achieves that.
WARNING: Don't forget to change the retention policy after the old data is gone, or you will continually delete data after 1 day, and chances are, it is not what you want in the long run.
If you are doing this in zshell /zsh and you only need simple one liner command then just update the values :
* Pattern
AWS_SECRET_ACCESS_KEY
AWS_ACCESS_KEY_ID
AWS_DEFAULT_REGION
Pattern can any text , you can also add ^ for begging of the line or $ for end of the line.
run the below command !
Pattern="YOUR_PATTERN" && setupKeys="AWS_ACCESS_KEY_ID=YOUR_KEY AWS_SECRET_ACCESS_KEY=YOUR_KEY AWS_DEFAULT_REGION=YOUR_REGION" &&
eval "${setupKeys} aws logs describe-log-groups --query 'logGroups[*].logGroupName' --output table | sed 's/|//g'| sed 's/\s//g'| grep -i ${Pattern} "| while read x; do echo "deleting $x" && $setupKeys aws logs delete-log-group --log-group-name $x; done
--log-group-name is not optional in aws cli, you can try using an * for --log-group-name value (in test environment)
aws logs delete-log-group --log-group-name my-logs
Reference URL:
http://docs.aws.amazon.com/cli/latest/reference/logs/delete-log-group.html
If you are using a prefix, you could use the following command.
aws logs describe-log-streams --log-group-name <log_group_name> --log-stream-name-prefix"<give_a_log_group_prefix>" --query 'logStreams[*].logStreamName' --output table | awk '{print $2}' | grep -v ^$ | while read x; do aws logs delete-log-stream --log-group-name <log_group_name> --log- stream-name $x;done;
I am trying to figure out on what the s3cmd command would be to download files from bucket by date, so for example i have a bucket named "test" and in that bucket there are different files from different dates. I am trying to get the files that were uploaded yesterday. what would the command be?
There is no single command that will allow you to do that. You have to write a script some thing like this. Or use a SDK that allows you to do this. Below script is a sample script that will get S3 files from last 30 days.
#!/bin/bash
# Usage: ./getOld "bucketname" "30 days"
s3cmd ls s3://$1 | while read -r line; do
createDate=`echo $line|awk {'print $1" "$2'}`
createDate=`date -d"$createDate" +%s`
olderThan=`date -d"-$2" +%s`
if [[ $createDate -lt $olderThan ]]
then
fileName=`echo $line|awk {'print $4'}`
echo $fileName
if [[ $fileName != "" ]]
then
s3cmd get "$fileName"
fi
fi
done;
I like s3cmd but to work with single line command, I prefer the JSon output of aws cli and jq JSon processor
The command will look like
aws s3api list-objects --bucket "yourbucket" |\
jq '.Contents[] | select(.LastModified | startswith("yourdate")).Key' --raw-output |\
xargs -I {} aws s3 cp s3://yourbucket/{} .
basically what the script does
list all object from a given bucket
(the interesting part) jq will parse the Contents array and select element where the LastModified value start with your pattern (you will need to change), get the Key of the s3 object and add --raw-output so it strips the quote from the value
pass the result to an aws copy command to download the file from s3
if you want to automate a bit further you can get yesterday from the command line
for mac os
$ export YESTERDAY=`date -v-1w +%F`
$ aws s3api list-objects --bucket "ariba-install" |\
jq '.Contents[] | select(.LastModified | startswith('\"$YESTERDAY\"')).Key' --raw-output |\
xargs -I {} aws s3 cp s3://ariba-install/{} .
for linux os (or other flavor of bash that I am not familiar)
$ export YESTERDAY=`date -d "1 day ago" '+%Y-%m-%d' `
$ aws s3api list-objects --bucket "ariba-install" |\
jq '.Contents[] | select(.LastModified | startswith('\"$YESTERDAY\"')).Key' --raw-output |\
xargs -I {} aws s3 cp s3://ariba-install/{} .
Now you get the idea if you want to change the YESTERDAY variable to have different kind of date