I'm trying to make a groovy script that list the objects on the AWS S3 that have been uploaded in the past three days. I installed the AWS CLI on the agent that the script runs on. The command I found that lists the objects by date is the following:
def cmd = "aws s3api list-objects --bucket (name of bucket) --query \"Contents[?LastModified>= '2018-10-16'].{Key: Key, LastModified: LastModified }\""
When I run this command on the agent directly from a putty session, it runs fine and lists the objects correctly. But when I try to execute the same command from the groovy script, I get the following error:
Bad value for --query "Contents[?LastModified: Bad jmespath expression: Unclosed " delimiter:
"Contents[?LastModified
^
I tried to replace the first and last quotation marks with single quotes but did not work. I tried to do the same thing with the quotation marks before contents and after LastModified but did not work as well. I tried passing Contents[?LastModified>= '2018-10-16'].{Key: Key, LastModified: LastModified } to a string variable and pass its value in the command after --query but that didn't work as well.
Please try:
Then try:
def date = new Date().format('yyyy-MM-dd')
def cmd = ['aws', 's3api', 'list-objects', '--bucket', 'Bucket-Name', '--query', "Contents[?LastModified>='${date}'].{Key: Key , LastModified: LastModified}"]
Remember to always pass the command as a list, not string.
Related
I have a lambda function that describes instances that are running in AWS Account, but when I have scaled out instances by using Auto Scaling, the Lambda function returning the wrong number of instances.
To check for this:
I have used the same logic and created a CLI command and the CLI command gives me the correct number of instances.
CLI command:
aws ec2 describe-instances --filters Name=instance-state-name,Values=running --query 'Reservations[].Instances[].{Instance:InstanceId}' --output json --region -eu-west-1
After that created a python script which I have executed in the server but this is giving the same answer as of lambda function.
Putting Lambda Function Code:
client = boto3.client('ec2',region_name = 'eu-west-1')
response_dict = client.describe_instances(Filters=[
{
'Name': 'instance-state-name',
'Values':['running']
}
])
instances_list = response_dict['Reservations']
result_dict={}
for instance_dict in instances_list:
instanceDetailsresult_dict = {}
instance_role=None
instance = instance_dict['Instances'][0]
instanceId = instance['InstanceId']
print(instanceId)
Note:
This is just a code snippet, all libraries are included.
This is a list: instance_dict['Instances'][0] and you are only getting the first instance out of that list. I suggest iterating over that list.
I'm running this shell command using groovy (which worked in bash):
aws --profile profileName --region us-east-1 dynamodb update-item --table-name tableName --key '{"group_name": {"S": "group_1"}}' --attribute-updates '{"attr1": {"Value": {"S": "STOP"},"Action": "PUT"}}'
This updates the value of an item to STOP in DynamoDB. In my groovy script, I'm running this command like so:
String command = "aws --profile profileName --region us-east-1 dynamodb update-item --table-name tableName --key '{\"group_name\": {\"S\": \"group_1\"}}' --attribute-updates '{\"attr1\": {\"Value\": {\"S\": \"STOP\"},\"Action\": \"PUT\"}}'"
println(command.execute().text)
When I run this with groovy afile.groovy, nothing is printed out and when I check the table in DynamoDB, it's not updated to STOP. There is something wrong with the way I'm escaping the quotes but I'm not sure what. Would appreciate any insights.
Sidenote: When I do a simple aws command like aws s3 ls it works and prints out the results so it's something with this particular command that is throwing it off.
You don't quote for groovy (and the underlying exec) -- you would have to quote for your shell. The execute() on a String does not work like a shell - the underlyting code just splits at whitespace - any quotes are just passed down as part of the argument.
Use ["aws", "--profile", profile, ..., "--key", '{"group_name": ...', ...].execute() and ignore any quoting.
And instead of banging strings together to generate JSON, use groovy.json.JsonOutput.toJson([group_name: [S: "group_1"]])
How to list object using delimeter and sort_by in aws s3 api?
I used below command to list object using delimeter to print second level folder only -
aws s3api list-objects-v2 --bucket $bucketNameToUse --prefix $prefixToUse --output text --delimiter "/"
COMMONPREFIXES firstlevelfolder1/
COMMONPREFIXES firstlevelfolder2/
It worked fine and print second level folders based on prefix that I passed.
And I used below command to list the objects based on last modified date.
aws s3api list-objects-v2 --bucket $bucketNameToUse --prefix $prefixToUse --output text --query 'sort_by(Contents,&LastModified)[*].[Key,LastModified]'
firstlevelfolder1/ 2018-12-28T11:41:07.000Z
firstlevelfolder2/ 2018-12-28T11:41:18.000Z
firstlevelfolder1/secondlevelfolder1/ 2018-12-28T11:41:30.000Z
firstlevelfolder2/secondlevelfolder2/ 2018-12-28T11:41:43.000Z
firstlevelfolder1/secondlevelfolder1/test.java 2018-12-28T11:42:28.000Z
firstlevelfolder2/secondlevelfolder2/test.java 2018-12-28T11:42:46.000Z
it worked fine and print the objects sorted by time.
But when I tried to combine both the command I got error -
aws s3api list-objects-v2 --bucket $bucketNameToUse --prefix $prefixToUse --output text --query 'sort_by(Contents,&LastModified)[*].[Key,LastModified]' --delimiter "/"
In function sort_by(), invalid type for value: None, expected one of: ['array'], received: "null"
Combination of both delimeter and sort_by by lastModified is not possible because the output of delimeter doesn't contain anything except object name and when we combine both delimeter with sort_by function it throws an error as it does not find any content to sort.
Your command is probably returning 0 objects (None value here) but the sort_by method will expect an array
Check if this returns objects or not:
aws s3api list-objects-v2 --bucket $bucketNameToUse --prefix $prefixToUse --output text --delimiter "/"
If this does not then that's your issue.
I am using AWS Command Line Interface (CLI) to list some AMI Images from AWS.
The Name of an Image is like:
XY_XYZ_Docker_1.13_XYZ_XXYY
When using
aws ec2 describe-images --filters 'Name=name,Values="*_Docker_1.13_*"'
it works as expected.
Now i want to use Regular Expression instead of static value for the Name-Filter.
In the AWS-Docs I read that filtering by RegEx is possible
My approach is:
1:
aws ec2 describe-images --filters 'Name=name,Values="[_]Docker[_][0-9][.][0-9]{2}[_]"'
The result is always null for this. I tried different ways of quoting the RegEx.
2:
[_]Docker[_][0-9][.][0-9]{2}[_]
(without quotes) leads to
Error parsing parameter '--filters': Expected: ',', received: 'D' for input:
Name=name,Values=[]Docker[][0-9][.][0-9]{2}[_]
3:
*[_]Docker[_][0-9][.][0-9]{2}[_]*
(with Asterisk) leads to
Error parsing parameter '--filters': Expected: ',', received: ']' for input:
Name=name,Values=[_]Docker[_][0-9][.][0-9]{2}[_]
I wasn't able to find if Jmespath or the --filters flag can support regex, so instead I just piped to Python to run through regex.
aws ec2 describe-images --filters 'Name=name,Values="*Docker*"' | \
python -c '
import json, sys, re
obj = json.load(sys.stdin)
matched_images = {"Images":[]}
for image in obj["Images"]:
if len(re.findall(r"[Dd]ocker\s?[0-9][.][0-9]{2}", image["Name"])) > 0:
matched_images["Images"].append(image)
print json.dumps(matched_images)
'
You can pipe the output (which is just a JSON string) to your next bash command if needed with a pipe character following the closing quote. Maybe this can address concerns with using grep since it returns a JSON string instead or regular text.
See the gist below.
It covers:
search ECR images sorted descending by imagePushDate
selecting the tag that meets a Regex criteria
using that to replace a key/value pair in a yaml
https://gist.github.com/pprogrammingg/69e7c85abede9822f2480e9b5e1e66fd
I am getting an extra None in aws-cli (version 1.11.160) with --query parameter and --output text when fetching the first element of the query output.
See the examples below.
$ aws kms list-aliases --query "Aliases[?contains(AliasName,'alias/foo')].TargetKeyId|[0]" --output text
a3a1f9d8-a4de-4d0e-803e-137d633df24a
None
$ aws kms list-aliases --query "Aliases[?contains(AliasName,'alias/foo-bar')].TargetKeyId|[0]" --output text
None
None
As far as I know this was working till yesterday but from today onwards this extra None comes in and killing our ansible tasks.
Anyone experienced anything similar?
Thanks
I started having this issue in the past few days too. In my case I was querying exports from a cfn stack.
My solution was (since I'll only ever get one result from the query) to change | [0].Value to .Value, which works with --output text.
Some examples:
$ aws cloudformation list-exports --query 'Exports[?Name==`kms-key-arn`] | []'
[
{
"ExportingStackId": "arn:aws:cloudformation:ap-southeast-2:111122223333:stack/stack-name/83ea7f30-ba0b-11e8-8b7d-50fae957fc4a",
"Name": "kms-key-arn",
"Value": "arn:aws:kms:ap-southeast-2:111122223333:key/a13a4bad-672e-45a3-99c2-c646a9470ffa"
}
]
$ aws cloudformation list-exports --query 'Exports[?Name==`kms-key-arn`] | [].Value'
[
"arn:aws:kms:ap-southeast-2:111122223333:key/a13a4bad-672e-45a3-99c2-c646a9470ffa"
]
$ aws cloudformation list-exports --query 'Exports[?Name==`kms-key-arn`] | [].Value' --output text
arn:aws:kms:ap-southeast-2:111122223333:key/a13a4bad-672e-45a3-99c2-c646a9470ffa
aws cloudformation list-exports --query 'Exports[?Name==`kms-key-arn`] | [0].Value' --output text
arn:aws:kms:ap-southeast-2:111122223333:key/a13a4bad-672e-45a3-99c2-c646a9470ffa
None
I'm no closer to finding out why it's happening, but it disproves #LHWizard's theory, or at least indicates there are conditions where that explanation isn't sufficient.
The best explanation is that not every match for your query statement has a TargetKeyId. On my account, there are several Aliases that only have AliasArn and AliasName key/value pairs. The None comes from a null value for TargetKeyId, in other words.
I came across the same issue when listing step functions. I consider it to be a bug. I don't like solutions that ignore the first or last element, expecting it will always be None at that position - at some stage the issue will get fixed and your workaround has introduced a nasty bug.
So, in my case, I did this as a safe workaround (adapt to your needs):
#!/usr/bin/env bash
arn="<step function arn goes here>"
arns=()
for arn in $(aws stepfunctions list-executions --state-machine-arn "$arn" --max-items 50 --query 'executions[].executionArn' --output text); do
[[ $arn == 'None' ]] || arns+=("$arn")
done
# process execution arns
for arn in "${arns[#]}"; do
echo "$arn" # or whatever
done
Supposing you need only the first value:
Replace --output text with --output json and you could parsed with jq
Therefore, you'll have something like
Ps. the -r option with jq is to remove the quotes around the response
aws kms list-aliases --query "Aliases[?contains(AliasName,'alias/foo')].TargetKeyId|[0]" --output | jq -r '.'