Is there a way to get describe-images to return list of AMIs sorted by time?
Right now it seems to sort them randomly (whereas the AWS Console shows me sorted by time each time I log in -- probably because I clicked the time column to sort it by that once and this preference is being persisted).
I've searched the doc for sorting (i.e., "sort", "order", ..) and cannot seem to find it: https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-images.html
Thanks
The sorting feature is generic to the CLI, not specific to the describe-instances command.
You can accomplish this using jmespath querying.
https://docs.aws.amazon.com/cli/latest/userguide/controlling-output.html
http://jmespath.org/specification.html#func-sort-by
Related
I need to get the list of all items under Platform key in a certain Dynamodb Table called JKOwner using AWS CLI. This is what I'm currently trying to test:
aws dynamodb get-item --table-name JKOwner --key '{"Platform": {"S": "GOVTEST"}}'
Using this cli command, I am able to get the item GOVTEST under the Platform key. But what command should I use if I need to get all items aside from GOVTEST so I could store it in a variable? There are around 200+ Platform values which I need to get into the list. The details of the item doesn't really matter as of now. Thanks in advance for your help.
If you truly need to get all items, you should use a Scan operation. Then you use a filter expression to omit items where platform = GOVTEST items.
If you want to make it really easy on yourself, you could use PartiQL for DynamoDB and do something like:
SELECT * FROM JKOwner WHERE Platform <> 'GOVTEST'
For an example of doing PartiQL on the command line, look on the PartiQL for DynamoDB tab on this page or do this:
aws dynamodb execute-statement --statement "SELECT * FROM Music \
WHERE Artist='Acme Band' AND SongTitle='Happy Day'"
Have you tried?
aws dynamodb query
--table-name JKOwner
--key-condition-expression "Platform = :v1"
--expression-attribute-values "{ \":v1\" : { \"S\" : \"GOVTEST\" } }"
I wish to use an aws cli commad that will return a list of dynamodb tables in my aws account where the CreationDateTime is less than a certain value. CreationDateTime is a property that is shown with the describe-table command, but the list-tables command only returns the names of the table. Is there any way I could possibly use a query to filter the names returned by list-tables in accordance with their corresponding CreationDateTime?
As noted, the answer is simply no, the AWS CLI can not do what you want in one query.
You need to resort to shell scripting to chain the necessary aws commands, or,
if you are willing to give up the hard requirement that it must be the AWS CLI, an alternative solution (and probably faster than forking off aws over and over) is to use a 5 line python script to get the result you want. You can still run this from the command-line, but it won't be the AWS CLI specifically.
Just perform the time arithmetic that suits your specific selection criteria.
#!/usr/bin/env python3
import boto3
from datetime import datetime, timedelta, timezone
ddb = boto3.resource('dynamodb')
for t in sorted(filter(lambda table: datetime.now(timezone.utc) - table.creation_date_time < timedelta(days=90), ddb.tables.all()), key=lambda table: table.creation_date_time):
print(f"{t.name}, {str(t.creation_date_time)}") # tables created in the last 90 days
No - As you noted, ListTables only lists the names of tables, and there is no request that provides additional information for each table, let alone filter on such information. You'll need to use ListTables and then DescribeTable on each of those tables. You can run all of those DescribeTable requests in parallel to make the whole thing much faster (although there is a practical snag - to do it in parallel, you'll need to have opened a bunch of connections to the server).
I'm trying to write a script that will hunt out all IAM service keys that have existed longer than 89 days to meet our security requirements, but I'm typing myself into knots.
My best attempt so far:
gcloud iam service-accounts keys list --quiet --managed-by=user --iam-account $SERVICE_ACCOUNT_EMAIL --filter='createTime<=P89D' --format='csv[no-heading](KEY_ID)'
But this appears to catch all of the keys. I'm struggling to get my head around Google's filter configurations. Any pointers gladly accepted.
The underlying REST method is projects.serviceAccounts.keys.list and the result is a ServiceAccountKey which contains valid[Before|After]Time which are strings in the protobuf Timestamp.
So, I think this needs to either be a string comparison of dates (!) or comparing durations (but I'm unfamiliar with the duration format and how to compare).
You can convert the validAfterTime to a duration, i.e. --filter=validAfterTime.duration() (see duration) and then compare (!) but as Durations
Or construct a date that's within your scope and compare as-is. The following is hacky, please proceed with caution:
PROJECT=...
ACCOUNT=...
PAST=$(date -d "-90 days" +%Y-%m-%dT%H:%M:%SZ)
EMAIL="${ACCOUNT}#${PROJECT}.iam.gserviceaccount.com"
gcloud iam service-accounts keys list \
--iam-account=${EMAIL} \
--project=${PROJECT} \
--filter="validAfterTime<${PAST}"
I think there's a better way to do this!
I've ended up using the above method with ISO dates generated in a script and it seems to be working now. It feels like the kind of thing that should be nicely handled with the filters, but getting it working is taking more time than a bash date
I want to query items from S3 within a specific subdirectory in a bucket by the date/time that they were added to S3. I haven't been able to find any explicit documentation around this, so I'm wondering how it can be accomplished?
The types of queries I want to perform look like this...
Return the URL of the most recently created file in S3 bucket images under the directory images/user1/
Return the URLs of all items created between datetime X and datetime Y in the S3 bucket images under the directory images/user1
Update 3/19/2019
Apparently s3api allows you to do this quite easily
One solution would probably to use the s3api. It works easily if you have less than 1000 objects, otherwise you need to work with pagination.
s3api can list all objects and has a property for the lastmodified attribute of keys imported in s3. It can then be sorted, find files after or before a date, matching a date ...
Examples of running such option
all files for a given date
DATE=$(date +%Y-%m-%d)
aws s3api list-objects-v2 --bucket test-bucket-fh --query 'Contents[?
contains(LastModified, `$DATE`)]'
all files after a certain date
export YESTERDAY=`date -v-1w +%F`
aws s3api list-objects-v2 --bucket test-bucket-fh --query 'Contents[?
LastModified > `$YESTERDAY`)]'
s3api will return a few metadata so you can filter for specific elements
DATE=$(date +%Y-%m-%d)
aws s3api list-objects-v2 --bucket test-bucket-fh --query 'Contents[?contains(LastModified, `$DATE`)].Key'
OLD ANSWER
AWS-SDK/CLI really should implement some sort of retrieve-by-date flag, it would make life easier and cheaper.
If you have not prefixed/labelled your files with the dates, you may also want to try using the flag
--start-after (string)
If you know the latest file you want to start listing from, you can use the list-objects-v2 command with the --start-after flag.
"StartAfter is where you want Amazon S3 to start listing from. Amazon S3 starts listing after this specified key. StartAfter can be any key in the bucket"
So --start-after will continually get your objects, so if you would like to limit the number of items try specifying a --max-items flag.
https://docs.aws.amazon.com/cli/latest/reference/s3api/list-objects-v2.html
S3 can list all objects in a bucket, or all objects with a prefix (such as a "directory"). However, this isn't a cheap operation, it's certainly not designed to be done on every request.
Generally speaking, you are best served by a database layer for this. It can be something light and fast (like redis), but you should know what objects you have and which one you need for a given request.
You can somewhat cheat by copying objects twice- for instance, images/latest.jpg or images/user1/latest.jpg. But in the "date query" example, you should certainly do this external to S3.
You could store the files prefixed by date in the final directory eg:
images/user1/2016-01-12_{actual file name}
Then in the script that is doing the querying you can generate the list of dates in the time period and construct the prefixes accordingly and query S3 for all the dates separately and meagre the results. It should be way faster than getting full list and filtering the LastModified field (well that depends how many files you have in given dir, I think than anything less than a 1000 is not worth the effort.)
There is actually better method with use of 'Marker' parameter in listObjects call, so you set the marker to a key and listObjects will return only keys witch are after that one in the directory. We do have dates and time in the key names.
I've got a very large bucket (hundreds of thousands of objects). I've got a path (lets say s3://myBucket/path1/path2). /path2 gets uploads that are also folders. So a sample might look like:
s3://myBucket/path1/path2/v6.1.0
s3://myBucket/path1/path2/v6.1.1
s3://myBucket/path1/path2/v6.1.102
s3://myBucket/path1/path2/v6.1.2
s3://myBucket/path1/path2/v6.1.25
s3://myBucket/path1/path2/v6.1.99
S3 doesn't take into account version number sorting (which makes sense) but alphabetically the last in the list is not the last uploaded. In that example .../v6.1.102 is the newest.
Here's what I've got so far:
aws s3api list-objects
--bucket myBucket
--query "sort_by(Contents[?contains(Key, \`path1/path2\`)],&LastModified)"ยด
--max-items 20000
So one problem here is max-items seems to start alphabetically from the all files recursively in the bucket. 20000 does get to my files but it's a pretty slow process to go through that many files.
So my questions are twofold:
1 - This is still searching the whole bucket but I just want to narrow it down to path2/ . Can I do this?
2 - This lists just objects, is it possible to pull up just a path list instead?
Basically the end goal is I just want a command to return the newest folder name like 'v6.1.102' from the example above.
To answer #1, you could add the --prefix path1/path2 to limit what you're querying in the bucket.
In terms of sorting by last modified, I can only think of using an SDK to combine the list_objects_v2 and head_object (boto3) to get last modified on the objects and programmatically sort
Update
Alternatively, you could reverse sort by LastModified in jmespath and return the first item to give you the most recent object and gather the directory from there.
aws s3api list-objects-v2 \
--bucket myBucket \
--prefix path1/path2 \
--query 'reverse(sort_by(Contents,&LastModified))[0]'
If you want general purpose querying e.g. "lowest version", "highest version", "all v6.x versions" then consider maintaining a separate database with the version numbers.
If you only need to know the highest version number and you need that to be retrieved quickly (quicker than a list object call) then you could maintain that version number independently. For example, you could use a Lambda function that responds to objects being uploaded to path1/path2 where the Lambda function is responsible for storing the highest version number that it has seen into a file at s3://mybucket/version.max.
Prefix works with list_object using boto3 client. But using boto3 resource might give some issues. Paginator in pagination is a great concept and works nice!. to find the latest changes(additions of objects) : sort_by(contents)[-1]