I'm having an error when I run a command to extract data to a csv file, using the AWS CLI with jq.
Command:
aws dynamodb scan --table-name MyTable --select ALL_ATTRIBUTES --page-size 500 --max-items 100000 --output json --profile production | jq -r '.Items' | jq -r '(.[0] | keys_unsorted) as $keys | $keys, map([.[ $keys[] ].S])[] | #csv' > export.my-table.csv
Error:
'charmap' codec can't encode characters in position 1-3: character maps to <undefined> parse error: Unfinished JSON term at EOF at line 5097, column 21
I believe that is a query that I wrote previously that does not work on nested attributes. You will have to modify it accordingly.
Related
I am using the below query:
aws ec2 describe-instances | jq -r '.Reservations[]|.Instances[]|[(.Tags[]?|select(.Key=="Name")|.Value), (.Tags[]?|select(.Key=="Group-Name")|.Value),.InstanceId,.PrivateIpAddress]|#csv'|sort
Which outputs as follows:
"sit-test1-zoo-1","i-01205c55a999bebbf","10.153.XX.XXX"
"sit-test2-zoo-2","i-064167c876934448","10.153.XX.XXX"
But I wanted to slide in the date the instance was created and the launch date. I can't seem to figure out the expected syntax having put .Launchdate and .Created in various places within the command. Can anyone please help?
I have come up with the below that produces output (sadly, the same output) and I feel like this is a step in the direction I need to go in, but obvs it does not give the columns I want to see...
aws ec2 describe-instances | jq -r '.Reservations[]|.Instances[] | select(.LaunchTime > "2015-01-28")|[(.Tags[]?|select(.Key=="Name")|.Value), (.Tags[]?|select(.Key=="Group-Name")|.Value),.InstanceId,.PrivateIpAddress,.Launchtime.Value]|#csv'|sort
The key is LaunchTime where you tried with Launchtime.
Here is the working query that will print launch time
aws ec2 describe-instances | jq -r '.Reservations[]|.Instances[] | select(.LaunchTime > "2015-01-28")|[(.Tags[]?|select(.Key=="Name")|.Value), (.Tags[]?|select(.Key=="Group-Name")|.Value),.InstanceId,.PrivateIpAddress,.LaunchTime]|#csv'|sort
Output
"demo","i-1234","10.0.4.54","2020-02-24T10:25:48+00:00"
Also, I will suggest to use jmespath-query instead of jq, Here is the same output without sorting etc
aws ec2 describe-instances --query 'Reservations[].Instances[].{InstanceId:InstanceId,LaunchTime:LaunchTime,Tags:Tags[?Key==`Name`].Value|[0],PrivateIpAddress:PrivateIpAddress}' --output table
Output
------------------------------------------------------------------------------------------------------------------------------
| DescribeInstances |
+---------------------+----------------------------+-------------------+-----------------------------------------------------+
| InstanceId | LaunchTime | PrivateIpAddress | Tags |
+---------------------+----------------------------+-------------------+-----------------------------------------------------+
| i-1234234234dsf | 2017-09-25T22:18:20+00:00 | 10.0.0.243 | demp |
aws-cli-cheatsheet
From EC2 console terminal, I am trying to list a part of an S3 bucket directory names into an array excluding a prefix "date=" but cannot figure out a complete solution.
I've already tried the following code and getting close:
origin="bucket/path/to/my/directory/"
for path in $(aws s3 ls $origin --profile crossaccount --recursive | awk '{print $4}');
do echo "$path"; done
note: directory contains multiple directories like /date=YYYYMMDD/ and all I want to be returned into an array is the YYYYMMDD where YYYYMMDD is >= a certain value.
I expect the output to be an array:
YYYYMMDD, YYYYMMDD, YYYYMMDD
actual result is:
path/to/my/directory/date=YYYYMMDD/file#1
path/to/my/directory/date=YYYYMMDD/file#2
path/to/my/directory/date=YYYYMMDD/file#3
https://docs.aws.amazon.com/cli/latest/reference/s3/ls.html
path="bucket/path/to/my/directory/date="
for i in $(aws s3 ls $path --profile crossaccount --recursive | awk -F'[=/]' '{if($6>20190000)print $6}');
do python3.6 my_python_program.py $i; done
I used awk. In the bracket are the column delimiters =/, and $6 is the 6th column after the directory full name has been delimited. It gave me the date I needed to feed into my python program.
I currently have below AWS CLI command:
$ /usr/local/bin/aws ec2 describe-instances --profile test --region eu-central-1 --query 'Reservations[].Instances[].[InstanceType,Tags[?Key==`Name`].Value[],Tags[?Key==`DomainName`].Value[]]' --output text
This returns:
nat.mgmt.
mgmt.
t2.micro
But I'd like to see it printed on the same line split by tabs or commas.
If you'd like to see everything in one line split by commas, you can write something like this:
/usr/local/bin/aws ec2 describe-instances --region eu-central-1 --query 'Reservations[].Instances[].[InstanceType,Tags[?Key==`Name`].Value[],Tags[?Key==`DomainName`].Value[]]' --output text | paste -sd "," -
Same with tabs:
/usr/local/bin/aws ec2 describe-instances --region eu-central-1 --query 'Reservations[].Instances[].[InstanceType,Tags[?Key==`Name`].Value[],Tags[?Key==`DomainName`].Value[]]' --output text | paste -sd "\t" -
So, the answer is to use bash pipeline and paste
you might have multiple instances returned and you could use awk to merge lines into one.
In you case, 3 values retuned for one instance, so we could merge 3 lines into 1
comma delimiter, add
| awk '{line=line "," $0} NR%3==0{print substr(line,2); line=""}'
tab delimiter, add
| awk '{line=line "\t" $0} NR%3==0{print substr(line,2); line=""}'
so that you have the information for one instance in each line. if you want to extract more information, you can change the 3 in NR%3 to the number of information you want to extract
In an aws cli jmespath query, with for example the output ["a","a","b","a","b"], how do i extract the unique values of it to get ["a","b"]?
Unfortunately this is not currently possible in jmespath.
It's not what you asked for but I've used the following:
aws ... | jq -r ".[]" | sort | uniq
This will convert ["a", "a", "b", "a"] to:
a
b
The closest I've come to "unique values"... is to deduplicate outside of JMESPath (so not really in JMESPath pipelines).
aws ec2 describe-images \
--region us-east-1 \
--filter "Name=architecture,Values=x86_64" \
--query 'Images[].ImageOwnerAlias | join(`"\n"`, #)' \
--output text \
| sort -u
Output:
amazon
aws-marketplace
If you use JMESPath standalone, you'd write things like this.
jp -u -f myjson.json 'Images[].ImageOwnerAlias | join(`"\n"`, #)' | sort -u
The idea is to get jp to spit out a list of values (on separate lines) and then apply all the power of your favorite sorter. The tricky part is to get the list (of course).
Using aws-cli 1.3.6 I am trying to get a simple table of my ec2 instances with the Name and state. I have been looking at the --query and JMESpath documentation and I have been able to select the "Value" item of a Map which "Key" item is equal to Name. This is useful to get the instance-name. Therefore, the code below seems to work
aws ec2 describe-instances --output table --query 'Reservations[].Instances[].Tags[?Key==`Name`].Value'
And delivers this:
-------------------
|DescribeInstances|
+-----------------+
| Name1 |
| Name2 |
+-----------------+
However, if I want to add the state, things get not as I would have expected. Using
aws ec2 describe-instances --output table --query 'Reservations[].Instances[].[Tags[?Key==`Name`].Value,State.Name]'
Delivers
-------------------
|DescribeInstances|
+-----------------+
| Name1 |
| stopped |
| Name2 |
| stopped |
+-----------------+
instead of a two column table with name and state.
If we turn the output to JSON, we can see that the Tags selection returns a list (one-element list) and that's probably the issue:
[
[
[
"Name1"
],
"stopped"
],
[
[
"Name2"
],
"stopped"
]
]
I have not been able to turn this list into an scalar by selecting the first element. This, does not work. Returns an empty list as the Name.
aws ec2 describe-instances --output json --query 'Reservations[].Instances[].[Tags[?Key==`Name`].Value[0],State.Name]'
The same as this
aws ec2 describe-instances --output json --query 'Reservations[].Instances[].[Tags[?Key==`Name`].Value[][0],State.Name]'
The only way I have figured out of addressing this is by means of the join function. Since I only expect one element, it is ok but I seems to be a little bit hacky.
aws ec2 describe-instances --output table --query 'Reservations[].Instances[].[join(`,`,Tags[?Key==`Name`].Value),State.Name]'
---------------------------
| DescribeInstances |
+-------------+-----------+
| Name1 | stopped |
| Name2 | stopped |
+-------------+-----------+
The question, therefore, is: is there any way of picking the first element of the result of the filter (?Key==XXXX) bearing in mind that suffixing it with [0] seems not to work?
Thanks in advance!
The question, therefore, is: is there any way of picking the first element of the result of the filter (?Key==XXXX) bearing in mind that suffixing it with [0] seems not to work?
The way you phrased this question hints towards the solution in fact, namely Pipe Expressions (only available as of version 1.3.7 of the aws-cli though, hence impossible to figure out at question time):
pipe-expression = expression "|" expression
A pipe expression combines two expressions, separated by the |
character. It is similar to a sub-expression with two important
distinctions:
Any expression can be used on the right hand side. A sub-expression restricts the type of expression that can be used on the right hand
side.
A pipe-expression stops projections on the left hand side for propagating to the right hand side. If the left expression creates a
projection, it does not apply to the right hand side.
The emphasized part is key, as shown in the subsequent examples, notably:
If you instead wanted only the first sub list, ["first1", "second1"],
you can use a pipe-expression:
foo[*].bar[0] -> ["first1", "first2"]
foo[*].bar | [0] -> ["first1", "second1"]
Solution
Thus, applying a pipe expression yields the desired result:
aws ec2 describe-instances --output table \
--query 'Reservations[].Instances[].[Tags[?Key==`Name`] | [0].Value, State.Name]'
----------------------------------
| DescribeInstances |
+--------------------+-----------+
| Name1 | stopped |
| Name2 | stopped |
+--------------------+-----------+
#!/bin/bash
for r in `aws ec2 describe-regions --query Regions[*].RegionName --output text`
do
#echo $r
aws ec2 describe-instances --region $r --query 'Reservations[*].Instances[*].{ID:InstanceId, type:InstanceType, launched:LaunchTime, name:Tags[?Key==`Name`].Value[]}' --output json | jq --arg R $r -r '.[] | .[] | [$R, .ID, .type, .launched, .name[0]] | #csv'
done
Output:
"us-east-1","i-054f8253b9ed0746d","t2.micro","2018-10-31T01:57:52.000Z","xxx"
"us-east-1","i-0638792b8b3057ce2","t2.nano","2018-10-23T03:49:24.000Z","yyy"
It tells the Instance ID along with the server state
Command:
aws ec2 describe-instances --filter Name=tag:Name,Values=eep --query 'Reservations[*].Instances[*].{id:State,ID:InstanceId}' --output table
Query part in the above command
It changes as per the requirement
--query 'Reservations[*].Instances[*].{id:State,ID:InstanceId}' --output table