AWS cli JMESpath query tag values - amazon-web-services

As a follow-up to my previous question, I am having trouble getting tags to work right in this type of output. I want to print a table with each property as a row. This is how I expect it to look:
% aws --output table ec2 describe-instances --instance-id $id --query "Reservations[].Instances[0].[{ Property: 'Type', Value: InstanceType }]"
-------------------------------
| DescribeInstances |
+-----------+-----------------+
| Property | Value |
+-----------+-----------------+
| Type | g4dn.12xlarge |
+-----------+-----------------+
But with tag names it looks like this:
% aws --output table ec2 describe-instances --instance-id $id --query "Reservations[].Instances[0].[{ Property: 'Name', Value: Tags[?Key =='Name'].Value }]"
-------------------
|DescribeInstances|
+-----------------+
| Property |
+-----------------+
| Name |
+-----------------+
|| Value ||
|+---------------+|
|| Elliott-TKD ||
|+---------------+|
The tag value is correct, but the formatting is weird, and when combined with more other rows the table gets really ugly.

The filter part of your query ([?Key == 'Name']) is creating what JMESPath is describing as a projection.
You will have to reset this projection in order to extract a single string out of it.
Resetting a projection can be achieved using pipes.
Projections are an important concept in JMESPath. However, there are times when projection semantics are not what you want. A common scenario is when you want to operate of the result of a projection rather than projecting an expression onto each element in the array.
For example, the expression people[*].first will give you an array containing the first names of everyone in the people array. What if you wanted the first element in that list? If you tried people[*].first[0] that you just evaluate first[0] for each element in the people array, and because indexing is not defined for strings, the final result would be an empty array, []. To accomplish the desired result, you can use a pipe expression, <expression> | <expression>, to indicate that a projection must stop.
Source: https://jmespath.org/tutorial.html#pipe-expressions
So your issue is very close to what they are describing here in the documentation and the reset of that projection can be achieved using:
Tags[?Key =='Name']|[0].Value
or, with:
Tags[?Key =='Name'].Value | [0]
which are two strictly identical queries.
Given the JSON:
{
"Reservations": [
{
"Instances": [
{
"Tags": [
{
"Key": "Name",
"Value": "Elliott-TKD"
},
{
"Key": "Foo",
"Value": "Bar"
}
]
}
]
}
]
}
The query
Reservations[].Instances[0].[{ Property: `Name`, Value: Tags[?Key == `Name`]|[0].Value }]
Will give the expected
[
[
{
"Property": "Name",
"Value": "Elliott-TKD"
}
]
]
So it will render properly in your table
------------------------------
| DescribeInstance |
+------------+---------------+
| Property | Value |
+------------+---------------+
| Name | Elliott-TKD |
+------------+---------------+

Related

BigQuery - JSONpath recursive operator (1/2)

Is there any way to perform a recursive search on a JSON string object in BigQuery in absence of the operator ".." which is apparently not supported ?
Motivation: access "name" without knowing "students" and "class" in the below.
Query
SELECT JSON_EXTRACT(json_text, '$..name') AS first_student
FROM UNNEST([
'{"class" : {"students" : {"name" : "Jane"}}}'
]) AS json_text;
Desired output
+-----------------+
| first_student |
+-----------------+
| "Jane" |
+-----------------+
Current output
Unsupported operator in JSONPath: ..
Try below
SELECT REGEXP_EXTRACT(json_text, r'"name" : "(\w+)"') AS first_student
FROM UNNEST([
'{"class" : {"students" : {"name" : "Jane"}}}'
]) AS json_text;
with output

Trying to add launchtime and created date to aws cli query

I am using the below query:
aws ec2 describe-instances | jq -r '.Reservations[]|.Instances[]|[(.Tags[]?|select(.Key=="Name")|.Value), (.Tags[]?|select(.Key=="Group-Name")|.Value),.InstanceId,.PrivateIpAddress]|#csv'|sort
Which outputs as follows:
"sit-test1-zoo-1","i-01205c55a999bebbf","10.153.XX.XXX"
"sit-test2-zoo-2","i-064167c876934448","10.153.XX.XXX"
But I wanted to slide in the date the instance was created and the launch date. I can't seem to figure out the expected syntax having put .Launchdate and .Created in various places within the command. Can anyone please help?
I have come up with the below that produces output (sadly, the same output) and I feel like this is a step in the direction I need to go in, but obvs it does not give the columns I want to see...
aws ec2 describe-instances | jq -r '.Reservations[]|.Instances[] | select(.LaunchTime > "2015-01-28")|[(.Tags[]?|select(.Key=="Name")|.Value), (.Tags[]?|select(.Key=="Group-Name")|.Value),.InstanceId,.PrivateIpAddress,.Launchtime.Value]|#csv'|sort
The key is LaunchTime where you tried with Launchtime.
Here is the working query that will print launch time
aws ec2 describe-instances | jq -r '.Reservations[]|.Instances[] | select(.LaunchTime > "2015-01-28")|[(.Tags[]?|select(.Key=="Name")|.Value), (.Tags[]?|select(.Key=="Group-Name")|.Value),.InstanceId,.PrivateIpAddress,.LaunchTime]|#csv'|sort
Output
"demo","i-1234","10.0.4.54","2020-02-24T10:25:48+00:00"
Also, I will suggest to use jmespath-query instead of jq, Here is the same output without sorting etc
aws ec2 describe-instances --query 'Reservations[].Instances[].{InstanceId:InstanceId,LaunchTime:LaunchTime,Tags:Tags[?Key==`Name`].Value|[0],PrivateIpAddress:PrivateIpAddress}' --output table
Output
------------------------------------------------------------------------------------------------------------------------------
| DescribeInstances |
+---------------------+----------------------------+-------------------+-----------------------------------------------------+
| InstanceId | LaunchTime | PrivateIpAddress | Tags |
+---------------------+----------------------------+-------------------+-----------------------------------------------------+
| i-1234234234dsf | 2017-09-25T22:18:20+00:00 | 10.0.0.243 | demp |
aws-cli-cheatsheet

Filtering by tags not working when using aws ec2 describe-instances from command line (cli)

I'm currently attempting to write an aws ec2 query from the command line (in AWS Linux, not that it should matter). I am trying to set a filter that matches both of the following:
Shows those instances that are in the off state (code 80), and;
Shows those instances who have a tag "ShortPurpose" whose value is "Fleet".
What is actually happening is that all instances in the off state are being returned, regardless of if they have the tag "ShortPurpose":"Fleet" set.
My instances are set up as so:
+-------------+--------------+------------------------+--+
| Instance ID | Tag | Tag Value | |
+-------------+--------------+------------------------+--+
| i-09876 | ShortPurpose | Fleet | |
| | Organisation | UmbrellaCorp | |
| | Name | cloud-01 | |
| | Owner | ORG-UMBR-ELLA | |
| | Purpose | Cloud processing fleet | |
+-------------+--------------+------------------------+--+
| | | | |
| i-12345 | (no tags) | | |
| | | | |
+-------------+--------------+------------------------+--+
The command I am using is:
aws ec2 describe-instances --query "Reservations[*].Instances[*].InstanceId" --filters "Name=tag:ShortPurpose,Values=Fleet,Name=instance-state-code,Values=80"
The results are the standard array style response. The instance state is successfully filtered upon, but not the tags.
I tried to verify your command and it produces errors as you wrote it:
Error parsing parameter '--filters': Second instance of key "Name" encountered for input:
Name=tag:ShortPurpose,Values=Fleet,Name=instance-state-code,Values=80
^
This is often because there is a preceeding "," instead of a space.
However, I was able to successful use it on my sandbox instances as follows:
aws ec2 describe-instances \
--query "Reservations[*].Instances[*].InstanceId" \
--filters Name=tag:ShortPurpose,Values=Fleet Name=instance-state-code,Values=80
I have discovered in one example in the AWS doumentation that I had the wrong query format. The correct query is:
aws ec2 describe-instances --query "Reservations[*].Instances[*].InstanceId" --filters "Name=tag-value,Values=Fleet" "Name=instance-state-code,Values=80"
Note that I'm ignoring the ShortPurpose tag and instead hunting directly for the value, which may exist in any tag.

django rawsql postgres nested json/jsonb query

I have a jsonb structure on postgres named data where each row (there are around 3 million of them) looks like this:
[
{
"number": 100,
"key": "this-is-your-key",
"listr": "20 Purple block, THE-CITY, Columbia",
"realcode": "LA40",
"ainfo": {
"city": "THE-CITY",
"county": "Columbia",
"street": "20 Purple block",
"var_1": ""
},
"booleanval": true,
"min_address": "20 Purple block, THE-CITY, Columbia LA40"
},
.....
]
I would like to query the min_address field in the fastest possible way. In Django I tried to use:
APModel.objects.filter(data__0__min_address__icontains=search_term)
but this takes ages to complete (also, "THE-CITY" is in uppercase, so, I have to use icontains here. I tried dropping to rawsql like so:
cursor.execute("""\
SELECT * FROM "apmodel_ap_model"
WHERE ("apmodel_ap_model"."data"
#>> array['0', 'min_address'])
#> %s \
""",\
[json.dumps([{'min_address': search_term}])]
)
but this throws me strange errors like:
LINE 4: #> '[{"min_address": "some lane"}]'
^
HINT: No operator matches the given name and argument type(s). You might need to add explicit type casts.
I am wondering what is the fastest way I can query the field min_address by using rawsql cursors.
Late answer, probably it won't help OP anymore. Also I'm not at all an expert in Postgres/JSONB, so this might be a terrible idea.
Given this setup;
so49263641=# \d apmodel_ap_model;
Table "public.apmodel_ap_model"
Column | Type | Collation | Nullable | Default
--------+-------+-----------+----------+---------
data | jsonb | | |
so49263641=# select * from apmodel_ap_model ;
data
-------------------------------------------------------------------------------------------
[{"number": 1, "min_address": "Columbia"}, {"number": 2, "min_address": "colorado"}]
[{"number": 3, "min_address": " columbia "}, {"number": 4, "min_address": "California"}]
(2 rows)
The following query "expands" objects from data arrays to individual rows. Then it applies pattern matching to the min_address field.
so49263641=# SELECT element->'number' as number, element->'min_address' as min_address
FROM apmodel_ap_model ap, JSONB_ARRAY_ELEMENTS(ap.data) element
WHERE element->>'min_address' ILIKE '%col%';
number | min_address
--------+---------------
1 | "Columbia"
2 | "colorado"
3 | " columbia "
(3 rows)
However, I doubt it will perform well on large datasets as the min_address values are casted to text before pattern matching.
Edit: Some great advice here on indexing JSONB data for search https://stackoverflow.com/a/33028467/1284043

Listing instance name among other data with aws-cli 1.3.6

Using aws-cli 1.3.6 I am trying to get a simple table of my ec2 instances with the Name and state. I have been looking at the --query and JMESpath documentation and I have been able to select the "Value" item of a Map which "Key" item is equal to Name. This is useful to get the instance-name. Therefore, the code below seems to work
aws ec2 describe-instances --output table --query 'Reservations[].Instances[].Tags[?Key==`Name`].Value'
And delivers this:
-------------------
|DescribeInstances|
+-----------------+
| Name1 |
| Name2 |
+-----------------+
However, if I want to add the state, things get not as I would have expected. Using
aws ec2 describe-instances --output table --query 'Reservations[].Instances[].[Tags[?Key==`Name`].Value,State.Name]'
Delivers
-------------------
|DescribeInstances|
+-----------------+
| Name1 |
| stopped |
| Name2 |
| stopped |
+-----------------+
instead of a two column table with name and state.
If we turn the output to JSON, we can see that the Tags selection returns a list (one-element list) and that's probably the issue:
[
[
[
"Name1"
],
"stopped"
],
[
[
"Name2"
],
"stopped"
]
]
I have not been able to turn this list into an scalar by selecting the first element. This, does not work. Returns an empty list as the Name.
aws ec2 describe-instances --output json --query 'Reservations[].Instances[].[Tags[?Key==`Name`].Value[0],State.Name]'
The same as this
aws ec2 describe-instances --output json --query 'Reservations[].Instances[].[Tags[?Key==`Name`].Value[][0],State.Name]'
The only way I have figured out of addressing this is by means of the join function. Since I only expect one element, it is ok but I seems to be a little bit hacky.
aws ec2 describe-instances --output table --query 'Reservations[].Instances[].[join(`,`,Tags[?Key==`Name`].Value),State.Name]'
---------------------------
| DescribeInstances |
+-------------+-----------+
| Name1 | stopped |
| Name2 | stopped |
+-------------+-----------+
The question, therefore, is: is there any way of picking the first element of the result of the filter (?Key==XXXX) bearing in mind that suffixing it with [0] seems not to work?
Thanks in advance!
The question, therefore, is: is there any way of picking the first element of the result of the filter (?Key==XXXX) bearing in mind that suffixing it with [0] seems not to work?
The way you phrased this question hints towards the solution in fact, namely Pipe Expressions (only available as of version 1.3.7 of the aws-cli though, hence impossible to figure out at question time):
pipe-expression = expression "|" expression
A pipe expression combines two expressions, separated by the |
character. It is similar to a sub-expression with two important
distinctions:
Any expression can be used on the right hand side. A sub-expression restricts the type of expression that can be used on the right hand
side.
A pipe-expression stops projections on the left hand side for propagating to the right hand side. If the left expression creates a
projection, it does not apply to the right hand side.
The emphasized part is key, as shown in the subsequent examples, notably:
If you instead wanted only the first sub list, ["first1", "second1"],
you can use a pipe-expression:
foo[*].bar[0] -> ["first1", "first2"]
foo[*].bar | [0] -> ["first1", "second1"]
Solution
Thus, applying a pipe expression yields the desired result:
aws ec2 describe-instances --output table \
--query 'Reservations[].Instances[].[Tags[?Key==`Name`] | [0].Value, State.Name]'
----------------------------------
| DescribeInstances |
+--------------------+-----------+
| Name1 | stopped |
| Name2 | stopped |
+--------------------+-----------+
#!/bin/bash
for r in `aws ec2 describe-regions --query Regions[*].RegionName --output text`
do
#echo $r
aws ec2 describe-instances --region $r --query 'Reservations[*].Instances[*].{ID:InstanceId, type:InstanceType, launched:LaunchTime, name:Tags[?Key==`Name`].Value[]}' --output json | jq --arg R $r -r '.[] | .[] | [$R, .ID, .type, .launched, .name[0]] | #csv'
done
Output:
"us-east-1","i-054f8253b9ed0746d","t2.micro","2018-10-31T01:57:52.000Z","xxx"
"us-east-1","i-0638792b8b3057ce2","t2.nano","2018-10-23T03:49:24.000Z","yyy"
It tells the Instance ID along with the server state
Command:
aws ec2 describe-instances --filter Name=tag:Name,Values=eep --query 'Reservations[*].Instances[*].{id:State,ID:InstanceId}' --output table
Query part in the above command
It changes as per the requirement
--query 'Reservations[*].Instances[*].{id:State,ID:InstanceId}' --output table