Determine velocity template request body property type? - amazon-web-services

I have an API Gateway that uses velocity templates as a thin wrapper to allow users to do CRUD operations on a DynamoDB table.
I'm trying to write the update operation as dynamically as possible, but where I'm stuck is with determining type from the request body's properties from within the velocity template. This is what I'm working with:
#set($body = $input.path('$'))
#set($updateExpression = "set")
#set($expressionAttributeNames = "")
#set($expressionAttributeValues = "")
#foreach($attrName in $body.keySet())
#set($updateExpression = "${updateExpression} #$attrName = :${attrName},")
#set($expressionAttributeNames = "${expressionAttributeNames}""#${attrName}"":""${attrName}""")
#set($attrValue = $input.json("$.${attrName}"))
#if($attrValue.matches("^-?\\d+$"))
#set($attrValue = """:${attrName}"": { ""N"": ${attrValue}, ")
#else
#set($attrValue = """:${attrName}"": { ""S"": """ + $util.escapeJavaScript($attrValue) + """ },")
#end
#set($expressionAttributeValues = "${expressionAttributeValues} ${attrValue}")
#if($foreach.hasNext)
#set($expressionAttributeNames = "${expressionAttributeNames}, ")
#end
#end
{
"TableName": "TABLE",
"Key": { "id": { "S": "$input.params('id')" } },
"UpdateExpression": "${updateExpression} updatedOn = :updatedOn",
"ExpressionAttributeNames": {$expressionAttributeNames},
"ExpressionAttributeValues": {
$expressionAttributeValues
":updatedOn": { "N": "$context.requestTimeEpoch" }
}
}
Edit: This would be a sample request body:
https://api/v1/endpoint/123
{
"location": {
"lat": 42,
"lon": -71
},
"rating": 4
}
This is the current transformation I get:
{
"TableName": "users",
"Key": { "gcn": { "S": "123" } },
"UpdateExpression": "set #number = :number, #location = :location, updatedOn = :updatedOn",
"ExpressionAttributeNames": {"#number":"number", "#location":"location"},
"ExpressionAttributeValues": {
":number": { "S": "1" }, ":location": { "S": "{\"lat\":26.89199858375187,\"lon\":75.77141155196833}" },
":updatedOn": { "N": "" }
}
}
I currently just have a test for checking if a value is a number...and it isn't working.

After doing some more digging I reached what I set out for. I have a dynamic Velocity Template mapping for AWS API Gateway for the purpose of updating DynamoDB items.
So far it supports strings, numbers, booleans, and string-escaped objects, as that's how my project stores them (they are not query-able). ExpressionAttributeNames exists in case you use a reserved keyword for an attribute name...like I did for 'location'.
If anyone has any improvements/enhancements please let me know, it's a beast of a script.
#set($body = $input.path('$'))
#set($updateExpression = "set")
#set($expressionAttributeNames = "")
#set($expressionAttributeValues = "")
#foreach($attrName in $body.keySet())
#set($updateExpression = "${updateExpression} #$attrName = :${attrName},")
#set($expressionAttributeNames = "${expressionAttributeNames}""#${attrName}"":""${attrName}""")
#set($attrValue = $input.json("$.${attrName}"))
#if($attrValue.toString().matches("[+-]?\d+"))
#set($attrValue = """:${attrName}"": { ""N"": ""${attrValue}"" }, ")
#elseif($attrValue.toString() == "true" || $attrValue.toString() == "false")
#set($attrValue = """:${attrName}"": { ""BOOL"": ${attrValue} }, ")
#elseif(($attrValue.toString().startsWith("{") && $attrValue.toString().endsWith("}")) ||
($attrValue.toString().startsWith("[") && $attrValue.toString().endsWith("]")) )
#set($attrValue = """:${attrName}"": { ""S"": """ + $util.escapeJavaScript($attrValue) + """ },")
#else
#set($attrValue = """:${attrName}"": { ""S"": " + $attrValue + " },")
#end
#set($expressionAttributeValues = "${expressionAttributeValues} ${attrValue}")
#if($foreach.hasNext)
#set($expressionAttributeNames = "${expressionAttributeNames}, ")
#end
#end
{
"TableName": "", ## Insert your table here.
"Key": { "gcn": { "S": "$input.params('')" } }, ## Insert your key expression here.
## Update below if `updatedOn` is not your audit attribute.
"UpdateExpression": "${updateExpression} updatedOn = :updatedOn",
"ExpressionAttributeNames": {$expressionAttributeNames},
"ExpressionAttributeValues": {
$expressionAttributeValues
":updatedOn": { "N": "$context.requestTimeEpoch.toString()" }
}
}
Sample Request Body:
{
"firstName": "John",
"isActive": true,
"_status": 1
}
Sample Transformation:
{
"TableName": "users",
"Key": {
"id": {
"S": "1"
}
},
"UpdateExpression": "set #firstName = :firstName, #isActive = :isActive, #_status = :_status, updatedOn = :updatedOn",
"ExpressionAttributeNames": {
"#firstName": "firstName",
"#isActive": "isActive",
"#_status": "_status"
},
"ExpressionAttributeValues": {
":firstName": {
"S": "John"
},
":isActive": {
"BOOL": true
},
":_status": {
"N": "1"
},
":updatedOn": {
"N": "123456789"
}
}
}

Related

Application Insights Data Limited Importing Into PowerBI

I'm attempting to import data from Azure Application Insights into PowerBI. The issue is that, regardless of the timespan I set, I seem to only be pulling about a week's worth of data. Here's what the M query looks like:
let AnalyticsQuery =
let Source = Json.Document(Web.Contents("https://api.applicationinsights.io/v1/apps/<uuid>/query",
[Query=[#"query"="customEvents
| project customDimensions
",#"x-ms-app"="AAPBI",#"timespan"="P30D"],Timeout=#duration(0,0,60,0)])),
TypeMap = #table(
{ "AnalyticsTypes", "Type" },
{
{ "string", Text.Type },
{ "int", Int32.Type },
{ "long", Int64.Type },
{ "real", Double.Type },
{ "timespan", Duration.Type },
{ "datetime", DateTimeZone.Type },
{ "bool", Logical.Type },
{ "guid", Text.Type },
{ "dynamic", Text.Type }
}),
DataTable = Source[tables]{0},
Columns = Table.FromRecords(DataTable[columns]),
ColumnsWithType = Table.Join(Columns, {"type"}, TypeMap , {"AnalyticsTypes"}),
Rows = Table.FromRows(DataTable[rows], Columns[name]),
Table = Table.TransformColumnTypes(Rows, Table.ToList(ColumnsWithType, (c) => { c{0}, c{3}}))
in
Table
in
AnalyticsQuery
I was thinking this was a size issue, but I've already narrowed it down to a single column (albeit a wide one) and it's still not returning any more data.
Narrowing the returned dataset to two columns has increased the dataset to include a few weeks instead of less than a week, but I'm still looking for a bigger dataset. Here's the latest query:
let AnalyticsQuery =
let Source = Json.Document(Web.Contents("https://api.applicationinsights.io/v1/apps/<uuid>/query",
[Query=[#"query"="customEvents
| extend d=parse_json(customDimensions)
| project timestamp, d[""Properties""]
| order by timestamp desc
| where timestamp <= now() and d_Properties <> """"
",#"x-ms-app"="AAPBI"],Timeout=#duration(0,0,4,0)])),
TypeMap = #table(
{ "AnalyticsTypes", "Type" },
{
{ "string", Text.Type },
{ "int", Int32.Type },
{ "long", Int64.Type },
{ "real", Double.Type },
{ "timespan", Duration.Type },
{ "datetime", DateTimeZone.Type },
{ "bool", Logical.Type },
{ "guid", Text.Type },
{ "dynamic", Text.Type }
}),
DataTable = Source[tables]{0},
Columns = Table.FromRecords(DataTable[columns]),
ColumnsWithType = Table.Join(Columns, {"type"}, TypeMap , {"AnalyticsTypes"}),
Rows = Table.FromRows(DataTable[rows], Columns[name]),
Table = Table.TransformColumnTypes(Rows, Table.ToList(ColumnsWithType, (c) => { c{0}, c{3}}))
in
Table,
#"Sorted Rows" = Table.Sort(AnalyticsQuery,{{"timestamp", Order.Ascending}})
in
#"Sorted Rows"
You should look into either table buffering or directquery: see this discussion

How to fix this error in my appsync while creating an array of data

I am creating an array of date and in my resolver it only return one output of date and count audited
I search throughout the google to find some answer and I found a code how to make an array of list but the problem is it didn't return well
https://imgur.com/a/1eDknYN
this is a result and the code I use it attached in the picture
#set ($tu = 0) #set ($pc = 0) #set ($fc = 0) #set ($da = [])#set ($cda = []) #foreach($item in $ctx.result.items)
#set($tu = $item.total_audits + $tu)
#set($pc = $item.passed_compliance + $pc)
#set($fc = $item.failed_compliance + $fc)
#set($date = $item.sort)
#set($count = $item.total_audits)
$util.qr($da.add("$date"))
$util.qr($cda.add("$count"))
#end
$util.toJson({"total_audits":$tu, "passed_compliance":$pc,
"failed_compliance":$fc, "daily_audit": [{"date": $da, "count": $cda}]})
here is the error
"errors": [
{
"path": [
"getAuditSummary",
"daily_audit",
0,
"date"
],
"locations": null,
"message": "Can't serialize value (/getAuditSummary/daily_audit[0]/date) : Unable to serialize `[2018-12-26, 2018-12-27, 2018-12-28]` as a valid date."
},
{
"path": [
"getAuditSummary",
"daily_audit",
0,
"count"
],
"locations": null,
"message": "Can't serialize value (/getAuditSummary/daily_audit[0]/count) : Expected type 'Int' but was 'ArrayList'."
what i want to make is it would return something like this
"daily_audit": [
{
"date": 2018-12-26,
"count": 1
}
{
"date": 2018-12-27,
"count": 4
}
{
"date": 2018-12-28,
"count": 2
}
]
This is happening because $da and $cda are arrays. So it's probably returning:
"daily_audit": [
{
"date": [2018-12-26,2018-12-27,2018-12-28],
"count": [1,4,2]
}
]
So, in your response mapping template, you can try something like:
#set ($tu = 0)
#set ($pc = 0)
#set ($fc = 0)
#set ($da = [])
#foreach($item in $ctx.result.items)
#set($tu = $item.total_audits + $tu)
#set($pc = $item.passed_compliance + $pc)
#set($fc = $item.failed_compliance + $fc)
#set($date = $item.sort)
#set($count = $item.total_audits)
$util.qr($da.add({"date":$date, "count":$count}))
#end
$util.toJson({"total_audits":$tu, "passed_compliance":$pc, "failed_compliance":$fc, "daily_audit": $da})

Using python re regex in MongoDB $filter condition

I have a document like below:
{
_id: 1,
data: [ { zip: 001, city: "abc" }, { zip: 002, city: "xyz" } ]
}
I want to filter data array using python regex. But it doesn't seem to be working.
city = "abc"
regx = re.compile("^%s$" %city, re.IGNORECASE|re.MULTILINE)
for doc in db.testusers.aggregate([ { "$project": { "data": { "$filter": { "input": "$data", "as": "item", "cond": { "$eq": [ "$$item.city", regx ] } } } } } ]):
json.dumps(doc)
It doesn't match anything.
Am I doing it right?
I think $filter does not support regex. See doc.
I cannot test this here but it should work like according to this sample:
city_list = ["cityAbc", "Metroid"]
city_list = [re.compile("^" + str(c_id) + "$", re.IGNORECASE) for c_id in city_list]
pipe = [ { "$match" : { "_id":{"$in" : city_list}}},
{ "$unwind" : "$rp"},
{"$group":{"_id": "$_id", "rp": { "$push": "$rp" }}} , {"$limit":500}]
res = list(db.coll.aggregate(pipeline = pipe,allowDiskUse=True))

How to concatenate method request parameters in body mapping template AWS API Gateway

I am trying to retrieve a list of entries from AWS DynamoDB with a primary hash and sort key composite.
The idea is to retrieve a list which matches a given query string value which is a part of the sort key, for example, the sort keys can be a_b_c, a_d_e, a_f_g and i need to get the all entries with b in it.
I am asking for three query strings from the client and concatenating them in the body mapping template in the integration request section of AWS API gateway.
I am searching the web to accomplish the same but haven't been successful in finding a solution.
It would save a lot of time if somebody could help me with this.
Below is my approach.
{
"TableName": "$util.escapeJavaScript("$context.stage\_TableName").replaceAll("\\","")",
"FilterExpression": "identityId = :v1 and aWithbWithc = :v2 + _ + :v3 + _ + :v4",
"ExpressionAttributeValues": {
":v1": {
"S": "$context.identity.cognitoIdentityId"
},
":v2" : {
"S" : "$input.params('a')"
},
":v3" : {
"S" : "$input.params('b')"
},
":v4" : {
"S" : "$input.params('c')"
}
}
}
I have found the solution and putting it out here.
{
"TableName": "$util.escapeJavaScript("$context.stage\_TableName").replaceAll("\\","")",
"FilterExpression": "identityId = :v1 and aWithbWithc = :v2",
"ExpressionAttributeValues": {
":v1": {
"S": "$context.identity.cognitoIdentityId"
},
":v2" : {
"S" : "$util.escapeJavaScript($input.params('a'))__$util.escapeJavaScript($input.params('b'))__$util.escapeJavaScript($input.params('c'))"
}
}
}
you can have something like
{
#set($tableNameSuffix = "_TableName")
"TableName": "$stageVariables.environment$tableNameSuffix",
"ConsistentRead": true,
"KeyConditionExpression": "XXX = :val",
"ExpressionAttributeValues": {
":val": {
"S": "$method.request.path.xxx"
}
}
}

PEG.js help - optional free text followed by key value pairs

I wrote the following parser (paste into http://pegjs.org/online and it works):
Expression = Pairs / FullTextWithPairs
Pairs = (';'? _ p:Pair { return p; })*
FullTextWithPairs = fto:FullText po:Pairs
{
var arr = [];
if (fto) {
arr.push(fto);
}
return arr.concat(po);
}
FullText = ft:ValueString ';'
{
return {'key' : 'fullText', 'op': '=', 'value': ft};
}
Pair = k:Field op:Operator v:ValueString
{
var res = {'key' : k, 'op': op, 'value': v};
console.log(res);
return res;
}
Operator = ('<>' / [=><])
ValueString = vs:[^;]+ {return vs.join('');}
Field = 'location' / 'application' / 'count'
_ = ' '*
It parses this string of key-value pairs: location=USA; application<>app; count>5
to this:
[
{
"key": "location",
"op": "=",
"value": "USA"
},
{
"key": "application",
"op": "<>",
"value": "app"
},
{
"key": "count",
"op": ">",
"value": "5"
}
]
The problem is I want to enable a free-text search as well, which is entered before the key-value pairs, for example:
this: free text foobar; location=USA; application<>app; count>5
and get this:
[
{
"key": "fullText",
"op": "=",
"value": "free text foobar"
},
{
"key": "location",
"op": "=",
"value": "USA"
},
{
"key": "application",
"op": "<>",
"value": "app"
},
{
"key": "count",
"op": ">",
"value": "5"
}
]
The parser should recognize that the first part is not a key-value pair (according to "Pair" rule) and insert it as "fullText" object.
Basically "Expression" rule should do it, according to what I read in the docs - A / B means if A doesn't pass the B is tried. In the second case "Paris" is faild because "free text foobar" doesn't pass the Pairs rule, but it just throws an exception instead of moving on.
Congrats to whomever survived up to here, what am I doing wrong? :)
Thank you
Played with the grammar some more, the solution was to use !Pair and (for some reason) to change the order of the "Expression" rule:
Expression = FullTextWithPairs / Pairs
Pairs = (';'? _ p:Pair { return p; })*
FullTextWithPairs = fto:FullText po:Pairs
{
var arr = [];
if (fto) {
arr.push(fto);
}
return arr.concat(po);
}
FullText = !Pair ft:ValueString ';'
{
return {'key' : 'fullText', 'op': '=', 'value': ft};
}
Pair = _? k:Field op:Operator v:ValueString
{
var res = {'key' : k, 'op': op, 'value': v};
console.log(res);
return res;
}
Operator = ('<>' / [=><])
ValueString = vs:[^;]+ {return vs.join('');}
Field = 'location' / 'application' / 'count'
_ = ' '*