how to handle nested lists in AWS APIG Mapping Template in VTL - amazon-web-services

(Here's my Model scheme:
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "QuestionsModel",
"type": "array",
"items": {
"type": "object",
"properties": {
"section_name": { "type": "string" },
"options" : {
"type" : "array",
"items" : {
"type" : "array",
"items" : {
"type" : "string"
}
}
}
}
Here's the Mapping template:
#set($inputRoot = $input.path('$'))
[
#foreach($question in $inputRoot) {
"section_name" : "$question.section_name.S",
"options" : [
#foreach($items in $question.options.L) {
[
#foreach($item in $items.L) {
"$item.S"
}#if($foreach.hasNext),#end
#end
]
}#if($foreach.hasNext),#end
#end
]
}#if($foreach.hasNext),#end
#end
]
Although this syntax correctly maps the data it results in "options" being an empty array.
Without the "options" specified then my iOS app receives valid JSON. But when I try various syntaxes for "options" then I either get invalid JSON or an "Internal Service Error" and CloudWatch isn't much better offering Unable to transform response.
The options valid is populated with this content: {L=[{"L":[{"S":"1"},{"S":"Dr"}]},{"L":[{"S":"2"},{"S":"Mr"}]},{"L":[{"S":"3"},{"S":"Ms"}]},{"L":[{"S":"4"},{"S":"Mrs"}]},{"L":[{"S":"5"},{"S":"Prof."}]}]} which is provided by a Lambda function.
I can only conclude, at this point, that API Gateway VTL doesn't support nested arrays.

AWS iOS SDK for Modelling doesn't support array of arrays.
You have to define a dictionary in between any nested arrays.
So instead of array/object/array/array you slip in an extra "awshack" object: array/object/array/awshack-object/array
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "QuestionsModel",
"type": "array",
"items": {
"type": "object",
"properties": {
"section_name": { "type": "string" },
"options" : { "type" : "array",
"items" : {
"type" : "object",
"properties" : {
"awshack" : {
"type" : "array",
"items" : { "type" : "string" }
}
}
}
}
}
}
}
In the mapping template the "awshack" is slipped in outside the innermost loop.
#foreach($items in $question.options.L)
{"awshack" :
[#foreach($item in $items.L)
"$item.S"#if($foreach.hasNext),#end
#end
#if($foreach.hasNext),#end
]}#if($foreach.hasNext),#end
#end
Amazon confirms this limitation.

Related

ElasticSearch reindexing with selected fields result into addition of non selected empty field

Scenario:
We are using AWS ElasticSearch 6.8. We got an index (index-A) with a mapping structure consist of multiple nested objects and JSON hierarchy. We need to create new index (index-B) and move all documents from index-A to index-B.
We need to create index-B with only specific fields.
We need to rename field names while reindexing
e.g.
index-A mapping:
{
"userdata": {
"properties": {
"payload": {
"type": "object",
"properties": {
"Alldata": {
"Username": {
"type": "keyword"
},
"Designation": {
"type": "keyword"
},
"Company": {
"type": "keyword"
},
"Region": {
"type": "keyword"
}
}
}
}
}
}}
Expected structure of index-B mapping after reindexing with rename (Company-cnm, Region-rg) :-
{
"userdata": {
"properties": {
"cnm": {
"type": "keyword"
},
"rg": {
"type": "keyword"
}
}
}}
Steps we are Following:
First we are using Create index API to create index-B with above mapping structure
Once index is created we are creating an ingest pipeline.
PUT ElasticSearch domain endpoint/_ingest/pipeline/my_rename_pipeline
{
"description": "rename field pipeline",
"processors": [{
"rename": {
"field": "payload.Company",
"target_field": "cnm",
"ignore_missing": true
}
},
{
"rename": {
"field": "payload.Region",
"target_field": "rg",
"ignore_missing": true
}
}
]
}
Perform reindexing operation, payload for the same below
let reindexParams = {
wait_for_completion: false,
slices: "auto",
body: {
"conflicts": "proceed",
"source": {
"size": 8000,
"index": "index-A",
"_source": ["payload.Company", "payload.Region"]
},
"dest": {
"index": "index-B",
"pipeline": "my_rename_pipeline",
"version_type": "external"
}
}
};
Problem:
Once the reindexing is complete as expected all documents transferred to new index with renamed fields but there is one additional field which is not selected. As you can see below the "payload" object with metadata is also added to the new index after reindexing. This field is empty and consist of no data.
index-B looks like below after reindexing:
{
"userdata": {
"properties": {
"cnm": {
"type": "keyword"
},
"rg": {
"type": "keyword"
},
"payload": {
"properties": {
"Alldata": {
"type": "object"
}
}
}
}
}}
We are unable to find the workaround and need help how to stop this field from creating. Any help will be appreciated.
Great job!! You're almost there, you simply need to remove the payload field within your pipeline using the remove processor and you're good:
{
"description": "rename field pipeline",
"processors": [
{
"rename": {
"field": "payload.Company",
"target_field": "cnm",
"ignore_missing": true
}
},
{
"rename": {
"field": "payload.Region",
"target_field": "rg",
"ignore_missing": true
}
},
{
"remove": { <--- add this processor
"field": "payload"
}
}
]
}

Outputs are not getting generated based on condition

In my cf template I have set of conditions defined and those conditions are invoked in the resource section as well however when i try to generate outputs using the conditions its not working as expected.
{
"AWSTemplateFormatVersion": "2010-09-09",
"Description": "CloudFormation to Deploy EMR clusters",
"Parameters": {
"Applications": {
"Default": "Core Hadoop",
"Description": "Installed Applications",
"Type": "String",
"AllowedValues" : ["Core Hadoop","HBase"],
"ConstraintDescription": "Must be valid Applications"
},
"awsRegion": {
"Default" : "<my-region>",
"Description": "awsRegion",
"Type": "String"
},
"Ec2KeyName": {
"Default": "<my-key.pem>",
"Description": "Ec2KeyName",
"Type": "String"
}
},
"Resources": {
"EMRCluster1": {
"Type" : "AWS::EMR::Cluster",
"Condition" : "CH",
"Properties" : {
"Applications" : [
{
"Name": "Hadoop"
},
{
"Name": "Hive"
},
{
"Name": "Hue"
},
{
"Name": "Pig"
},
{
"Name": "Mahout"
},
{
"Name": "Tez"
}
]
}
}
"EMRCluster2": {
"Type" : "AWS::EMR::Cluster",
"Condition" : "HB",
"Properties" : {
"Applications": [
{
"Name": "Hadoop"
},
{
"Name": "Hive"
},
{
"Name": "Hue"
},
{
"Name": "HBase"
},
{
"Name": "Phoenix"
},
{
"Name": "ZooKeeper"
}
]
}
}
}
"Conditions" : {
"CH" : {"Fn::Equals" : [{"Ref" : "Applications"}, "Core Hadoop"]},
"HB" : {"Fn::Equals" : [{"Ref" : "Applications"}, "HBase"]}
},
"Outputs": {
"MasterPublicDnsName": {
"Condition" : "CH",
"Value": {
"Fn::GetAtt": [
"EMRCluster1",
"MasterPublicDNS"
]
},
"Description": "MasterPublicDNS for cluster"
},
"MasterPublicDnsName": {
"Condition" : "HB",
"Value": {
"Fn::GetAtt": [
"EMRCluster2",
"MasterPublicDNS"
]
},
"Description": "MasterPublicDNS for cluster"
}
}
Expected: If i choose "CH" in the parameters then it should give cluster1 masterdns and if i choose "HB" it should provide cluster2's masterdns
Actual: If i choose "HB" which is the last section in output, it will give me the masterdns of cluster2 however if i choose "CH" outputs section in cloudformation gives me "No outputs found.".
Can somebody help me on this please.
The problem probably exists because you are using the same name for both outputs. Since the behaviour of JSON with duplicate keys is undefined, the implementing language can choose how to behave in this situation. Presumably, the second time you are using the MasterPublicDnsName as the output name, you are overwriting the first name, which is consistent with the behaviour you are seeing.
You can either opt to use two different names, but this might makes using cross-stack references difficult, or use an Fn::If statement in the value of the output:
"Fn::If": [condition_name, value_if_true, value_if_false]
Or in your case:
"Fn::If": ["CH", {"Fn::GetAtt": ["EMRCluster1", "MasterPublicDNS"]}, {"Fn::GetAtt": ["EMRCluster2", "MasterPublicDNS"]}]
If you need more than two options, you'll need to nest your Fn::If statements

How to correctly apply MethodResponse to "filter" AWS api gateway response

When trying to apply MethodResponse template I am failing to see any difference in final response. My goal is to successfully apply schema with minItems and maxItems for array property.
Example response from lambda method:
{
"_id": "5d5110f52e8b560af82dec69",
"index": 0,
"friends": [
{
"id": 0,
"name": "Mcconnell Pugh"
},
{
"id": 1,
"name": "Peggy Caldwell"
},
{
"id": 2,
"name": "Jocelyn Mccarthy"
}
]
}
Schema I have tried to apply in MethodResponse:
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title" : "Empty Schema",
"type" : "object",
"properties" : {
"friends" : {
"type" : "array",
"minItems" : 1,
"maxItems" : 2,
"items" : {
"type" : "object",
"properties" : {
"name" : {
"type" : "string"
},
"id": {
"type" : "integer"
}
}
}
},
"index" : {
"type" : "string"
}
}
}
I would expect to see only two "friends" in final response, not all of them.
After long research and a lot of AWS documentations I have found that:
It support JSON Schema 4, however not all features are supported -> related docs
Method Response basically does not apply validation. In my understanding, it just bring use if you want to export your API to Swagger to have well described specs -> related docs last Paragraph is important
So final answer to my question would be - you just cannot use Method Response as a filter, that's not the purpose of it.

Integration Response cannot map array of items

My goal is to to be able to produce a csv file. For now just trying to return a list of ids in JSON.
I'm struggling to map a response, so my JSON is:
{
"items": [
{
"id": "2017-07-02_n_Eleana Qorgyle",
"groupId": "n_2017-07"}
]
}
and my model is
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Log",
"type": "object",
"properties": {
"items": {
"type": "array",
"items": {
"type": "object",
"properties": {
"id": { "type": "string" }
}
}
}
}
}
My Velocity template is ...
#set($inputRoot = $input.path('$'))
{
"items" : [
##TODO: Update this foreach loop to reference array from input json
#foreach($elem in $inputRoot.items)
{
"id" : "foo"
}
#if($foreach.hasNext),#end
#end
]
}
However, I am getting this output:
{
"items": []
}
Have I missed something? $inputRoot.body outputs everything, but iterating is an issue.
I think this is what you are trying to do:
#set($inputRoot = $input.path('$')) {
"items" : [
#foreach($elem in $inputRoot.items){
"id" : "$elem.get('id')"
}
#if($foreach.hasNext),#end
#end
]
}
Thanks!

How do i define ElasticSearch Dynamic Templates?

I'm trying to define dynamic templates in Elastic Search to automatically set analysers for currently undefined properties for translations.
E.g. The following does exactly what i want, which is to set lang.en.title to use the english analyzer:
PUT /cl
{
"mappings" : {
"titles" : {
"properties" : {
"id" : {
"type" : "integer",
"index" : "not_analyzed"
},
"lang" : {
"type" : "object",
"properties" : {
"en" : {
"type" : "object",
"properties" : {
"title" : {
"type" : "string",
"index" : "analyzed",
"analyzer" : "english"
}
}
}
}
}
}
}
}
}
Which stems lang.en.title as expected e.g.
GET /cl/_analyze?field=lang.en.title&text=knocked
{
"tokens": [
{
"token": "knock",
"start_offset": 0,
"end_offset": 7,
"type": "<ALPHANUM>",
"position": 1
}
]
}
But what i'm trying to do is set all future string properties of lang.en to use the english analyser using a dynamic template which i can't seem to get working...
PUT /cl
{
"mappings" : {
"titles" : {
"dynamic_templates" : [{
"lang_en" : {
"path_match" : "lang.en.*",
"mapping" : {
"type" : "string",
"index" : "analyzed",
"analyzer" : "english"
}
}
}],`enter code here`
"properties" : {
"id" : {
"type" : "integer",
"index" : "not_analyzed"
},
"lang" : {
"type" : "object"
}
}
}
}
}
The english analyser isn't being applied as lang.en.title isn't stemmed as desired -
GET /cl/_analyze?field=lang.en.title&text=knocked
{
"tokens": [
{
"token": "knocked",
"start_offset": 0,
"end_offset": 7,
"type": "<ALPHANUM>",
"position": 1
}
]
}
What am i missing? :)
Your dynamic template is defined correctly. The issue is that you will need to index a document with the lang.en.title field in it before the dynamic template will apply the appropriate mapping. I ran the same dynamic mapping that you have defined above in your question locally and got the same result as you.
However, then I added a single document to the index.
POST /cl/titles/1
{
"lang.en.title": "Knocked out"
}
After adding the document, I ran the analyzer again and I got the expected output:
GET /cl/_analyze?field=lang.en.title&text=knocked
{
"tokens": [
{
"token": "knock",
"start_offset": 0,
"end_offset": 7,
"type": "<ALPHANUM>",
"position": 1
}
]
}
The index needs to have a document inserted so that it can execute the defined mapping template for the inserted fields. Once that field exists in the index and has the dynamic mapping applied, _analyze API calls will execute as expected.