I'm trying to update a method in a file in AWS lambda function. Previously, the method accepted URLs such as
https://yh74mgrokc.execute-api.us-west-1.amazonaws.com/dev/ga/answer/qn_type?tag=alexa
I updated the function to accept URL like the ones below, as well as new tags like to and from.
https://yh74mgrokc.execute-api.us-west-1.amazonaws.com/dev/ga/answer/qn_type?tag=alexa&from=2020-05-29&to=2021-05-29
So, I modified one of the methods as follows:
exports.getAnswerQnTypeSchema = [
query('tag')
.exists()
.withMessage('tag is required'),
query().custom((query) => {
const allowedKeys = [
"id",
"name",
"tag",
"from",
"to"
];
for (const key of Object.keys(query)) {
if (!allowedKeys.includes(key)) {
throw new Error(`Unknown property: ${key} please resolve`,query);
}
}
return true;
})
];
to allow for 'from' and 'to' tags. Then I redeployed the function and uploaded the updated lambda function. But still the function works as the previous version. when I send the request
https://yh74mgrokc.execute-api.us-west-1.amazonaws.com/dev/ga/answer/qn_type?tag=alexa&from=2020-05-29&to=2021-05-29
It does not support the from and to tags, and the error message is as follows:
I'm not sure why this is the case. I'm new to AWS services, and any assistance with this matter would be greatly appreciated.
Related
Here is a modified version of an Event type I am receiving in my handler for a lambda function with a DynamoDB someTableName table trigger that I logged using cargo lambda.
Event {
records: [
EventRecord {
change: StreamRecord {
approximate_creation_date_time: ___
keys: {"id": String("___")},
new_image: {
....
"valid": Boolean(true),
},
...
},
...
event_name: "INSERT",
event_source: Some("aws:dynamodb"),
table_name: None
}
]
}
Goal: Correctly filter with event_name=INSERT && valid=false
I have tried a number of options, for example;
{"eventName": ["INSERT"]}
While the filter is added correctly, it does not trigger the lambda on item inserted.
Q1) What am I doing incorrectly here?
Q2) Why is table_name returning None? The lambda function is created with a specific table name as trigger. The returned fields are returning an option (Some(_)) so I'm asssuming it returns None if the table name is specified on lambda creation, but seems odd to me?
Q3) From AWS Management Console > Lambda > ... > Trigger Detail, I see the following (which is slightly different from my code mentioned above), where does "key" come from and what does it represent in the original Event?
Filters must follow the documented syntax for filtering in the Event Source Mapping between Lambda and DynamoDB Streams.
If you are entering the filter in the Lambda console:
{ "eventName": ["INSERT"], "dynamodb": { "NewImage": {"valid": { "BOOL" : [false]}} } }
The attribute name is actually eventName, so your filter should look like this:
{"eventName": ["INSERT"]}
I'm trying to access data from my stack where I'm creating an AppSync API. I want to be able to use the generated Stacks' url and apiKey but I'm running into issues with them being encoded/tokenized.
In my stack I'm setting some fields to the outputs of the deployed stack:
this.ApiEndpoint = graphAPI.url;
this.Authorization = graphAPI.graphqlApi.apiKey;
When trying to access these properties I get something like ${Token[TOKEN.209]} and not the values.
If I'm trying to resolve the token like so: this.resolve(graphAPI.graphqlApi.apiKey) I instead get { 'Fn::GetAtt': [ 'AppSyncAPIApiDefaultApiKey537321373E', 'ApiKey' ] }.
But I would like to retrieve the key itself as a string, like da2-10lksdkxn4slcrahnf4ka5zpeemq5i.
How would I go about actually extracting the string values for these properties?
The actual values of such Tokens are available only at deploy-time. Before then you can safely pass these token properties between constructs in your CDK code, but they are opaque placeholders until deployed. Depending on your use case, one of these options can help retrieve the deploy-time values:
If you define CloudFormation Outputs for a variable, CDK will (apart from creating it in CloudFormation), will, after cdk deploy, print its value to the console and optionally write it to a json file you pass with the --outputs-file flag.
// AppsyncStack.ts
new cdk.CfnOutput(this, 'ApiKey', {
value: this.api.apiKey ?? 'UNDEFINED',
exportName: 'api-key',
});
// at deploy-time, if you use a flag: --outputs-file cdk.outputs.json
{
"AppsyncStack": {
"ApiKey": "da2-ou5z5di6kjcophixxxxxxxxxx",
"GraphQlUrl": "https://xxxxxxxxxxxxxxxxx.appsync-api.us-east-1.amazonaws.com/graphql"
}
}
Alternatively, you can write a script to fetch the data post-deploy using the listGraphqlApis and listApiKeys commands from the appsync JS SDK client. You can run the script locally or, for advanced use cases, wrap the script in a CDK Custom Resource construct for deploy-time integration.
Thanks to #fedonev I was able to extract the API key and url like so:
const client = new AppSyncClient({ region: "eu-north-1" });
const command = new ListGraphqlApisCommand({ maxResults: 1 });
const res = await client.send(command);
if (res.graphqlApis) {
const apiKeysCommand = new ListApiKeysCommand({
apiId: res.graphqlApis[0].apiId,
});
const apiKeyResponse = await client.send(apiKeysCommand);
const urls = flatMap(res.graphqlApis[0].uris);
if (apiKeyResponse.apiKeys && res.graphqlApis[0].uris) {
sendSlackMessage(urls[1], apiKeyResponse.apiKeys[0].id || "");
}
}
I am working on a POC using Kendra and Salesforce. The connector allows me to connect to my Salesforce Org and index knowledge articles. I have been able to set this up and it is currently working as expected.
There are a few custom fields and data points I want to bring over to help enrich the data even more. One of these is an additional answer / body that will contain key information for the searching.
This field in my data source is rich text containing HTML and is often larger than 2048 characters, a limit that seems to be imposed in a String data field within Kendra.
I came across two hooks that are built in for Pre and Post data enrichment. My thought here is that I can use the pre hook to strip HTML tags and truncate the field before it gets stored in the index.
Hook Reference: https://docs.aws.amazon.com/kendra/latest/dg/API_CustomDocumentEnrichmentConfiguration.html
Current Setup:
I have added a new field to the index called sf_answer_preview. I then mapped this field in the data source to the rich text field in the Salesforce org.
If I run this as is, it will index about 200 of the 1,000 articles and give an error that the remaining articles exceed the 2048 character limit in that field, hence why I am trying to set up the enrichment.
I set up the above enrichment on my data source. I specified a lambda to use in the pre-extraction, as well as no additional filtering, so run this on every article. I am not 100% certain what the S3 bucket is for since I am using a data source, but it appears to be needed so I have added that as well.
For my lambda, I create the following:
exports.handler = async (event) => {
// Debug
console.log(JSON.stringify(event))
// Vars
const s3Bucket = event.s3Bucket;
const s3ObjectKey = event.s3ObjectKey;
const meta = event.metadata;
// Answer
const answer = meta.attributes.find(o => o.name === 'sf_answer_preview');
// Remove HTML Tags
const removeTags = (str) => {
if ((str===null) || (str===''))
return false;
else
str = str.toString();
return str.replace( /(<([^>]+)>)/ig, '');
}
// Truncate
const truncate = (input) => input.length > 2000 ? `${input.substring(0, 2000)}...` : input;
let result = truncate(removeTags(answer.value.stringValue));
// Response
const response = {
"version" : "v0",
"s3ObjectKey": s3ObjectKey,
"metadataUpdates": [
{"name":"sf_answer_preview", "value":{"stringValue":result}}
]
}
// Debug
console.log(response)
// Response
return response
};
Based on the contract for the lambda described here, it appears pretty straight forward. I access the event, find the field in the data called sf_answer_preview (the rich text field from Salesforce) and I strip and truncate the value to 2,000 characters.
For the response, I am telling it to update that field to the new formatted answer so that it complies with the field limits.
When I log the data in the lambda, the pre-extraction event details are as follows:
{
"s3Bucket": "kendrasfdev",
"s3ObjectKey": "pre-extraction/********/22736e62-c65e-4334-af60-8c925ef62034/https://*********.my.salesforce.com/ka1d0000000wkgVAAQ",
"metadata": {
"attributes": [
{
"name": "_document_title",
"value": {
"stringValue": "What majors are under the Exploratory track of Health and Life Sciences?"
}
},
{
"name": "sf_answer_preview",
"value": {
"stringValue": "A complete list of majors affiliated with the Exploratory Health and Life Sciences track is available online. This track allows you to explore a variety of majors related to the health and life science professions. For more information, please visit the Exploratory program description. "
}
},
{
"name": "_data_source_sync_job_execution_id",
"value": {
"stringValue": "0fbfb959-7206-4151-a2b7-fce761a46241"
}
},
]
}
}
The Problem:
When this runs, I am still getting the same field limit error that the content exceeds the character limit. When I run the lambda on the raw data, it strips and truncates it as expected. I am thinking that the response in the lambda for some reason isn't setting the field value to the new content correctly and still trying to use the data directly from Salesforce, thus throwing the error.
Has anyone set up lambdas for Kendra before that might know what I am doing wrong? This seems pretty common to be able to do things like strip PII information before it gets indexed, so I must be slightly off on my setup somewhere.
Any thoughts?
since you are still passing the rich text as a metadata filed of a document, the character limit still applies so the document would fail at validation step of the API call and would not reach the enrichment step. A work around is to somehow append those rich text fields to the body of the document so that your lambda can access it there. But if those fields are auto generated for your documents from your data sources, that might not be easy.
Is there a way to filter messages based on Regex or substring in AWS SNS?
AWS Documentation for filtering messages mentions three types of filtering for strings:
Exact matching (whitelisting)
Anything-but matching (blacklisting)
Prefix matching
I want to filter out messages based on substrings in the messages, for example
I have a S3 event that sends a message to SNS when a new object is added to S3, the contents of the message are as below:
{
"Records": [
{
"s3": {
"bucket": {
"name": "images-bucket"
},
"object": {
"key": "some-key/more-key/filteringText/additionaldata.png"
}
}
}
]
}
I want to keep the messages if only filteringText is present in key field.
Note: The entire message is sent as text by S3 notification service, so Records is not a json object but string.
From what I've seen in the documentation, you can't do regex matches or substrings, but you can match prefixes and create your own attributes in the MessageAttributes field.
To do this, I send the S3 event to a simple Lambda that adds MessageAttributes and then sends to SNS.
In effect, S3 -> Lambda -> SNS -> other consumers (with filtering).
The Lambda can do something like this (where you'll have to programmatically decide when to add the attribute):
let messageAttributes = {
myfilterkey: {DataType: "String", StringValue:"filteringText"}
};
let params = {
Message: JSON.stringify(payload),
MessageAttributes: messageAttributes,
MessageStructure: 'json',
TargetArn: SNS_ARN
};
await sns.publish(params).promise();
Then in SNS you can filter:
{"myfilterkey": ["filtertext"]}
It seems a little convoluted to put the Lambda in there, but I like the idea of being able to plug and unplug consumers from SNS on the fly and use filtering to determine who gets what.
I have am AWS Lambda function written in Java that I would like to use as part of a response to an AWS CloudFormation function. Amazon provides two detailed examples on how to create a CloudFormation custom resource that returns its value based on an AWS Lambda function written in Node.js, however I have been having difficulty translating the Lambda examples into Java. How can we setup our AWS Java function so that it reads the value of the pre-signed S3 URL passed in as a parameter to the Lambda function from CloudFormation and send back our desired response to the waiting CloudFormation template?
After back and forth conversation with AWS, here are some code samples I've created that accomplish this.
First of all, assuming you want to leverage the predefined interfaces for creating Handlers, you can implement RequestsHandler and define the HandleRequest methods like so:
public class MyCloudFormationResponder implements RequestHandler<Map<String, Object>, Object>{
public Object handleRequest(Map<String,Object> input, Context context) {
...
}
}
The Map<String, Object>is a Map of the values sent from your CloudFormation resource to the Lambda function. An example CF resource:
"MyCustomResource": {
"Type" : "Custom::String",
"Version" : "1.0",
"Properties": {
"ServiceToken": "arn:aws:lambda:us-east-1:xxxxxxx:function:MyCloudFormationResponderLambdaFunction",
"param1": "my value1",
"param2": ["t1.micro", "m1.small", "m1.large"]
}
}
can be analyzed with the following code
String responseURL = (String)input.get("ResponseURL");
context.getLogger().log("ResponseURLInput: " + responseURL);
context.getLogger().log("StackId Input: " + input.get("StackId"));
context.getLogger().log("RequestId Input: " + input.get("RequestId"));
context.getLogger().log("LogicalResourceId Context: " + input.get("LogicalResourceId"));
context.getLogger().log("Physical Context: " + context.getLogStreamName());
#SuppressWarnings("unchecked")
Map<String,Object> resourceProps = (Map<String,Object>)input.get("ResourceProperties");
context.getLogger().log("param 1: " + resourceProps.get("param1"));
#SuppressWarnings("unchecked")
List<String> myList = (ArrayList<String>)resourceProps.get("param2");
for(String s : myList){
context.getLogger().log(s);
}
The key things to point out here, beyond what is explained in the NodeJS examples in the AWS documentation are
(String)input.get("ResponseURL") is the pre-signed S3 URL that you need to respond back to (more on this later)
(Map<String,Object>)input.get("ResourceProperties") returns the map of your CloudFormation custom resource "Properties" passed into the Lambda function from your CF template. I provided a String and ArrayList as two examples of object types that can be returned, though several others are possible
In order to respond back to the CloudFormation template custom resource instantiation, you need to execute an HTTP PUT call back to the ResponseURL previously mentioned and include most of the following fields in the variable cloudFormationJsonResponse. Below is how I've done this
try {
URL url = new URL(responseURL);
HttpURLConnection connection=(HttpURLConnection)url.openConnection();
connection.setDoOutput(true);
connection.setRequestMethod("PUT");
OutputStreamWriter out = new OutputStreamWriter(connection.getOutputStream());
JSONObject cloudFormationJsonResponse = new JSONObject();
try {
cloudFormationJsonResponse.put("Status", "SUCCESS");
cloudFormationJsonResponse.put("PhysicalResourceId", context.getLogStreamName());
cloudFormationJsonResponse.put("StackId", input.get("StackId"));
cloudFormationJsonResponse.put("RequestId", input.get("RequestId"));
cloudFormationJsonResponse.put("LogicalResourceId", input.get("LogicalResourceId"));
cloudFormationJsonResponse.put("Data", new JSONObject().put("CFAttributeRefName", "some String value useful in your CloudFormation template"));
} catch (JSONException e) {
e.printStackTrace();
}
out.write(cloudFormationJsonResponse.toString());
out.close();
int responseCode = connection.getResponseCode();
context.getLogger().log("Response Code: " + responseCode);
} catch (IOException e) {
e.printStackTrace();
}
Of particular note is the node "Data" above which references an additional com.amazonaws.util.json.JSONObject in which I include any attributes that are required in my CloudFormation template. In this case, it would be retrieved in CF template with something like { "Fn::GetAtt": [ "MyCustomResource", "CFAttributeRefName" ] }
Finally, you can simply return null since nothing would be returned from this function as it's the HTTPUrlConnection that actually responds to the CF call.
Neil,
I really appreciate your great documentation here. I would add a few things that I found useful:
input.get("RequestType") - This comes back as "Create", "Delete", etc. You can use this value to determine what to do when a stack is created, deleted, etc..
As far as security, I uploaded the Lambda Functions and set the VPC, subnets, and security group (default) manually so I can reuse it with several cloudformationn scripts. That seems to be working okay.
I created one Lambda function that gets called by the CF scripts and one I can run manually in case the first one fails.
This excellent gradle aws plugin makes it easy to upload Java Lambda functions to AWS.
Gradle AWS Plugin