I am using stream processor 4.3.0
I have created one siddhi app which has source as mqtt and message type as json
and in sink as well I am using mqtt and message as json. Basically, there is no transformation of message required.
in source mqtt topic messages are in following way
{
"value1" : 59.698437,
"value2" : 14.977777,
"valid" : true
}
which ideally should be sent to the sink mqtt broker topic.
Now, to test this, I am using event simulator in /editor to test the sidhi app. After entering the dummy values, this generates feed as
{
"event" : {
"value1" : "59.698437",
"value2" : "14.977777",
"valid" : true
}
which is successfully transferred to the sink topic.
Now, in the actual message feed and generated by simulator has difference. It has event object in message which is why the editor understand that and make other messages (which does not has event object) as invalid.
Is there any way, that stream processor can process the feeds without events as well and how can this be checked that sinked messages are not having event?
You have to use custom mapping in JSON mapper type to parse the desired JSON input
#source(type='mqtt',
#map(type='json', enclosing.element="$", #attributes(value1 = "value1", value2 = "value2", isValid = "valid")))
define stream InputStream(value1 string, value2 string, isValid bool);
See examples under API docs for more info, https://siddhi-io.github.io/siddhi-map-json/api/4.1.1/#json-source-mapper
You can use log sink type to check the published event. Make sure to use the same map configs,
#sink(type='log',
#map(type='json', enclosing.element="$", #attributes(value1 = "value1", value2 = "value2", isValid = "valid")))
Related
Is there a way to filter messages based on Regex or substring in AWS SNS?
AWS Documentation for filtering messages mentions three types of filtering for strings:
Exact matching (whitelisting)
Anything-but matching (blacklisting)
Prefix matching
I want to filter out messages based on substrings in the messages, for example
I have a S3 event that sends a message to SNS when a new object is added to S3, the contents of the message are as below:
{
"Records": [
{
"s3": {
"bucket": {
"name": "images-bucket"
},
"object": {
"key": "some-key/more-key/filteringText/additionaldata.png"
}
}
}
]
}
I want to keep the messages if only filteringText is present in key field.
Note: The entire message is sent as text by S3 notification service, so Records is not a json object but string.
From what I've seen in the documentation, you can't do regex matches or substrings, but you can match prefixes and create your own attributes in the MessageAttributes field.
To do this, I send the S3 event to a simple Lambda that adds MessageAttributes and then sends to SNS.
In effect, S3 -> Lambda -> SNS -> other consumers (with filtering).
The Lambda can do something like this (where you'll have to programmatically decide when to add the attribute):
let messageAttributes = {
myfilterkey: {DataType: "String", StringValue:"filteringText"}
};
let params = {
Message: JSON.stringify(payload),
MessageAttributes: messageAttributes,
MessageStructure: 'json',
TargetArn: SNS_ARN
};
await sns.publish(params).promise();
Then in SNS you can filter:
{"myfilterkey": ["filtertext"]}
It seems a little convoluted to put the Lambda in there, but I like the idea of being able to plug and unplug consumers from SNS on the fly and use filtering to determine who gets what.
I Am trying to publish a Json Message to AWS SNS topic from my C# Application using AWS SDk. Its [enter image description here][1]populating message in string format and message attribute filed is not populated.
Code sample is as below:
var snsClient = new AmazonSimpleNotificationServiceClient(accessId, secretrkey, RegionEndpoint.USEast1);
PublishRequest publishReq = new PublishRequest()
{
TargetArn = topicARN,
MessageStructure = "json",
Message = JsonConvert.SerializeObject(message)
};
var msgAttributes = new Dictionary<string, MessageAttributeValue>();
var msgAttribute = new MessageAttributeValue();
msgAttribute.DataType = "String";
msgAttribute.StringValue = "123";
msgAttributes.Add("Objectcd", msgAttribute);
publishReq.MessageAttributes = msgAttributes;
PublishResponse response = snsClient.Publish(publishReq);
Older question but answering as I came across when dealing with similar issue
When you set the MessageStructure to "json".
The json must contain at least a top-level JSON key of "default" with a value that is a string.
So json needs to look like
{
"default" : "my message"
}
My solution looks something like
var messageDict = new Dictionary<string,object>()
messageDict["default"] = "my message";
PublishRequest publishReq = new PublishRequest()
{
TargetArn = topicARN,
MessageStructure = "json",
Message = JsonConvert.SerializeObject(messageDict)
};
// if json is an object
// then
messageDict["default"] = JsonConvert.SerializeObject(myMessageObject);
I'm am using PublishAsync on v3
From the documentation
https://docs.aws.amazon.com/sdkfornet/v3/apidocs/items/SNS/TPublishRequest.html
Message structure
Gets and sets the property MessageStructure.
Set MessageStructure to json if you want to send a different message for each protocol. For example, using one publish action, you can send a short message to your SMS subscribers and a longer message to your email subscribers. If you set MessageStructure to json, the value of the Message parameter must:
be a syntactically valid JSON object; and
contain at least a top-level JSON key of "default" with a value that is a string.
You can define other top-level keys that define the message you want to send to a specific transport protocol (e.g., "http").
Valid value: json
Great coincidence!
I was just busy writing a C# implementation to publish a message to SNS when I stumbled up on this post. Hopefully this helps you.
The messageBody argument we pass down to PublishMessageAsync is a string, it can be deserialized JSON for example.
public class SnsClient : ISnsClient
{
private readonly IAmazonSimpleNotificationService _snsClient;
private readonly SnsOptions _snsOptions; // You can inject any options you want here.
public SnsClient(IOptions<SnsOptions> snsOptions, // I'm using the IOptionsPattern as I have the TopicARN defined in the appsettings.json
IAmazonSimpleNotificationService snsClient)
{
_snsOptions = snsOptions.Value;
_snsClient = snsClient;
}
public async Task<PublishResponse> PublishMessageAsync(string messageBody)
{
return await _snsClient.PublishAsync(new PublishRequest
{
TopicArn = _snsOptions.TopicArn,
Message = messageBody
});
}
}
Also note the above setup uses Dependency Injection, so it would require you to set up an ISnsClient and you register an instance when bootstrapping the application, something as following:
services.TryAddSingleton<ISnsClient, SnsClient>();
I'm trying to set an AWS IOT rule to send data to DynamoDB without the help of a lambda.
My rule query statement is : SELECT *, topic() AS topic, timestamp() AS timestamp FROM '+/#'
My data is fine in AWS IOT as I'm successfully retrieving it with a lambda. However, even by following the developer guide to create the rule, in order to get the information passed on to Dynamo, by setting the 2 form fields with ${topic} and ${timestamp} as it should work, I get nothing in DynamoDB and I can find the following exception in Cloudwatch :
MESSAGE:Dynamo Insert record failed. The error received was NoSuchElementException. Message arrived on: myTopic/data, Action: dynamo, Table: myTable, HashKeyField: topic, HashKeyValue: , RangeKeyField: Some(timestamp), RangeKeyValue:
HashKeyValue and RangeKeyValue seem to be empty. Why ?
I also posted the question on the AWS forum : https://forums.aws.amazon.com/thread.jspa?threadID=267987
Suppose your devide sends this payload:
mess={"reported":
{"light": "blue","Temperature": int(temp_data)),
"timestamp": str(pd.to_datetime(time.time()))}}
args.message=mess
You should query as:
SELECT message.reported.* FROM '#'
Then, set up DynamoDB hash key value as ${MessageID()}
You will get:
MessageID || Data
1527010174562 { "light" : { "S" : "blue" }, "Temperature" : { "N" : "41" }, "timestamp" : { "S" : "1970-01-01 00:00:01.527010174" }}
Then you can easily extract values using Lambda and send to S3 via Data Pipeline or to Firehose to create a data stream.
I have created one AWS Lex bot and I am invoking one lambda function from that bot. When testing the lambda function I am getting proper response but at bot I am getting below error:
An error has occurred: Received invalid response from Lambda: Can not
construct instance of IntentResponse: no String-argument
constructor/factory method to deserialize from String value
('2017-06-22 10:23:55.0') at [Source: "2017-06-22 10:23:55.0"; line:
1, column: 1]
Not sure, what is wrong and where I am missing. Could anyone assist me please?
The solution to above problem is that we need to make sure response returned by lambda function, to be used at AWS lex chat bot should be in below format:
{
"sessionAttributes": {
"key1": "value1",
"key2": "value2"
...
},
"dialogAction": {
"type": "ElicitIntent, ElicitSlot, ConfirmIntent, Delegate, or Close",
Full structure based on the type field. See below for details.
}
}
By this, chat bot expectd DialogAction and corresponding elements in order to process the message i.e. IntentResponse.
Reference: http://docs.aws.amazon.com/lex/latest/dg/lambda-input-response-format.html
no String-argument constructor/factory method to deserialize from String value
You are getting this error because you must be passing string values in the response of lambda function. You have to pass a predefined json object blueprint in the response.
Because the communication between Lex and Lambda is not simple value passing like normal functions. Amazon Lex expects output from Lambda in a particular JSON format and data is sent to Lambda in a particular JSON. The examples are here: Lambda Function Input Event and Response Format.
And just copying and pasting the blueprint won't work because in some fields you have choose between some predefined values and in some fields you have to entry valid input.
For example in,
"dialogAction": {
"type": "Close",
"fulfillmentState": "Fulfilled or Failed",
"message": {
"contentType": "PlainText or SSML",
"content": "Thanks, your pizza has been ordered."
}
}
you have assign a value "Fulfilled" or "Failed" to field 'fulfillmentState'. And same goes for 'contentType'.
I see in the examples how to to pass a message string to amazon sns sdk's publish method. However, is there an exmaple of how to pass a custom object as the message? I tried setting "MessageStructure" to "json" but then I get InvalidParameter: Invalid parameter: Message Structure - No default entry in JSON message body error. Where should I be passing the object values into in the params?
Any examples?
var params = {
Message: JSON.stringify(item),
MessageStructure: 'json',
TopicArn: topic
//MessageAttributes: item
};
return sns.publishAsync(params);
There is no SDK-supported way to pass a custom object as a message-- messages are always strings. You can, of course, make the string a serialized version of your object.
MessageStructure: 'json' is for a different purpose-- when you want to pass different strings to different subscription types. In that case, you make the message a serialized json object with AWS-defined structure, where each element defines the message to send to a particular type of subscription (email, sqs, etc). Even in that case, the messages themselves are just strings.
MessageAttributes are parameters you add to the message to support specific subscription types. If you are using SNS to talk to Apple's IOS notification service, for example, you might have to supply additional message parameters or authentication keys-- MessageAttributes provide a mechanism to do this. This is described in this AWS documentation.
An example is shown here: https://docs.aws.amazon.com/sns/latest/api/API_Publish.html#API_Publish_Example_2
The JSON format for Message is as follows:
{
"default": "A message.",
"email": "A message for email.",
"email-json": "A message for email (JSON).",
"http": "A message for HTTP.",
"https": "A message for HTTPS.",
"sqs": "A message for Amazon SQS."
}
So, assuming what you wanted to pass is an object, the way it worked for me was:
const messageObjToSend = {
...
}
const params = {
Message: JSON.stringify({
default: JSON.stringify( messageObjToSend )
}),
MessageStructure: 'json',
TopicArn: 'arn:aws:sns...'
}
Jackson 2 has pretty good support to convert object to JSON String and vice versa.
To String
Cat c = new Cat();
ObjectMapper mapper = new ObjectMapper();
String s = mapper.writeValueAsString(c);
To Object
Cat obj = mapper.readValue(s,Cat.class);
The message needs to be a JSON object and the default property needs to be added and should contain the JSON you want included in the email.
var defaultMessage = { "default": item };
var params = {
Message: defaultMessage, /*JSON.stringify(item),*/
---------^
MessageStructure: 'json',
TopicArn: topic
//MessageAttributes: item
};
return sns.publishAsync(params);
Using python,
boto3.client("sns").publish(
TopicArn=sns_subscription_arn,
Subject="subject",
Message=json.dumps({"default": item}),
--------^
MessageStructure="json",
)
FYI, if you go to this SNS topic in the AWS Console you can "publish message" and choose "Custom payload for each delivery protocol.". Here you will see a template of the email and the "default" property is tagged for "Sample fallback message".