I would like to use GCP PubSub with proto schema validation where messages will be in JSON. A part of the message is also TimeStamp so I added TimeStamp definition into the message (because imports are not supported) and my schema definition looks like this:
syntax = "proto3";
message NewSourceEvent {
message Source {
Timestamp time = 1;
string username = 2;
}
message Timestamp {
int64 seconds = 1;
int32 nanos = 2;
}
Source source = 1;
}
Test message:
{
"source": {
"time": "1999-05-11T05:17:10Z",
"username": "un"
}
}
And when I test the message from my service (which is serialized protobuf object into JSON) it's failed because it cannot serialize the TimeStamp - Invalid schema message: (source.time): invalid value "1999-05-11T05:17:10Z" for type type.googleapis.com/NewSourceEvent.Timestamp.
Is there any way how to define the schema to be able to parse the message and time will be still TimeStamp?
the timestamp type is imported from a external definition, like import "google/protobuf/timestamp.proto";
BUT, Pub/Sub does not support external import for now, as you check here , so I'm afraid it's not going to work. The workaround would be change it to string type.
I could not find a way to enable imports or to use timestamp without it.
The definition you have created is actually validating a message like this:
{
"source":{
"time":{
"seconds":"999",
"nanos":"999"
},
"username":"un"
}
You could try using a unix timestamp format in a int64 variable: 1643886443340 = Thursday, February 3, 2022 11:07:23.340 AM
Related
I am working on a lambda function that gets called from API Gateway and updates information in dynamoDB. I have half of this working really dynamically, and im a little stuck on updating. Here is what im working with:
dynamoDB table with a partition key of guild_id
My dummy json code im using:
{
"guild_id": "126",
"guild_name": "Posted Guild",
"guild_premium": "true",
"guild_prefix": "z!"
}
Finally the lambda code:
import json
import boto3
def lambda_handler(event, context):
client = boto3.resource("dynamodb")
table = client.Table("guildtable")
itemData = json.loads(event['body'])
guild = table.get_item(Key={'guild_id':itemData['guild_id']})
#If Guild Exists, update
if 'Item' in guild:
table.update_item(Key=itemData)
responseObject = {}
responseObject['statusCode'] = 200
responseObject['headers'] = {}
responseObject['headers']['Content-Type'] = 'application/json'
responseObject['body'] = json.dumps('Updated Guild!')
return responseObject
#New Guild, Insert Guild
table.put_item(Item=itemData)
responseObject = {}
responseObject['statusCode'] = 200
responseObject['headers'] = {}
responseObject['headers']['Content-Type'] = 'application/json'
responseObject['body'] = json.dumps('Inserted Guild!')
return responseObject
The insert part is working wonderfully, How would I accomplish a similar approach with update item? Im wanting this to be as dynamic as possible so I can throw any json code (within reason) at it and it stores it in the database. I am wanting my update method to take into account adding fields down the road and handling those
I get the follow error:
Lambda execution failed with status 200 due to customer function error: An error occurred (ValidationException) when calling the UpdateItem operation: The provided key element does not match the schema.
A "The provided key element does not match the schema" error means something is wrong with Key (= primary key). Your schema's primary key is guild_id: string. Non-key attributes belong in the AttributeUpdate parameter. See the docs.
Your itemdata appears to include non-key attributes. Also ensure guild_id is a string "123" and not a number type 123.
goodKey={"guild_id": "123"}
table.update_item(Key=goodKey, UpdateExpression="SET ...")
The docs have a full update_item example.
I am using stream processor 4.3.0
I have created one siddhi app which has source as mqtt and message type as json
and in sink as well I am using mqtt and message as json. Basically, there is no transformation of message required.
in source mqtt topic messages are in following way
{
"value1" : 59.698437,
"value2" : 14.977777,
"valid" : true
}
which ideally should be sent to the sink mqtt broker topic.
Now, to test this, I am using event simulator in /editor to test the sidhi app. After entering the dummy values, this generates feed as
{
"event" : {
"value1" : "59.698437",
"value2" : "14.977777",
"valid" : true
}
which is successfully transferred to the sink topic.
Now, in the actual message feed and generated by simulator has difference. It has event object in message which is why the editor understand that and make other messages (which does not has event object) as invalid.
Is there any way, that stream processor can process the feeds without events as well and how can this be checked that sinked messages are not having event?
You have to use custom mapping in JSON mapper type to parse the desired JSON input
#source(type='mqtt',
#map(type='json', enclosing.element="$", #attributes(value1 = "value1", value2 = "value2", isValid = "valid")))
define stream InputStream(value1 string, value2 string, isValid bool);
See examples under API docs for more info, https://siddhi-io.github.io/siddhi-map-json/api/4.1.1/#json-source-mapper
You can use log sink type to check the published event. Make sure to use the same map configs,
#sink(type='log',
#map(type='json', enclosing.element="$", #attributes(value1 = "value1", value2 = "value2", isValid = "valid")))
I Am trying to publish a Json Message to AWS SNS topic from my C# Application using AWS SDk. Its [enter image description here][1]populating message in string format and message attribute filed is not populated.
Code sample is as below:
var snsClient = new AmazonSimpleNotificationServiceClient(accessId, secretrkey, RegionEndpoint.USEast1);
PublishRequest publishReq = new PublishRequest()
{
TargetArn = topicARN,
MessageStructure = "json",
Message = JsonConvert.SerializeObject(message)
};
var msgAttributes = new Dictionary<string, MessageAttributeValue>();
var msgAttribute = new MessageAttributeValue();
msgAttribute.DataType = "String";
msgAttribute.StringValue = "123";
msgAttributes.Add("Objectcd", msgAttribute);
publishReq.MessageAttributes = msgAttributes;
PublishResponse response = snsClient.Publish(publishReq);
Older question but answering as I came across when dealing with similar issue
When you set the MessageStructure to "json".
The json must contain at least a top-level JSON key of "default" with a value that is a string.
So json needs to look like
{
"default" : "my message"
}
My solution looks something like
var messageDict = new Dictionary<string,object>()
messageDict["default"] = "my message";
PublishRequest publishReq = new PublishRequest()
{
TargetArn = topicARN,
MessageStructure = "json",
Message = JsonConvert.SerializeObject(messageDict)
};
// if json is an object
// then
messageDict["default"] = JsonConvert.SerializeObject(myMessageObject);
I'm am using PublishAsync on v3
From the documentation
https://docs.aws.amazon.com/sdkfornet/v3/apidocs/items/SNS/TPublishRequest.html
Message structure
Gets and sets the property MessageStructure.
Set MessageStructure to json if you want to send a different message for each protocol. For example, using one publish action, you can send a short message to your SMS subscribers and a longer message to your email subscribers. If you set MessageStructure to json, the value of the Message parameter must:
be a syntactically valid JSON object; and
contain at least a top-level JSON key of "default" with a value that is a string.
You can define other top-level keys that define the message you want to send to a specific transport protocol (e.g., "http").
Valid value: json
Great coincidence!
I was just busy writing a C# implementation to publish a message to SNS when I stumbled up on this post. Hopefully this helps you.
The messageBody argument we pass down to PublishMessageAsync is a string, it can be deserialized JSON for example.
public class SnsClient : ISnsClient
{
private readonly IAmazonSimpleNotificationService _snsClient;
private readonly SnsOptions _snsOptions; // You can inject any options you want here.
public SnsClient(IOptions<SnsOptions> snsOptions, // I'm using the IOptionsPattern as I have the TopicARN defined in the appsettings.json
IAmazonSimpleNotificationService snsClient)
{
_snsOptions = snsOptions.Value;
_snsClient = snsClient;
}
public async Task<PublishResponse> PublishMessageAsync(string messageBody)
{
return await _snsClient.PublishAsync(new PublishRequest
{
TopicArn = _snsOptions.TopicArn,
Message = messageBody
});
}
}
Also note the above setup uses Dependency Injection, so it would require you to set up an ISnsClient and you register an instance when bootstrapping the application, something as following:
services.TryAddSingleton<ISnsClient, SnsClient>();
I'm getting an exception which feels like it might be a defect. I'm wondering if anyone else sees a problem with this. Here is simplified but runnable service I am writing.
import ballerina.net.http;
import ballerina.lang.messages;
import ballerina.lang.jsons;
import ballerina.lang.system;
#http:BasePath("/weather")
service WeatherService {
#http:GET
#http:Path("/current")
resource current(message m) {
string url = "http://api.mesowest.net/v2/stations/nearesttime?stid=KBFI&within=60&vars=air_temp,wind_speed,wind_direction&obtimezone=local&token=demotoken";
http:ClientConnector weatherConnector = create http:ClientConnector(url);
message request = {};
message jsonResponse = http:ClientConnector.get(weatherConnector, "", request);
json jsonDocument = messages:getJsonPayload(jsonResponse);
json timestamp;
string timeString;
try {
timestamp = jsons:getJson(jsonDocument, "$..PERIOD_OF_RECORD.end");
}
catch (exception e) {
system:println("Error getting timestamp");
}
messages:setJsonPayload(m, timestamp);
reply m;
}
}
When I run this in the debugger, the json variable 'timestamp' is assigned the appropriate value from the JSON excerpt below:
"STATION": [
{
"STATUS": "ACTIVE",
"MNET_ID": "1",
"PERIOD_OF_RECORD": {
"start": "1969-12-31T16:00:00-0800",
"end": "2017-02-27T19:40:00-0800"
}
When I replace the line:
timestamp = jsons:getJson(jsonDocument, "$..PERIOD_OF_RECORD.end");
with the line
timeString = jsons:getString(jsonDocument, "$..PERIOD_OF_RECORD.end");
and stop and restart the service and test it, it throws an exception on the getString method. I haven't found a way to print an exception yet or get attributes of the exception in order to find why its failing. The console out is as follows.
Running weather2.bal service.
ballerina: deploying service(s) in '/Users/clarkm2/Projects/Ballerina/Weather/weather2.bal'
ballerina: started server connector http-9090
Error getting timestamp
An idea on this? If this is a defect, do these get reported on the wso2.com JIRA site?
Thanks.
The reason for this error is that, the element returned by $..PERIOD_OF_RECORD.end is not a string. It returns the following
"PERIOD_OF_RECORD": {
"end": "2017-02-27T21:00:00-0800",
"start": "1969-12-31T16:00:00-0800"
},
which cannot be converted to text. If you log the exception in the catch block you will be able to observe the following error.
Error getting timestamp : Failed to get string from json. Error while executing jsonpath: The element matching path: $..PERIOD_OF_RECORD.end is not a String.
To log the exception modify your code as :
catch (exception e) {
system:println("Error getting timestamp : " + exceptions:getMessage(e));
}
with import ballerina.lang.exceptions import statement.
Try changing the jsonpath as follow:
string timeString = jsons:getString(j, "$.STATION[0].PERIOD_OF_RECORD.end");
I see in the examples how to to pass a message string to amazon sns sdk's publish method. However, is there an exmaple of how to pass a custom object as the message? I tried setting "MessageStructure" to "json" but then I get InvalidParameter: Invalid parameter: Message Structure - No default entry in JSON message body error. Where should I be passing the object values into in the params?
Any examples?
var params = {
Message: JSON.stringify(item),
MessageStructure: 'json',
TopicArn: topic
//MessageAttributes: item
};
return sns.publishAsync(params);
There is no SDK-supported way to pass a custom object as a message-- messages are always strings. You can, of course, make the string a serialized version of your object.
MessageStructure: 'json' is for a different purpose-- when you want to pass different strings to different subscription types. In that case, you make the message a serialized json object with AWS-defined structure, where each element defines the message to send to a particular type of subscription (email, sqs, etc). Even in that case, the messages themselves are just strings.
MessageAttributes are parameters you add to the message to support specific subscription types. If you are using SNS to talk to Apple's IOS notification service, for example, you might have to supply additional message parameters or authentication keys-- MessageAttributes provide a mechanism to do this. This is described in this AWS documentation.
An example is shown here: https://docs.aws.amazon.com/sns/latest/api/API_Publish.html#API_Publish_Example_2
The JSON format for Message is as follows:
{
"default": "A message.",
"email": "A message for email.",
"email-json": "A message for email (JSON).",
"http": "A message for HTTP.",
"https": "A message for HTTPS.",
"sqs": "A message for Amazon SQS."
}
So, assuming what you wanted to pass is an object, the way it worked for me was:
const messageObjToSend = {
...
}
const params = {
Message: JSON.stringify({
default: JSON.stringify( messageObjToSend )
}),
MessageStructure: 'json',
TopicArn: 'arn:aws:sns...'
}
Jackson 2 has pretty good support to convert object to JSON String and vice versa.
To String
Cat c = new Cat();
ObjectMapper mapper = new ObjectMapper();
String s = mapper.writeValueAsString(c);
To Object
Cat obj = mapper.readValue(s,Cat.class);
The message needs to be a JSON object and the default property needs to be added and should contain the JSON you want included in the email.
var defaultMessage = { "default": item };
var params = {
Message: defaultMessage, /*JSON.stringify(item),*/
---------^
MessageStructure: 'json',
TopicArn: topic
//MessageAttributes: item
};
return sns.publishAsync(params);
Using python,
boto3.client("sns").publish(
TopicArn=sns_subscription_arn,
Subject="subject",
Message=json.dumps({"default": item}),
--------^
MessageStructure="json",
)
FYI, if you go to this SNS topic in the AWS Console you can "publish message" and choose "Custom payload for each delivery protocol.". Here you will see a template of the email and the "default" property is tagged for "Sample fallback message".