In AWS IOT, I'm able to publish an MQTT message from a device to the topic "topic/messages" :
{
"id":"messageID",
"value":"messageValue"
}
And I want to "augment" it in the server by adding a timestamp on it and let the message to continue on the SAME TOPIC.
So I'd like that subscribers on "topic/messages" receive this:
{
"id":"messageID",
"value":"messageValue",
"serverTimestamp":"1637867431920" <--- Here
}
However, I don't find the way how to process this message and let it flow in the same topic:
I can add a rule
SELECT * , timestamp() as serverTimestamp FROM 'topic/#'
But the rule does not augment the original message but creates an augmented copy and redirects it to some other service (Lambda, DynamoDB, Republish, etc..)
Those services work with the copy of the given value, but not with the original message, so the subscribers still receives the original sent message.
{
"id":"messageID",
"value":"messageValue"
}
I can republish the message using the same topic BUT as the topic has an attached rule, after the republishing, the rule's action is triggered again and again in a recursive loop)...
All the AWS examples I've read are meant to take the message, transform it , and do some other different thing with it (save in DynamoDB, save in a Bucket, end to Salesforce....) but none of them modify the message to be sent.
So what I'm looking for is a way to receive the message, add a field (or more) to it and let it flow in the same topic .
What is the simplest way to do this?
Related
Problem: My use case is I want to publish thousends of messages to Google Cloud Pub/Sub with a 5min retention period but only retrieve specific messages by their ID - So a cloud function will retrieve one message by ID using the Nodejs SDK and all the untreated messages will be deleted by the retention policy. All the current examples mention are to handle random messages from the subscriber.
Is it possible to just pull 1 message by id or any other metadata and close the connection.
There is no way to retrieve individual messages by ID, no. It doesn't really fit into the expected use cases for Cloud Pub/Sub where the publishers and subscribers are meant to be decoupled, meaning the subscriber inherently doesn't know the message IDs prior to receiving the messages.
You may instead want to transmit the messages via whatever mechanism you are using to making the subscribers aware of the message IDs. Or, if you know at publish time which messages will ultimately need to be retrieved, you could add an attribute to the message to indicate this and use filtering.
currently i have a cloud-build application which is being kicked off by a pub-sub trigger , subscribing to eg. topic1
I would like to know if i can kick off another cloud-build application from subscribing to the same topic. Is there a way to configure the message (or the trigger) so that if message1 is published to topic1, then cloudbuild1 is kicked off, and if message2 is published to topic1, then cloudbuild2 is kicked off?
Kind regards
marco
When you create a subscription on a topic, all the published messages in the topic are replicated in each subscription.
Therefore, if you have TOPIC and Sub1 and Sub2, if you publish 1 message in TOPIC, you will have this message in Sub1 and Sub2.
However, you can set up a filter on messages when you create a subscription. You can set this filter only at the creation and you can't update it later. You need to delete and recreate the subscription if you want to update the filter.
In addition, you can filter only on message attributes, not on the message body content.
Therefore, with filter, think wisely your filter from the beginning and when you publish a message in TOPIC, add attributes that allow your to route the messages to the correct subscription.
I know how to do this in java, but I just can't figure it out in Go at all.
All I want to do, is have a way to detect that an item got created in an S3 bucket, then have that trigger an SNS topic, which then notifies me of the file location in S3.
Has anybody got a working example of how I can do the go side of this for subscribing to the SNS topic or the SNS queue if I need one? Because all I seem to be able to find is Java and Node. I can find publish examples for go, but they are of little use to my use case.
To use SNS you will need a simple HTTP/HTTPS endpoint to receive SNS notifications. Which is divided into two parts (Confirm the subscription and Processing messages from HTTP/HTTPS endpoint)
1. Confirm the subscription
Do something as simple as this:
func confirmSubscription(subcribeURL string) {
response, err := http.Get(subcribeURL)
if err != nil {
fmt.Printf("Unbale to confirm subscriptions")
} else {
fmt.Printf("Subscription Confirmed sucessfully. %d", response.StatusCode)
}
}
2. Processing messages from HTTP/HTTPS endpoint
Parse the request's body, the documentations mentions how the body should be structured.
Sources:
https://docs.aws.amazon.com/sns/latest/dg/sns-http-https-endpoint-as-subscriber.html
https://github.com/viveksyngh/aws-sns-subscriber/blob/master/subscriber/subscriber.go
I am publishing a String message as message payload using SNS notification from Raspberry Pi using Python program and I want to pass that message payload to a Lambda function.
I have configured the requirement in the SNS console on AWS i.e., I have created a topic and added the lambda function to its subscribers.
Now, I want to get that message payload in the lambda function. But I can't find any method that can help me do that. For example, something like getMessage or something similar to that.
So my questions are: Since I have configured the publishing and subscription on AWS, can I assume that the clients are connected and if I publish a message I should be getting that at the subscriber's end which is my lambda function here?
Also, what's the technique in which I can get the message payload in my lambda function?
I am adding the below as per cjwfuller's suggestion.
Below I have written down the method for publishing in Python
client_boto = boto3.client('sns', aws_access_key_id='###',
aws_secret_access_key='###', region_name='us-west-2')
REGION = 'us-west-2'
TOPIC = 'arn:aws:sns:us-west-2:***:topic_name'
MSG = ntpath.basename(f_string)
SUBJECT_boto = 'File Name'
pub =client_boto.publish(TopicArn = TOPIC, Message = MSG,
Subject=SUBJECT_boto)
I am writing the subscribing code in Java.
Since my lambda func is already subscribed to it on AWS console,
should my Java program include the subscription again or is there a
way to get the msg payload directly.
Which language are you writing the function in? JavaScript as an example:
exports.myHandler = function(event, context, callback) {
console.log("value1 = " + event.key1);
console.log("value2 = " + event.key2);
// ...
}
Source: http://docs.aws.amazon.com/lambda/latest/dg/nodejs-prog-model-handler.html
It's useful testing the sort of stuff in the AWS Lambda console before writing all the code.
Since I have configured the publishing and subscription on AWS, can I assume that the clients are connected
Clients aren't really "connected", they're simply subscribed to a topic.
publish a message I should be getting that at the subscriber's end which is my lambda function here?
Sounds like you're doing the right sort of thing - posting example code will help us come up with more precise answers.
On searching, I have found the class for SNSEvent which is, https://github.com/aws/aws-lambda-java-libs/blob/master/aws-lambda-java-events/src/main/java/com/amazonaws/services/lambda/runtime/events/SNSEvent.java
This Class contains all the methods related to and needed to get the message payload.
The Lambda function handler in Java goes something like this,
example;
public void handleRequest(SNSEvent input, Context context){
String this_takes_message=input.getRecords().get(0).getSNS().getMessage();
}
I'm trying to follow along the amazon sns publish example using the Amazon documentation site but it's vague on Message, MessageAttributes and MessageStructure.
First of all, is the Message property going to be a string even if you set MessageStructure to json? e.g. If I want to send an object instead of just a string message. If it's string - do I need to JSON.stringify my object before passing it as a Message property?
http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/SNS.html#publish-property
Should I be doing this using MessageAttributes instead? What is that property - the amazon documentation merely states it "Message attributes for Publish action" which seems like a tautology.
http://docs.aws.amazon.com/sns/latest/APIReference/API_Publish.html
Setting MessageStructure to json is only used if you are going to send a json-formatted message structure to SNS in the specific jormat SNS understands. This is only used when you are publishing to multiple endpoint types and want to vary the message body by endpoint type. This isn't the same as "I want to send a message where the body has been serialized as JSON."
If you are sending "a JSON object," you need to stringify it, and send it just as you would any other (non-JSON) messages, because SNS messages are, fundamentally, strings.
MessageAttributes are something else entirely. They allow you to send pseudo-out-of-band key/value pairs along with your message, which can be useful for example if your message has been gzipped and base64 encoded (again, for example) you could attach an "external_id" attribute that the recipient could evaluate to decide whether it needed to unpack the whole message or could just discard it.