My thing would publish string payload something like "[v1:ThingName]" to a topic (someTopic/topic1). And I have a rule applied in the rule engine to capture this topic and send to a Lambda function.
SELECT * FROM 'someTopic/+'
I want to send the topic1 part which will be captured from Rule engine along with the payload. So it can be captured from topic(2).
How do I concatenate topic(2) with * in select statement?
After so much of trial and error, figured that you cannot do it like that because payload is string.
But how I got away with this was using the encode function. Essentially I encode the whole payload into base64 as a json object and then have the topic(2) value as well.
So my Rule SQL query would like this.
SELECT encode(*, 'base64') as encode, topic(2) as topic FROM 'someTopic/+'
Subsequently you decode the payload in the Lambda function.
Related
Is it possible to access the raw url using AWS API Gateway (and Lambda)?
Alternatively, is it possible to access the original, undecoded query string paramters?
We are integrating against a third party service, that calls our API and encodes the query string params from Windows-1252. (E.g. the finnish letter Ä is encoded as %C4 instead of %C3%84). API Gateway seems to automatically decode the query string parameters and assume UTF-8, which means, that Ä (and Ö and Å) result in \ufffd.
For reference: https://www.w3schools.com/tags/ref_urlencode.asp
Damn, it really doesn't look possible...
I started off writing how you can use Lambda Proxy Integration with event.queryStringParameters, but that parses the data into a key-value object.
Then I went down the road of Mapping Templates in API Gateway, but again there doesn't seem to be any property that shows the whole querystring.
As much as I didn't want it to be true, I can only conclude that it is not possible...
I think your best option is to encode the parameter as base64 on the client, then decode in the Lambda function using Object.keys(event.queryStringParameters)[0].
I have created a rule to send the incoming IoT messages to a S3 bucket.
The problem is that any time IoT recieves a messages is sended and stored in a new file (with the same name) in S3.
I want this S3 file to keep all the data from before and not truncate each time a new message is stored.
How can I do that?
When you set up an IoT S3 rule action, you need to specify a bucket and a key. The key is what we might think of as a "path and file name". As the docs say, we can specify the key string by using a substitution template, which is just a fancy way of saying "build a path out of these pieces of information". When you are building your substitution template, you can reference fields inside the message as well as use use a bunch of other functions
Especially look at the functions topic, timestamp, as well as some of the string manipulator functions.
Let's say your topic names are something like things/thing-id-xyz/location and you just want to store each incoming JSON message in a "folder" for the thing-id it came in from. You might specify a key like:
${topic(2)}/${timestamp()).json
it would evaluate to something like:
thing-id-xyz/1481825251155.json
where the timestamp part is the time the message came in. That will be different for each message, and then the messages would not overwrite each other.
You can also specify parts of the message itself. Let's imagine our incoming messages look something like this:
{
"time": "2022-01-13T10:04:03Z",
"latitude": 40.803274,
"longitude": -74.237926,
"note": "Great view!"
}
Let's say you want to use the nice ISO date value you have in your data instead of the timestamp of the file. You could reference the time field no problem, like:
${topic(2)}/${time}.json
Now the file would be written as the key:
thing-id-xyz/2022-01-13T10:04:03Z.json
You should be able to find some combination of values that works for your needs, and that most importantly, is UNIQUE for each message so they don't overwrite each other in S3.
You can do it using AWS IoT SQL variable expressions. For example use following as a key ${newuuid()}. This will create new s3 object for each message received.
See more about SQL Functions https://docs.aws.amazon.com/iot/latest/developerguide/iot-sql-functions.html
You can't do this with the S3 IoT Rule Action. You can get similar results using AWS Firehose, which will batch up several messages and write to one file. You will still end up with multiple files though.
I have setup a rule in cloudwatch to monitor Glue ETL. In the state change I am sending a notification to SNS. I have modified the input transformer to get a custom body of the email but not getting how to change the subject line of the email . It still giving the default "AWS Notification Message"
My Input transformer :
{"state":"$.detail.state"}
"The JOB has changed state to <state>."
Use a Lambda function—rather than the Amazon SNS topic—as a target for the CloudWatch Events rule. Then, configure the Lambda function to publish a custom message to the Amazon SNS topic when triggered by the CloudWatch Events rule.
Documented here: https://aws.amazon.com/premiumsupport/knowledge-center/change-sns-email-for-cloudwatch-events/
Transformer (no way Jose)
As far as I can tell there is currently no way to control the email subject with a transformer. Typically you will control the notification body for a rule through the transformer which modifies the input json message (e.g. in the case of a build https://docs.aws.amazon.com/codebuild/latest/userguide/sample-build-notifications.html#sample-build-notifications-ref ). Based on what I see in the documentation this only modifies part of the body embedded between the header and the footer of the email payload.
JSON (also not possible)
1. Since all notifications are generated with an API call with json payload you can experiment and configure. Using the CLI you can specify a json format using --message-structure attribute. However the subject is not part of the json payload itself and is sent as a separate parameter "--subject" (see example below) you won't be able to configure that unless they either modify the UI or the json payload.
2.In order to exercise greater control over your output you might have to use JSON (select "Constant (JSON text)") which is documented for mobile messaging https://docs.aws.amazon.com/sns/latest/dg/mobile-push-send-custommessage.html but not very well for HTTP https://docs.aws.amazon.com/sns/latest/dg/sns-message-and-json-formats.html but decent for the CLI https://docs.aws.amazon.com/cli/latest/reference/sns/publish.html
3. You can go to the console https://console.aws.amazon.com/sns/v2/ and click on "Publish a Message" which allows you to specify a subject. Notice that there is a "JSON message generator" but that's only for the body.
Coding Workaround (possible ...kinda)
If you feel really determined you can explore a workaround: look at the API and figure out what call is equivalent to sending a call which includes a subject. Create a lambda function that executes that call. From the rule invoke the lambda :-) and you are done. If there is will, there is a way...
Notes:
aws sns publish --topic-arn arn:aws:sns:us-east-1:652499160872:DP-Build --message-structure json --subject "Test Build subject" --message "{ \"default\":\"Foo\", \"email\":\"Bar\"}"
According to the docs there is a "Subject" key you can pass as a parameter:
Blockquote
Subject
The Subject parameter specified when the notification was published to the topic. Note that this is an optional parameter. If no Subject was specified, then this name/value pair does not appear in this JSON document.
Blockquote
set "detail-type":"Glue ETL State-change Notification"
you might need to look at https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/CloudWatch-Events-Input-Transformer-Tutorial.html
I've got a lambda function called by an IoT rule and I would like to know the topic name from inside this lambda function.
So far i'm only able to retrieve the message data from the event parameter. Nothing in the context parameter neither.
I haven't found anything in the documentation...
Is it even possible ?
You need to use topic() function in AWS IoT SQL query. Like this:
SELECT * as data, topic() as topic FROM 'desired/+/topic'
In this case, your event will include the original message in 'data' field and the used topic in 'topic' field. You can also use integer number as a parameter inside topic() function, to return only sub-group.
More data in oficial documentation: http://docs.aws.amazon.com/iot/latest/developerguide/iot-sql-functions.html#iot-function-topic
I'm a newbie rails programmer, and I have even less experience with all the AWS products. I'm trying to use lambda to subscribe to and consume an rss feed from youtube. I am able to send the subscription request just fine with HTTParty from my locally hosted rails app:
query = {'hub.mode':'subscribe', 'hub.verify':'sync', 'hub.topic': 'https://www.youtube.com/feeds/videos.xml?channel_id=CHANNELID', 'hub.callback':'API Endpoint for Lambda'}
subscribe = 'HTTParty.post(https://pubsubhubbub.appspot.com/subscribe, :query=>query)
and it will ping the lambda function with a get request. I know that I need to echo back a hub.challenge string, but I don't know how. The lambda event is empty, I didn't see anything useful in the context. I tried formatting the response in the API gateway but that didn't work either. So right now when I try to subscribe I get back a 'Challenge Mismatch' error.
I know this: https://pubsubhubbub.googlecode.come/git/pubsubhubbub-core-0.3.html#subscribing explains what I'm trying to do better than what I just did, and section 6.2.1 is where the breakdown is. How do I set up either the AWS Lambda function and/or the API Gateway to reflect back the 'hub.challenge' verification token string?
You need to use the parameter mapping functionality of API Gateway to map the parameters from the incoming query string to a parameter passed to your Lambda function. From the documentation link you provided, it looks like you'll at least need to map the hub.challenge query string parameter, but you may also need the other parameters (hub.mode, hub.topic, and hub.verify_token) depending on what validation logic (if any) that you're implementing.
The first step is to declare your query string parameters in the method request page. Once you have declared the parameters open the integration request page (where you specify which Lambda function API Gateway should call) and use the "+" icon to add a new template. In the template you will have to specify a content type (application/json), and then the body you want to send to Lambda. You can read both query string and header parameters using the params() function. In that input mapping field you are creating the event body that is posted to AWS Lambda. For example: { "challenge": "$input.params('hub.challenge')" }
Documentation for mapping query string parameters