I see in the examples how to to pass a message string to amazon sns sdk's publish method. However, is there an exmaple of how to pass a custom object as the message? I tried setting "MessageStructure" to "json" but then I get InvalidParameter: Invalid parameter: Message Structure - No default entry in JSON message body error. Where should I be passing the object values into in the params?
Any examples?
var params = {
Message: JSON.stringify(item),
MessageStructure: 'json',
TopicArn: topic
//MessageAttributes: item
};
return sns.publishAsync(params);
There is no SDK-supported way to pass a custom object as a message-- messages are always strings. You can, of course, make the string a serialized version of your object.
MessageStructure: 'json' is for a different purpose-- when you want to pass different strings to different subscription types. In that case, you make the message a serialized json object with AWS-defined structure, where each element defines the message to send to a particular type of subscription (email, sqs, etc). Even in that case, the messages themselves are just strings.
MessageAttributes are parameters you add to the message to support specific subscription types. If you are using SNS to talk to Apple's IOS notification service, for example, you might have to supply additional message parameters or authentication keys-- MessageAttributes provide a mechanism to do this. This is described in this AWS documentation.
An example is shown here: https://docs.aws.amazon.com/sns/latest/api/API_Publish.html#API_Publish_Example_2
The JSON format for Message is as follows:
{
"default": "A message.",
"email": "A message for email.",
"email-json": "A message for email (JSON).",
"http": "A message for HTTP.",
"https": "A message for HTTPS.",
"sqs": "A message for Amazon SQS."
}
So, assuming what you wanted to pass is an object, the way it worked for me was:
const messageObjToSend = {
...
}
const params = {
Message: JSON.stringify({
default: JSON.stringify( messageObjToSend )
}),
MessageStructure: 'json',
TopicArn: 'arn:aws:sns...'
}
Jackson 2 has pretty good support to convert object to JSON String and vice versa.
To String
Cat c = new Cat();
ObjectMapper mapper = new ObjectMapper();
String s = mapper.writeValueAsString(c);
To Object
Cat obj = mapper.readValue(s,Cat.class);
The message needs to be a JSON object and the default property needs to be added and should contain the JSON you want included in the email.
var defaultMessage = { "default": item };
var params = {
Message: defaultMessage, /*JSON.stringify(item),*/
---------^
MessageStructure: 'json',
TopicArn: topic
//MessageAttributes: item
};
return sns.publishAsync(params);
Using python,
boto3.client("sns").publish(
TopicArn=sns_subscription_arn,
Subject="subject",
Message=json.dumps({"default": item}),
--------^
MessageStructure="json",
)
FYI, if you go to this SNS topic in the AWS Console you can "publish message" and choose "Custom payload for each delivery protocol.". Here you will see a template of the email and the "default" property is tagged for "Sample fallback message".
Related
When using console.log to add log rows to AWS CloudWatch, the Lambda Request ID is added on each row as described in the docs
A simplified example based on the above mentioned doc
exports.handler = async function(event, context) {
console.log("Hello");
return context.logStreamName
};
Would produce output such as
START RequestId: c793869b-ee49-115b-a5b6-4fd21e8dedac Version: $LATEST
2019-06-07T19:11:20.562Z c793869b-ee49-115b-a5b6-4fd21e8dedac INFO Hello
END RequestId: c793869b-ee49-115b-a5b6-4fd21e8dedac
REPORT RequestId: c793869b-ee49-115b-a5b6-4fd21e8dedac Duration: 170.19 ms Billed Duration: 200 ms Memory Size: 128 MB Max Memory Used: 73 MB
The relevant detail here regarding this question is the Request ID, c793869b-ee49-115b-a5b6-4fd21e8dedac which is added after the timestamp on the row with "Hello".
The AWS documentation states
To output logs from your function code, you can use methods on the console object, or any logging library that writes to stdout or stderr.
The Node.js runtime logs the START, END, and REPORT lines for each invocation, and adds a timestamp, request ID, and log level to each entry logged by the function.
When using Winston as a logger, the Request ID is lost. Could be an issued with formatters or transports. The logger is created like
const logger = createLogger({
level: 'debug',
format: combine(
timestamp(),
printf(
({ timestamp, level, message }) => `${timestamp} ${level}: ${message}`
)
),
transports: [new transports.Console()]
});
I also tried simple() formatter instead of printf(), but that has no effect on whether Request ID is present or not. Also removing formatting altogether still prints the plain text, i.e. no timestamp or request id.
I also checked the source code of Winston Console transport, and it uses either console._stdout.write if present, or console.log for writing, which is what the AWS documentation said to be supported.
Is there some way to configure Winston to keep the AWS Lambda Request ID as part of the message?
P.S. There are separate Winston Transports for AWS CloudWatch that I am aware of, but they require other setup functionality that I'd like to avoid if possible. And since the Request ID is readily available, they seem like an overkill.
P.P.S. Request ID can also be fetched from Lambda Context and custom logger object initialized with it, but I'd like to also avoid that, pretty much for the same reasons: extra work for something that should be readily available.
The issue is with the usage of console._stdout.write() / process._stdout.write(), which Winston built-in Console Transport uses when present.
For some reason lines written to stdout go to CloudWatch as is, and timestamp/request ID are not added to log rows as they are with console.log() calls.
There is a discussion on Github about making this a constructor option that could be selected on transport creation, but it was closed as a problem related to specific IDEs and how they handle stdout logs. The issue with AWS Lambdas is mentioned only as a side note in the discussion.
My solution was to make a custom transport for Winston, which always uses console.log() to write the messages and leave timestamp and request ID to be filled in by AWS Lambda Node runtime.
Addition 5/2020:
Below is an examples of my solution. Unfortunaly I cannot remember much of the details of this implementation, but I pretty much looked at Winston sources in Github and took the bare minimum implementation and forced use of console.log
'use strict';
const TransportStream = require('winston-transport');
class SimpleConsole extends TransportStream {
constructor(options = {}) {
super(options);
this.name = options.name || 'simple-console';
}
log(info, callback) {
setImmediate(() => this.emit('logged', info));
const MESSAGE = Symbol.for('message');
console.log(info[MESSAGE]);
if (callback) {
callback();
}
}
};
const logger = createLogger({
level: 'debug',
format: combine(
printf(({ level, message }) => `${level.toUpperCase()}: ${message}`)
),
transports: [new SimpleConsole()]
});
const debug = (...args) => logger.debug(args);
// ... And similar definition to other logging levels, info, warn, error etc
module.exports = {
debug
// Also export other logging levels..
};
Another option
As pointed out by #sanrodari in the comments, the same can be achieved by directly overriding the log method in built-in Console transport and force the use of console.log.
const logger = winston.createLogger({
transports: [
new winston.transports.Console({
log(info, callback) {
setImmediate(() => this.emit('logged', info));
if (this.stderrLevels[info[LEVEL]]) {
console.error(info[MESSAGE]);
if (callback) {
callback();
}
return;
}
console.log(info[MESSAGE]);
if (callback) {
callback();
}
}
})
]
});
See full example for more details
I know OP said they would like to avoid using the Lambda context object to add the request ID, but I wanted to share my solution with others who may not have this requirement. While the other answers require defining a custom transport or overriding the log method of the Console transport, for this solution you just need to add one line to the top of your handler function.
import { APIGatewayTokenAuthorizerEvent, Callback, Context } from "aws-lambda";
import { createLogger, format, transports } from "winston";
const logger = createLogger({
level: "debug",
format: format.json({ space: 2 }),
transports: new transports.Console()
});
export const handler = (
event: APIGatewayTokenAuthorizerEvent,
context: Context,
callback: Callback
): void => {
// Add this line to add the requestId to logs
logger.defaultMeta = { requestId: context.awsRequestId };
logger.info("This is an example log message"); // prints:
// {
// "level": "info",
// "message": "This is an example log message",
// "requestId": "ac1de841-ca30-4a09-9950-dd4fe7e37af8"
// }
};
Documentation for Lambda context object in Node.js
For other Winston formats like printf, you will need to add the requestId property to the format string. Not only is this more concise, but it has the benefit of allowing you to customize where the request ID appears in your log output, rather than always prepending the request ID like CloudWatch does.
As already mentioned by #kaskelloti AWS does not transforms messages logged by console._stdout.write() and console._stderr.write()
here is my modified solution which respects levels in AWS logs
const LEVEL = Symbol.for('level');
const MESSAGE = Symbol.for('message');
const logger = winston.createLogger({
transports: [
new winston.transports.Console({
log(logPayload, callback) {
setImmediate(() => this.emit('logged', logPayload));
const message = logPayload[MESSAGE]
switch (logPayload[LEVEL]) {
case "debug":
console.debug(message);
break
case "info":
console.info(message);
break
case "warn":
console.warn(message);
break
case "error":
console.error(message);
break
default:
//TODO: handle missing levels
break
}
if (callback) {
callback();
}
}
})
],
})
according to the AWS docs
To output logs from your function code, you can use methods on the console object, or any logging library that writes to stdout or stderr.
I ran a quick test using the following Winston setup in a lambda:
const path = require('path');
const { createLogger, format, transports } = require('winston');
const { combine, errors, timestamp } = format;
const baseFormat = combine(
timestamp({ format: 'YYYY-MM-DD HH:mm:ss' }),
errors({ stack: true }),
format((info) => {
info.level = info.level.toUpperCase();
return info;
})(),
);
const splunkFormat = combine(
baseFormat,
format.json(),
);
const prettyFormat = combine(
baseFormat,
format.prettyPrint(),
);
const createCustomLogger = (moduleName) => createLogger({
level: process.env.LOG_LEVEL,
format: process.env.PRETTY_LOGS ? prettyFormat : splunkFormat,
defaultMeta: { module: path.basename(moduleName) },
transports: [
new transports.Console(),
],
});
module.exports = createCustomLogger;
and in CloudWatch, I am NOT getting my Request ID. I am getting a timestamp from my own logs, so I'm less concerned about it. Not getting the Request ID is what bothers me
Is there a way to filter messages based on Regex or substring in AWS SNS?
AWS Documentation for filtering messages mentions three types of filtering for strings:
Exact matching (whitelisting)
Anything-but matching (blacklisting)
Prefix matching
I want to filter out messages based on substrings in the messages, for example
I have a S3 event that sends a message to SNS when a new object is added to S3, the contents of the message are as below:
{
"Records": [
{
"s3": {
"bucket": {
"name": "images-bucket"
},
"object": {
"key": "some-key/more-key/filteringText/additionaldata.png"
}
}
}
]
}
I want to keep the messages if only filteringText is present in key field.
Note: The entire message is sent as text by S3 notification service, so Records is not a json object but string.
From what I've seen in the documentation, you can't do regex matches or substrings, but you can match prefixes and create your own attributes in the MessageAttributes field.
To do this, I send the S3 event to a simple Lambda that adds MessageAttributes and then sends to SNS.
In effect, S3 -> Lambda -> SNS -> other consumers (with filtering).
The Lambda can do something like this (where you'll have to programmatically decide when to add the attribute):
let messageAttributes = {
myfilterkey: {DataType: "String", StringValue:"filteringText"}
};
let params = {
Message: JSON.stringify(payload),
MessageAttributes: messageAttributes,
MessageStructure: 'json',
TargetArn: SNS_ARN
};
await sns.publish(params).promise();
Then in SNS you can filter:
{"myfilterkey": ["filtertext"]}
It seems a little convoluted to put the Lambda in there, but I like the idea of being able to plug and unplug consumers from SNS on the fly and use filtering to determine who gets what.
I Am trying to publish a Json Message to AWS SNS topic from my C# Application using AWS SDk. Its [enter image description here][1]populating message in string format and message attribute filed is not populated.
Code sample is as below:
var snsClient = new AmazonSimpleNotificationServiceClient(accessId, secretrkey, RegionEndpoint.USEast1);
PublishRequest publishReq = new PublishRequest()
{
TargetArn = topicARN,
MessageStructure = "json",
Message = JsonConvert.SerializeObject(message)
};
var msgAttributes = new Dictionary<string, MessageAttributeValue>();
var msgAttribute = new MessageAttributeValue();
msgAttribute.DataType = "String";
msgAttribute.StringValue = "123";
msgAttributes.Add("Objectcd", msgAttribute);
publishReq.MessageAttributes = msgAttributes;
PublishResponse response = snsClient.Publish(publishReq);
Older question but answering as I came across when dealing with similar issue
When you set the MessageStructure to "json".
The json must contain at least a top-level JSON key of "default" with a value that is a string.
So json needs to look like
{
"default" : "my message"
}
My solution looks something like
var messageDict = new Dictionary<string,object>()
messageDict["default"] = "my message";
PublishRequest publishReq = new PublishRequest()
{
TargetArn = topicARN,
MessageStructure = "json",
Message = JsonConvert.SerializeObject(messageDict)
};
// if json is an object
// then
messageDict["default"] = JsonConvert.SerializeObject(myMessageObject);
I'm am using PublishAsync on v3
From the documentation
https://docs.aws.amazon.com/sdkfornet/v3/apidocs/items/SNS/TPublishRequest.html
Message structure
Gets and sets the property MessageStructure.
Set MessageStructure to json if you want to send a different message for each protocol. For example, using one publish action, you can send a short message to your SMS subscribers and a longer message to your email subscribers. If you set MessageStructure to json, the value of the Message parameter must:
be a syntactically valid JSON object; and
contain at least a top-level JSON key of "default" with a value that is a string.
You can define other top-level keys that define the message you want to send to a specific transport protocol (e.g., "http").
Valid value: json
Great coincidence!
I was just busy writing a C# implementation to publish a message to SNS when I stumbled up on this post. Hopefully this helps you.
The messageBody argument we pass down to PublishMessageAsync is a string, it can be deserialized JSON for example.
public class SnsClient : ISnsClient
{
private readonly IAmazonSimpleNotificationService _snsClient;
private readonly SnsOptions _snsOptions; // You can inject any options you want here.
public SnsClient(IOptions<SnsOptions> snsOptions, // I'm using the IOptionsPattern as I have the TopicARN defined in the appsettings.json
IAmazonSimpleNotificationService snsClient)
{
_snsOptions = snsOptions.Value;
_snsClient = snsClient;
}
public async Task<PublishResponse> PublishMessageAsync(string messageBody)
{
return await _snsClient.PublishAsync(new PublishRequest
{
TopicArn = _snsOptions.TopicArn,
Message = messageBody
});
}
}
Also note the above setup uses Dependency Injection, so it would require you to set up an ISnsClient and you register an instance when bootstrapping the application, something as following:
services.TryAddSingleton<ISnsClient, SnsClient>();
I have created one AWS Lex bot and I am invoking one lambda function from that bot. When testing the lambda function I am getting proper response but at bot I am getting below error:
An error has occurred: Received invalid response from Lambda: Can not
construct instance of IntentResponse: no String-argument
constructor/factory method to deserialize from String value
('2017-06-22 10:23:55.0') at [Source: "2017-06-22 10:23:55.0"; line:
1, column: 1]
Not sure, what is wrong and where I am missing. Could anyone assist me please?
The solution to above problem is that we need to make sure response returned by lambda function, to be used at AWS lex chat bot should be in below format:
{
"sessionAttributes": {
"key1": "value1",
"key2": "value2"
...
},
"dialogAction": {
"type": "ElicitIntent, ElicitSlot, ConfirmIntent, Delegate, or Close",
Full structure based on the type field. See below for details.
}
}
By this, chat bot expectd DialogAction and corresponding elements in order to process the message i.e. IntentResponse.
Reference: http://docs.aws.amazon.com/lex/latest/dg/lambda-input-response-format.html
no String-argument constructor/factory method to deserialize from String value
You are getting this error because you must be passing string values in the response of lambda function. You have to pass a predefined json object blueprint in the response.
Because the communication between Lex and Lambda is not simple value passing like normal functions. Amazon Lex expects output from Lambda in a particular JSON format and data is sent to Lambda in a particular JSON. The examples are here: Lambda Function Input Event and Response Format.
And just copying and pasting the blueprint won't work because in some fields you have choose between some predefined values and in some fields you have to entry valid input.
For example in,
"dialogAction": {
"type": "Close",
"fulfillmentState": "Fulfilled or Failed",
"message": {
"contentType": "PlainText or SSML",
"content": "Thanks, your pizza has been ordered."
}
}
you have assign a value "Fulfilled" or "Failed" to field 'fulfillmentState'. And same goes for 'contentType'.
I would like to send an SMS message from an AWS Lambda function using the boto3 publish method to notify the user of issues via SMS. My lambda function is written in Python and I am using the boto3 module. My lambda function has full rights to SNS. I have this code,
sns = boto3.client('sns')
sns.publish(
PhoneNumber = '+11234567890',
Message = 'Simple text message'
)
According to the boto3 documentation, the publish method accepts the following parameters,
response = client.publish(
TopicArn='string',
TargetArn='string',
PhoneNumber='string',
Message='string',
Subject='string',
MessageStructure='string',
MessageAttributes={
'string': {
'DataType': 'string',
'StringValue': 'string',
'BinaryValue': b'bytes'
}
}
)
It requires a "Message" parameter and one of the following three parameters as described in the docs:
TopicArn (string) -- The topic you want to publish to.
If you don't specify a value for the TopicArn parameter, you must
specify a value for the PhoneNumber or TargetArn parameters.
TargetArn (string) -- Either TopicArn or EndpointArn, but not both.
If you don't specify a value for the TargetArn parameter, you must
specify a value for the PhoneNumber or TopicArn parameters.
PhoneNumber (string) -- The phone number to which you want to deliver
an SMS message. Use E.164 format.
If you don't specify a value for the PhoneNumber parameter, you must
specify a value for the TargetArn or TopicArn parameters.
When my code is executed a parameter validation error is returned. It states,
Unknown parameter in input: "PhoneNumber", must be one of: TopicArn,
TargetArn, >Message, Subject, MessageStructure, MessageAttributes".
So the documentation seems to indicate that PhoneNumber is a valid parameter, but when used, an error occurs and the feedback from the error indicates that PhoneNumber is not a possible parameter. I suspect I am missing something obvious and simple, but could use some help.
I know there are other avenues to send SMS messages such as email gateways and other vendor supplied solutions like Twilio, but I would like to pursue the SNS based route and understand where I have gone wrong.
Actually your example looks right. Here is what I tried
import boto3
sns = boto3.client('sns')
number = '+17702233322'
sns.publish(PhoneNumber = number, Message='example text message' )
Worked like a charm. I recommend using awscli configured with your root account credentials first and take the code for a test drive. Once its working either create a new user with just the rights you need, or apply it to an Instance role.
You need to create a policy that allows SNS:Publish on resource:* (allow texting to everyone) or resource: '+17702233322' (allow text to a specific number).