Src Code
class ReportingUtil {
companion object {
private val gson = Gson()
private val amazonSNS = AmazonSNSClientBuilder.standard().build()
private val topicArn = AppConfig.findString(TOPIC_ARN)
}
/**
* This function will convert object to json format and will publish to SNS topic.
* [invoiceData] -
*/
fun publishFailureInvoiceToSNS(invoiceData: InvoiceData) {
val jsonInvoiceMap = gson.toJson(invoiceData)
log.info("The Invoice : $invoiceData is converted to json format : $jsonInvoiceMap")
amazonSNS.publish(topicArn, jsonInvoiceMap)
log.info("Json invoice: $jsonInvoiceMap is published to the sns topic")
}
}
Unit Test
#Test
fun `test to publish invoice data to SNS topic`(){
initConfig()
val amazonSNS = mockk<AmazonSNS>()
val invoiceData = InvoiceData("a", "a", "a", BigDecimal(6), "a", "a", mutableMapOf("a" to "a"))
val reportingUtil = ReportingUtil()
every {
amazonSNS.publish("topicArn", "jsonInvoiceMap")
} returns PublishResult().withMessageId("MESSAGE_ID")
reportingUtil.publishFailureInvoiceToSNS(invoiceData)
}
This is giving me following error on src publish line,
com.amazonaws.services.sns.model.InvalidParameterException: Invalid
parameter: TopicArn or TargetArn Reason: no value for required
parameter (Service: AmazonSNS; Status Code: 400; Error Code:
InvalidParameter; Request ID: 5f1732ac-d63e-5e13-964c-65c312e218d7;
Proxy: null)
My use-case does not allow me to use dependency injection, therefore I made client static here.
I also want to add my custom optimization plans in catalyst which will also be triggered at build time. Is there any way to do all this before execution?
The amazonSNS you have mocked is not the one used to publish data in publishFailureInvoiceToSNS is not the one. So, when you call publishFailureInvoiceToSNS you are actually calling the real SNS API.
You have to refactor your code to use some kind of dependency injection. At least you should make it configurable. There are no use-cases that "does not allow to use DI".
If you're using JUnit 5, you could use these helpers for AWS clients injections in test. I would also recommend using localstack for unit tests.
Related
I have set up lambda function, firehose, and S3 bucket to save deleted DynamoDB data to S3.
My lambda function was written in C#.
var client = new AmazonKinesisFirehoseClient();
try
{
context.Logger.LogInformation($"Write to Kinesis Firehose: {list.Count}");
var request = new PutRecordBatchRequest
{
DeliveryStreamName = _kinesisStream,
Records = new List<Amazon.KinesisFirehose.Model.Record> ()
};
foreach (var item in list)
{
var stringWrite = new StringWriter();
string json = JsonConvert.SerializeObject(item, new JsonSerializerSettings { NullValueHandling = NullValueHandling.Ignore });
byte[] byteArray = UTF8Encoding.UTF8.GetBytes(ToLiteral(json));
var record = new Amazon.KinesisFirehose.Model.Record
{
Data = new MemoryStream(byteArray)
};
request.Records.Add(record);
}
if (request.Records.Count > 0)
{
var response = await client.PutRecordBatchAsync(request);
Console.WriteLine($"FailedPutCount: {response.FailedPutCount} status: {response.HttpStatusCode}");
}
}
catch (Exception e)
{
Console.WriteLine(e);
}
The "list" is a list of objects
There are some messages in Firehose logs:
"message": "Check your function and make sure the output is in required format. In addition to that, make sure the processed records contain valid result status of Dropped, Ok, or ProcessingFailed",
"errorCode": "Lambda.FunctionError"
I also see some error msg S3 bucket:
Error>
AccessDenied
Access Denied
TCG5YV3ZM3EQ4DWE
jRDkHxATNADXilsiy59IYkkechd6nqlyAEe0UDuN7qaNZS3zEIjblZJS9mGMktdCSb8AIFUam5I=
However, when when I downloaded the error file. I saw the following:
attemptsMade":4,"arrivalTimestamp":1661897166462,"errorCode":"Lambda.FunctionError","errorMessage":"Check your function and make sure the output is in required format. In addition to that, make sure the processed records contain valid result status of Dropped, Ok, or ProcessingFailed","attemptEndingTimestamp":1661897241573,"rawData":"XXXXXXXXX"
The "rawData" can be decoded to the original json string writing to firehose using
https://www.base64decode.org/
to decode it.
I have struggled for a couple of days. Please help.
I found my problem. The lambda function to put records to firehose works. The problem was I enabled the data transformation and add C# lambda there. That causes the data format issue. The data transformation function needs to return a different format data.
The solution is to disable the Data transformation.
I'm trying to add todo from lambda function in AWS.
I created flutter project, added api (Graphql basic Todo sample), added function (enabled mutations).
Lambda is working and effectively adding entry to TODO list. Unfortunately, subscription started in flutter returns error:
Cannot return null for non-nullable type: 'AWSDateTime' within parent 'Chat' (/onCreateChat/createdAt)
I see this problem got solved in Github where aleksvidak states.
I had the similar problem and what I realised is that mutation which in fact triggers the subscription has to have the same response fields as the ones specified for the subscription response. This way it works for me.
This seems to solve many people's problem. Unfortunately, I don't understand what it means for basic TODO sample code.
Mutation:
type Mutation {
createTodo(input: CreateTodoInput!, condition: ModelTodoConditionInput): Todo
...
Subscription:
type Subscription {
onCreateTodo(filter: ModelSubscriptionTodoFilterInput): Todo
#aws_subscribe(mutations: ["createTodo"])
...
Isn't this code aligned with what Aleksvidak said? Mutation has (Todo) the same response type as Subscription (Todo), right?
For my case it was missing updatedAt within Query that I perform inside Lambda function.
const query = /* GraphQL */ `
mutation CREATE_TODO($input: CreateTodoInput!) {
createTodo(input: $input) {
id
name
updatedAt // <-- this was missing
}
}
`;
What would be the best way to make a step function workflow wait for an sqs topic to be empty and then continue the workflow. Is there a way to check a topics length so i can react to in within a step function workflow? And would this really be the proper way to tackle this problem ?
Or would i just create a lambda that would do exactly that and trigger it in my step function workflow
Thanks in advance :)
You can build Lambda functions that uses the Lambda runtime API. Then within that Lambda function use the SQS Java API. Look at retrieving the queue attributes using this method.
https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/services/sqs/SqsClient.html#getQueueAttributes-software.amazon.awssdk.services.sqs.model.GetQueueAttributesRequest-
Look at the following attribute.
ApproximateNumberOfMessages - Returns the approximate number of visible messages in a queue
Here is a code example that uses the SQS Java V2 API to obtain number of messages.
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.sqs.SqsClient;
import software.amazon.awssdk.services.sqs.model.*;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
public class GetQueueAttributes {
public static void main(String[] args) {
final String USAGE = "\n" +
"Usage: AddQueueTags <queueName>\n\n" +
"Where:\n" +
" queueName - the name of the queue to which tags are applied.\n\n";
if (args.length != 1) {
System.out.println(USAGE);
System.exit(1);
}
String queueName = args[0];
SqsClient sqsClient = SqsClient.builder()
.region(Region.US_WEST_2)
.build();
try {
GetQueueUrlResponse getQueueUrlResponse =
sqsClient.getQueueUrl(GetQueueUrlRequest.builder().queueName(queueName).build());
String queueUrl = getQueueUrlResponse.queueUrl();
// Specify the attributes to retrieve.
List<QueueAttributeName> atts = new ArrayList();
atts.add(QueueAttributeName.APPROXIMATE_NUMBER_OF_MESSAGES);
GetQueueAttributesRequest attributesRequest= GetQueueAttributesRequest.builder()
.queueUrl(queueUrl)
.attributeNames(atts)
.build();
GetQueueAttributesResponse response = sqsClient.getQueueAttributes(attributesRequest);
Map<String,String> queueAtts = response.attributesAsStrings();
for (Map.Entry<String,String> queueAtt : queueAtts.entrySet())
System.out.println("Key = " + queueAtt.getKey() +
", Value = " + queueAtt.getValue());
} catch (SqsException e) {
System.err.println(e.awsErrorDetails().errorMessage());
System.exit(1);
}
}
}
What is nice is you can create different Lambda functions and then hook each Lambda function as a workflow step by using Step Functions. This is a good way to build workflows and have each step in that workflow use the AWS Java API to perform a specific AWS Service operation.
To learn how to hook in Lambda functions into a workflow, see this tutorial.
Create AWS serverless workflows by using the AWS SDK for Java
I Am trying to publish a Json Message to AWS SNS topic from my C# Application using AWS SDk. Its [enter image description here][1]populating message in string format and message attribute filed is not populated.
Code sample is as below:
var snsClient = new AmazonSimpleNotificationServiceClient(accessId, secretrkey, RegionEndpoint.USEast1);
PublishRequest publishReq = new PublishRequest()
{
TargetArn = topicARN,
MessageStructure = "json",
Message = JsonConvert.SerializeObject(message)
};
var msgAttributes = new Dictionary<string, MessageAttributeValue>();
var msgAttribute = new MessageAttributeValue();
msgAttribute.DataType = "String";
msgAttribute.StringValue = "123";
msgAttributes.Add("Objectcd", msgAttribute);
publishReq.MessageAttributes = msgAttributes;
PublishResponse response = snsClient.Publish(publishReq);
Older question but answering as I came across when dealing with similar issue
When you set the MessageStructure to "json".
The json must contain at least a top-level JSON key of "default" with a value that is a string.
So json needs to look like
{
"default" : "my message"
}
My solution looks something like
var messageDict = new Dictionary<string,object>()
messageDict["default"] = "my message";
PublishRequest publishReq = new PublishRequest()
{
TargetArn = topicARN,
MessageStructure = "json",
Message = JsonConvert.SerializeObject(messageDict)
};
// if json is an object
// then
messageDict["default"] = JsonConvert.SerializeObject(myMessageObject);
I'm am using PublishAsync on v3
From the documentation
https://docs.aws.amazon.com/sdkfornet/v3/apidocs/items/SNS/TPublishRequest.html
Message structure
Gets and sets the property MessageStructure.
Set MessageStructure to json if you want to send a different message for each protocol. For example, using one publish action, you can send a short message to your SMS subscribers and a longer message to your email subscribers. If you set MessageStructure to json, the value of the Message parameter must:
be a syntactically valid JSON object; and
contain at least a top-level JSON key of "default" with a value that is a string.
You can define other top-level keys that define the message you want to send to a specific transport protocol (e.g., "http").
Valid value: json
Great coincidence!
I was just busy writing a C# implementation to publish a message to SNS when I stumbled up on this post. Hopefully this helps you.
The messageBody argument we pass down to PublishMessageAsync is a string, it can be deserialized JSON for example.
public class SnsClient : ISnsClient
{
private readonly IAmazonSimpleNotificationService _snsClient;
private readonly SnsOptions _snsOptions; // You can inject any options you want here.
public SnsClient(IOptions<SnsOptions> snsOptions, // I'm using the IOptionsPattern as I have the TopicARN defined in the appsettings.json
IAmazonSimpleNotificationService snsClient)
{
_snsOptions = snsOptions.Value;
_snsClient = snsClient;
}
public async Task<PublishResponse> PublishMessageAsync(string messageBody)
{
return await _snsClient.PublishAsync(new PublishRequest
{
TopicArn = _snsOptions.TopicArn,
Message = messageBody
});
}
}
Also note the above setup uses Dependency Injection, so it would require you to set up an ISnsClient and you register an instance when bootstrapping the application, something as following:
services.TryAddSingleton<ISnsClient, SnsClient>();
I see in the examples how to to pass a message string to amazon sns sdk's publish method. However, is there an exmaple of how to pass a custom object as the message? I tried setting "MessageStructure" to "json" but then I get InvalidParameter: Invalid parameter: Message Structure - No default entry in JSON message body error. Where should I be passing the object values into in the params?
Any examples?
var params = {
Message: JSON.stringify(item),
MessageStructure: 'json',
TopicArn: topic
//MessageAttributes: item
};
return sns.publishAsync(params);
There is no SDK-supported way to pass a custom object as a message-- messages are always strings. You can, of course, make the string a serialized version of your object.
MessageStructure: 'json' is for a different purpose-- when you want to pass different strings to different subscription types. In that case, you make the message a serialized json object with AWS-defined structure, where each element defines the message to send to a particular type of subscription (email, sqs, etc). Even in that case, the messages themselves are just strings.
MessageAttributes are parameters you add to the message to support specific subscription types. If you are using SNS to talk to Apple's IOS notification service, for example, you might have to supply additional message parameters or authentication keys-- MessageAttributes provide a mechanism to do this. This is described in this AWS documentation.
An example is shown here: https://docs.aws.amazon.com/sns/latest/api/API_Publish.html#API_Publish_Example_2
The JSON format for Message is as follows:
{
"default": "A message.",
"email": "A message for email.",
"email-json": "A message for email (JSON).",
"http": "A message for HTTP.",
"https": "A message for HTTPS.",
"sqs": "A message for Amazon SQS."
}
So, assuming what you wanted to pass is an object, the way it worked for me was:
const messageObjToSend = {
...
}
const params = {
Message: JSON.stringify({
default: JSON.stringify( messageObjToSend )
}),
MessageStructure: 'json',
TopicArn: 'arn:aws:sns...'
}
Jackson 2 has pretty good support to convert object to JSON String and vice versa.
To String
Cat c = new Cat();
ObjectMapper mapper = new ObjectMapper();
String s = mapper.writeValueAsString(c);
To Object
Cat obj = mapper.readValue(s,Cat.class);
The message needs to be a JSON object and the default property needs to be added and should contain the JSON you want included in the email.
var defaultMessage = { "default": item };
var params = {
Message: defaultMessage, /*JSON.stringify(item),*/
---------^
MessageStructure: 'json',
TopicArn: topic
//MessageAttributes: item
};
return sns.publishAsync(params);
Using python,
boto3.client("sns").publish(
TopicArn=sns_subscription_arn,
Subject="subject",
Message=json.dumps({"default": item}),
--------^
MessageStructure="json",
)
FYI, if you go to this SNS topic in the AWS Console you can "publish message" and choose "Custom payload for each delivery protocol.". Here you will see a template of the email and the "default" property is tagged for "Sample fallback message".