Why sending different message to AWS SQS return same message ID - amazon-web-services

I want to send messages to a FIFO sqs queue. Given a array of list different user ids, for each id, I want to call sendMessage command to send the id as message body. I'm expecting every time it will return a different message id, but actually they all return same messageId. Sample code below:
const sendMessage = async (params:ISqsRequestParam) => {
try {
const sqsResponse = await sqsClient.send(new SendMessageCommand(params));
console.log(`send message response: ${JSON.stringify(sqsResponse)}`);
return sqsResponse.MessageId; // For unit tests.
} catch (err) {
console.error('SQS sending', err);
}
};
export const handler = async function (event: IEventBridgeAddtionalParams, context: Context): Promise<string[]> {
console.info(`${context.functionName} triggered at ${event.time} under ${process.env.NODE_ENV} mode`);
console.info(`customor parameter value is ${event.custom_parameter}`);
try {
const sqsUrl: string = event.custom_parameter === 'Creator' ? process.env.BATCH_CREATOR_QUEUE_URL : process.env.BATCH_PROCESSOR_QUEUE_URL;
console.info(`SQS Url is: ${sqsUrl}`);
const tenantData: ITenantResponse = await fetchAllTenantIds();
console.info(`ResponseDate from tenant service: ${JSON.stringify(tenantData.value)}`);
// change to fix tenantId for development environment for better debugging and test
const data : ITenantDetails[] = process.env.NODE_ENV !== 'production' ? fixedTenantDataForNonProd() : tenantData.value;
const promise = data.map(async tenantDetails => {
if (!tenantDetails.tenantFailed) {
console.info(`Tenant Id in message body: ${tenantDetails.id}`);
const params: ISqsRequestParam = {
MessageBody: tenantDetails.id,
MessageDeduplicationId: `FP_Tenant_populator_${event.custom_parameter}`, // Required for FIFO queues
MessageGroupId: `FP_Tenant_populator_Group_${event.custom_parameter}`, // Required for FIFO queues
QueueUrl: sqsUrl //SQS_QUEUE_URL; e.g., 'https://sqs.REGION.amazonaws.com/ACCOUNT-ID/QUEUE-NAME'
};
const messageId:string = await sendMessage(params);
return messageId; //for unit testing
} else {
return null; //for unit testing
}
});
const messageIds:string[] = await Promise.all(promise); //for unit testing
const activeMessageIds:string[] = messageIds.filter(id=> id!= null); //for unit testing
console.info(`Success, ${activeMessageIds.length} messages sent. MessageID:${activeMessageIds[0]}`);
return activeMessageIds; //for unit testing
} catch (error) {
console.error(`Fetch tenant details error: ${error}`);
}
};

Related

AWS Websocket API

I am using the API Gateway Websocket.
When I send a message to the client from a Lambda function, it sometimes doesn't send a message and other times it sends 2-3 messages.
const AWS = require("aws-sdk")
const api = new AWS.ApiGatewayManagementApi({
endpoint : process.env.API_ENDPOINT
})
exports.handler = async (event) => {
console.log(event)
const body = JSON.parse(event.body)
const connectionId = event.requestContext.connectionId
sendMessage(connectionId, "My Message")
return {}
};
const sendMessage = (connectionId, response) => {
const data = { message : response }
const params = {
ConnectionId : connectionId,
Data : Buffer.from(JSON.stringify(data))
}
return api.postToConnection(params).promise()
}
Actually i had to await the sendMessage function. This solved my problem.

Using AWS Lambda Console to send push using SNS

I tried every possible solution on the internet with no hope
What I am trying to do is simply use aws lambda functions (through the aws console) to fetch user fcm token from lets say DynamoDB (not included in the question), use that token to create endpointArn, send push to that specific device
I tested to send Using SNS console and the push gets to the device successfully but I failed to get it to the device using Lambda functions although it gives success status and message ID
Here is the code I used
// Load the AWS SDK for Node.js
var AWS = require('aws-sdk');
// Set region
AWS.config.update({region: 'us-east-1'});
const sns = new AWS.SNS()
const sampleMessage = {
"GCM": {
"notification": {
"body": "Sample message for Android endpoints",
"title":"Title Test"
}
}
}
exports.handler = async (event) => {
const snsPayload = JSON.stringify(sampleMessage);
const response = {
statusCode: 200,
body: JSON.stringify('Hello from Lambda!'),
};
const params = {
PlatformApplicationArn: '<Platform Arn>',
Token: '<FCM Token>'
};
try {
const endpointData = await sns.createPlatformEndpoint(params).promise();
const paramsMessage = {
Message: snsPayload,
TargetArn: endpointData.EndpointArn
};
var publishTextPromise = await sns.publish(paramsMessage).promise();
response.MessageId = publishTextPromise.MessageId;
response.result = 'Success';
}
catch (e) {
console.log(e.stack)
response.result = 'Error'
}
return response;
};
After some trials and errors I figured out the solution for my own question
1- The GCM part of the payload should be a string not a json
2- The message parameter should have an attribute that explicitly sets the mime type of the payload to Json
Taking all that into consideration
const GCM_data = {
'notification': {
'body': 'Hellow from lambda function',
'title': 'Notification Title'
}
}
const data = {
"GCM": JSON.stringify(GCM_data)
}
const snsPayload = JSON.stringify(data)
and the params should look like
const paramsMessage = {
Message: snsPayload,
TargetArn: endpointData.EndpointArn,
MessageStructure: 'json'
};
and this will work :)

Self invoking lambda invocation timing out

We're trying to develop a self-invoking lambda to process S3 files in chunks. The lambda role has the policies needed for the invocation attached.
Here's the code for the self-invoking lambda:
export const processFileHandler: Handler = async (
event: S3CreateEvent,
context: Context,
callback: Callback,
) => {
let bucket = loGet(event, 'Records[0].s3.bucket.name');
let key = loGet(event, 'Records[0].s3.object.key');
let totalFileSize = loGet(event, 'Records[0].s3.object.size');
const lastPosition = loGet(event, 'position', 0);
const nextRange = getNextSizeRange(lastPosition, totalFileSize);
context.callbackWaitsForEmptyEventLoop = false;
let data = await loadDataFromS3ByRange(bucket, key, nextRange);
await database.connect();
log.debug(`Successfully connected to the database`);
const docs = await getParsedDocs(data, lastPosition);
log.debug(`upserting ${docs.length} records to database`);
if (docs.length) {
try {
// upserting logic
log.debug(`total documents added: ${await docs.length}`);
} catch (err) {
await recurse(nextRange.end, event, context);
log.debug(`error inserting docs: ${JSON.stringify(err)}`);
}
}
if (nextRange.end < totalFileSize) {
log.debug(`Last ${context.getRemainingTimeInMillis()} milliseconds left`);
if (context.getRemainingTimeInMillis() < 10 * 10 * 10 * 6) {
log.debug(`Less than 6000 milliseconds left`);
log.debug(`Invoking next iteration`);
await recurse(nextRange.end, event, context);
callback(null, {
message: `Lambda timed out processing file, please continue from LAST_POSITION: ${nextRange.start}`,
});
}
} else {
callback(null, { message: `Successfully completed the chunk processing task` });
}
};
Where recurse is an invocation call to the same lambda. Rest of the things work as expected it just times out whenever the call stack comes on this invocation request:
const recurse = async (position: number, event: S3CreateEvent, context: Context) => {
let newEvent = Object.assign(event, { position });
let request = {
FunctionName: context.invokedFunctionArn,
InvocationType: 'Event',
Payload: JSON.stringify(newEvent),
};
let resp = await lambda.invoke(request).promise();
console.log('Invocation complete', resp);
return resp;
};
This is the stack trace logged to CloudWatch:
{
"errorMessage": "connect ETIMEDOUT 63.32.72.196:443",
"errorType": "NetworkingError",
"stackTrace": [
"Object._errnoException (util.js:1022:11)",
"_exceptionWithHostPort (util.js:1044:20)",
"TCPConnectWrap.afterConnect [as oncomplete] (net.js:1198:14)"
]
}
Not a good idea to create a self-invoking lambda function. In case of an error (could also be a bad handler call on AWS side) a lambda function might re-run several times. Very hard to monitor and debug.
I would suggest using Step Functions. I believe this tutorial can help Iterating a Loop Using Lambda
From the top of my head, if you prefer not dealing with Step Functions, you could create a Lambda trigger for an SQS queue. Then you pass a message to the queue if you want to run the lambda function another time.

Step Function Triggered In a loop

I am starting a step function from Lambda and the Lambda function is tied to an API Gateway. For some reason, when I try to test the Lambda function, I see hundreds of executions failed and running in loop. I just triggered the step function once. I am missing something here. Can you please advise.
const AWS = require("aws-sdk");
const uuidv4 = require("uuid/v4");
/*----------------------------------------------------------------------- */
/* Implementation */
/*----------------------------------------------------------------------- */
exports.handler = async event => {
var _dt = await ExecuteStepFunction()
return _dt;
}
function ExecuteStepFunction() {
const stepFunctions = new AWS.StepFunctions();
return new Promise((res, rej) => {
var params = {
stateMachineArn: 'arn:aws:states:us-east-1:xxxxxxxxxxxxx:stateMachine:xxTestSateMachine',
input: JSON.stringify(''),
name: uuidv4()
};
stepFunctions.startExecution(params, function (err, data) {
if (err) {
rej(err);
}
else {
res(data);
}
});
});
}
I tried thIS approach provided in the this link (https://docs.aws.amazon.com/step-functions/latest/dg/tutorial-api-gateway.html) where the API gateway directly triggers the step function but I am receiving the following error. After trying to fix this, I move to the above option of starting the function using the API.
{
"__type": "com.amazon.coral.service#UnrecognizedClientException",
"message": "The security token included in the request is invalid"
}

Cloud Function (trigger by HTTP) that would publish a message to PubSub

I am trying to create HTTP API in Cloud Function - that eventually published a message t PubSub. Understood, that there is PubSub REST API - but it enforced me to set up the authentication (in client side) - that I would like to skip and move it to the server side.
Below code is deployed as Google Cloud Function with this command gcloud functions deploy helloGET --runtime nodejs8 --trigger-http
But while tested in browser, it is errored out Error: could not handle the request
Any suggestion is appreciated, thanks!
"use strict";
// [START functions_pubsub_setup]
const { PubSub } = require("#google-cloud/pubsub");
// Instantiates a client
const pubsub = new PubSub();
// [END functions_pubsub_setup]
const Buffer = require("safe-buffer").Buffer;
exports.helloGET = (req, res) => {
const topic = pubsub.topic("projects/myproject/topics/openit");
const message = {
data: {
message: "req.body.message"
}
};
// Publishes a message
res.send(
topic
.publish(message)
.then(() => res.status(200).send("Message published."))
.catch(err => {
err = `Catch block ... ${err}`;
console.error(err);
res.status(500).send(err);
return Promise.reject(err);
})
);
};
Below code will work. But it will take around 30 seconds or plus for the subscriber to receive the event - it is way too slow for my used case :S
"use strict";
const { PubSub } = require("#google-cloud/pubsub");
const pubsub = new PubSub();
const Buffer = require("safe-buffer").Buffer;
exports.helloGET = async (req, res) => {
var toPublish = `hello ${Date.now()}`;
publishMessage("_REPLACE_WITH_TOPIC_NAME_", toPublish);
res.send(`Published ${toPublish}`);
};
async function publishMessage(topicName, data) {
console.log("=> publishMessage, data = ", data);
const dataBuffer = Buffer.from(data);
const topic = pubsub.topic(topicName);
const publisher = topic.publisher();
publisher.publish(dataBuffer, { a: "XYZ" }, function() {
console.log("Published eventually ...");
});
}