I have a SeedIndicatorInformationSQS-dev.fifo queue (FiFo) that connects to a SeedIndicatorInformationSQS-dev-d13dfe0 lambda. I'd like to send a message within the SeedIndicatorInformationSQS-dev-d13dfe0 lambda to the EvaluationConfigSQS-dev standard queue. But no messages are being sent/received. Whereas if I try sending it from a non-SQS connected lambda (via AppSync) it works.
The SeedIndicatorInformationSQS-dev-d13dfe0 lambda has the following permissions:
I have checked:
That the lambda has the right access to send SQS messages (you can see it there).
That the EvaluationConfigSQS-devstandard queue is correctly configured as I've successfully sent messages to it from another lambda (non-SQS).
That the SQS URL is correct.
That no errors are shown in the console.
That async/await are correctly placed (I've tried both with and without them)
Here's the CloudWatch log for the SeedIndicatorInformationSQS-dev-d13dfe0lambda trying to dispatch the content: Successfully sent to the right URL, JSON parsed to string, but nothing.
Here's the CloudWatch Log: You can see. SeedIndicatorInformationSQS-dev-d13dfe0 successfully receives the message from another lambda function and processes it, but no further messages are sent.
No errors reported within SeedIndicatorInformationSQS-dev-d13dfe0
No logs within EvaluationConfigSQS-dev
But, if I try to send it within a non-SQS lambda, it works.
Received event:
This is the classes-dev-eefa2af lambda that sends successfully to EvaluationConfigSQS-dev (and coincidentally is the one which triggers the SeedIndicatorInfromationSQS-dev.fifo SQS.
Here are the permissions for EvaluationConfigSQS-dev-6da8b90 (lambda that the EvaluationConfigSQS-dev standard queue triggers)
By any chance, do I need to add special permissions to the SeedIndicatorInformatioNSQS-dev.fifo queue?
Here's the JS that gets dispatched (I'm using a mediator pattern, and it's successfully getting dispatched, you can see it in the logs above "Dispatching CREATED_INSTITUTION_CLASS". I have also managed to print the URL and verified that it's actually the one that corresponds to it.
export async function institutionClassCreatedEventHandler(
evt: InstitutionClassCreatedEvent
) {
const json = JSON.stringify({
...evt,
type: "CLASS_CREATED",
});
sqsDispatchMessage(
"InstitutionClassCreatedEvent",
evt.tenantId + evt.subject.id,
json,
Config.SQS.evaluationConfigSQS.url,
false
);
}
Here's the sqsDispatchMessage function. As you can see, there's a catch block that will print me whenever there's an error (and it works). But so far, no error has been recorded.
export async function sqsDispatchMessage(
eventName: string,
uniqueId: string,
jsonObjStringifiedToSend: string,
sqsURL: string,
isFifoQueue: boolean = true
) {
try {
await sqs
.sendMessage({
MessageAttributes: {
EventName: {
DataType: "String",
StringValue: eventName,
},
},
...(isFifoQueue && { MessageGroupId: eventName }),
MessageBody: jsonObjStringifiedToSend,
QueueUrl: sqsURL,
...(isFifoQueue && { MessageDeduplicationId: uniqueId }),
})
.promise();
} catch (e) {
console.error(`Error While Sending the ${eventName}`);
console.error(e.message);
console.log(jsonObjStringifiedToSend);
}
}
Any ideas? Is it even possible?
The problem was in my dispatcher:
It used to be like this:
export async function dispatchOfEvents({
type,
evtArgs,
}: MediatorEvents): Promise<void> {
logTime(type);
(events as any)[type].forEach((evt: Function) => {
evt(evtArgs);
});
}
I changed it to:
export async function dispatchOfEvents({
type,
evtArgs,
}: MediatorEvents): Promise<void> {
logTime(type);
const evts: Promise<any>[] = [];
for (const evt of (events as any)[type]) {
evts.push(evt(evtArgs));
}
await Promise.all(evts);
}
Related
I'm new to AWS and working on SQS to trigger a lambda function.
The order of the function is,
Lambda A and pass queue to SQS, then SQS trigger Lambda B.
When I run Lambda A, I see Lambda B is triggered as I expected. However, when I run Lambda A within 5 minutes after the last run of Lambda A, Lambda B is not triggered. I'm wondering if there is a default time set of "5 minutes" to use SQS triggering Lambda function. If so, how can I change to "0 minute"?
Does anybody have the similar issue before? Thanks,
// Lambda A
// sends multiple messages to an Amazon SQS queue
module.exports.handler = async (event, context, callback) => {
try {
// sendQueue function is imported in the head of the file
await sendQueue({ queueMessage }, QUEUE_URL)
} catch (e) {
console.error(e)
}
return {
statusCode: 200,
body: JSON.stringify(
{
message: 'Lambda A is done',
input: event,
},
null,
2
),
}
}
// Lambda B
// After SQS receives queues from Lambda A, SQS triggers to invoke Lambda B
module.exports.handler = async (event, context, callback) => {
// here calling Athena query with "await"
return `Successfully processed a message.`;
}
In Lambda B, it gets a message content everytime the function receives a message from SQS like below.
[
{
messageId: 'b4146e42-a542-4e0e-bff9-ba7f96xxxxxx',
receiptHandle: 'xxxxxxxxxxxxxxx',
body: '{"queueMessage": "queueMessage sample text"}',
attributes: {
ApproximateReceiveCount: '1',
SentTimestamp: '1643094105466',
SequenceNumber: '18867376164708847616',
MessageGroupId: '123',
SenderId: 'xxxxxxxx:lambda-function-name',
MessageDeduplicationId: 'xxxxxx',
ApproximateFirstReceiveTimestamp: '1643094105466'
},
messageAttributes: { ReceiveMessageWaitTimeSeconds: [Object] },
md5OfMessageAttributes: 'aaaaaaaa',
md5OfBody: 'bbbbbbb',
eventSource: 'aws:sqs',
eventSourceARN: 'arn:aws:xxx.fifo',
awsRegion: 'ap-northeast-1'
}
]
After going over AWS documentation, I found the line below
If a message with a particular message deduplication ID is sent
successfully, any messages sent with the same message deduplication ID
are accepted successfully but aren't delivered during the 5-minute
deduplication interval.
Basically, using FIFO queue of SQS, the messages sent during 5 minutes won't be delivered to avoid sending duplicate messages.
Just ensure that you have not set the below setting in SQS. This setting could delay your Lambda function to running after x time.
I am using AWS SQS service and currently trying to insert the message in the queue and I am successfully able to insert the first message but not able to insert second one using loop... it returns the same message-id and sequence number
MessageId: '****-a938-4fdc-abed-2efb1e302a2a',
SequenceNumber: '18848*******2174575616'
can anyone is able to help me below is code I am using:-
function insertMessage(data){
var sqs = new AWS.SQS({apiVersion: '2012-11-05'});
var params = {
MessageGroupId:data.messageGroupId,
MessageAttributes: {
"operation": {
DataType: "String",
StringValue: "insert_comment"
},
"comment_id": {
DataType: "Number",
StringValue: ""+data.comment_id+""
},
"timestamp": {
DataType: "String",
StringValue: ""+data.created_on+""
}
},
MessageBody: "New comment send ",
QueueUrl: "******"
};
//console.log(params);
sqs.sendMessage(params, function (err, data) {
if (err) {
console.log("Error", err);
callback(err);
} else {
console.log("Success", data.MessageId);
callback(data);
}
});
}
I am calling above function in loop.
And every time I am getting the same MessageId and SequenceNumber in response in the callback.
{ ResponseMetadata: { RequestId: '762a88d4-****-5298-***-dd3cb27f7dc1' },
MD5OfMessageBody: '33d9f3a9*****bad5f72014ea836062',
MD5OfMessageAttributes: '*****af0c13771494a53fdb2',
MessageId: '*****-****-4fdc-abed-2efb1e302a2a',
SequenceNumber: '188******174575616'
}
Apparently you don't add MessageDeduplicationId attribute which indicates that content based duplication is enabled. In this case, MessageDeduplicationId is automatically generated for you based on the message body content hash. Here, 5 minutes deduplication interval applies. If you want to add message to the queue with same message body and within the deduplication interval then you will need to generate a unique MessageDeduplicationId for each message.
References
If multiple messages are sent in succession to a FIFO queue, each with a distinct message deduplication ID, Amazon SQS stores the messages and acknowledges the transmission. Then, each message can be received and processed in the exact order in which the messages were transmitted.
MessageDeduplicationId
The token used for deduplication of sent messages. If a message with a particular message deduplication ID is sent successfully, any messages sent with the same message deduplication ID are accepted successfully but aren't delivered during the 5-minute deduplication interval.
Deduplication configuration
Enable content-based deduplication. This instructs Amazon SQS to use a SHA-256 hash to generate the message deduplication ID using the body of the message—but not the attributes of the message
Is the second message the same as the first? You can't put duplicate messages into a FIFO queue.
I am using AWS Websocket api-gateway. I am sending message to connected client by aws:execute-api:region:account-id:api-id/stage-name/POST/#connections.However, the message is getting delayed somehow and the UI won't be able to receive the chat message immediately. I don't why any suggestion where getting something wrong????
One thing very exceptional, when I am sending message, last message getting time.
Example : If I sent one message that take much time. If I send 1st message, as I sent second message, First message will received to client immediately but now 2nd second message take much time. SO basically when I am sending nth message, then every nth message will deliver to client when (n+1)th message sent and (n+1)th messages takes much time to deliver.
Your help much appreciated!!!
This is only Code Snippet which I have
import * as AWS from 'aws-sdk';
export class AWSWebsocketGateway {
websocketInstance;
constructor() {
AWS.config.update({
accessKeyId: <accessKeyId>,
secretAccessKey: <secretAccessKey>,
region: <region>
});
this.websocketInstance = new AWS.ApiGatewayManagementApi({
endpoint: <webSocketDomainName> + <webSocketStage>
});
}
sendMessage(connection, messageObject, callback: Function) {
messageObject = JSON.stringify(messageObject);
this.websocketInstance.postToConnection({ ConnectionId: connection, Data: JSON.stringify(messageObject) }, function (error, data) {
console.log('Error', error, 'Send Message...', data);
return callback(error, data);
});
}
}
I retrieving connection Id from DB in one query and calling this function in loop to send message.
For anyone else who get's bitten by this one ->
You need to wait for the API call to return it's result before finishing the lambda's execution.
E.g. this won't work
module.exports.ping = async (event) => {
... //setting up api gateway, etc
apig.postToConnection({ ConnectionId: connectionId, Data: JSON.stringify(message) }, (err, res) => {})
return {
statusCode: 200
}
}
But this will:
module.exports.ping = async (event) => {
... //setting up api gateway, etc
await apig.postToConnection({ ConnectionId: connectionId, Data: JSON.stringify(message) }).promise();
return {
statusCode: 200
}
}
Any async code that is executing after the return of the lambda appears to be paused til that handler is run again, which is why you see the delayed n+ issue you outlined above.
I have created a FIFO SQS queue.
When sending a message to the queue using the below params,
var params= {
MessageBody: payload,
QueueUrl: sqsURL + body.device + ".fifo"
}
sqs.sendMessage(params, function(err, res) {
err ? callback(err) : callback(null, res);
});
I get a warning that says "missing parameter messagegroupid"
I then proceed to add in MessageGroupId into my params.
var params= {
MessageBody: payload,
MessageGroupId: "posts",
QueueUrl: sqsURL + body.device + ".fifo"
}
sqs.sendMessage(params, function(err, res) {
err ? callback(err) : callback(null, res);
});
SQS then throws me "UnexpectedParameter: Unexpected key 'MessageGroupId' found in params".
Am confused by this set of conflicting instructions.
I'm using the same SDK region in both my createQueue and sendMessage command.
var sqs= new aws.SQS({ region: "us-east-2" });
I ran into this same error using Lambda (hence finding your post). Wondering if they forgot to refresh the native SDK, I uploaded the latest version of the SDK(2.7.7) with my function and the error went away.
Since uploading the SDK removes the ability to edit in Lambda, I then deleted it and re-uploaded the function the error came back :(.
So for now, if you want to use it with Lambda, you have to include the SDK with your deployment package.
You've missed passing "MessageGroupId" parameter. You should pass Message group id as a parameter. It is mandatory for FIFO.
Please refer the below-mentioned link,
https://docs.aws.amazon.com/aws-sdk-php/v3/api/api-sqs-2012-11-05.html#sendmessage
I have and preordered AWS snowball that has been ordered with all its events raised.
I need to know what the event is that is running (it will send a lambda when its shipped and when it arrives back to amazon, how can i get different operations for this?)
The Examples are too simple
my lambda is
var http = require('http');
exports.handler = function(event, context) {
var httpOptions = {
host: "myip",
port: '80',
path: "/createFirs",
method: "GET"
};
if (false) {
http.get(httpOptions, function(response) {
response.on('data', function (chunk) {
console.log(chunk);
//context.done(null, chunk);
context.succeed(event);
});
response.on('error', function(err) {
console.log(err);
context.done(err, null);
});
});
} else {
context.done(new Error("snowball lambda isn't complete yet"));
}
};
Information about AWS Import/Export Snowball jobs is available via the DescribeJob API.
There are also notifications generated by Snowball that are sent via Amazon SNS. Notifications include:
In transit
Delivered
Importing
Completed
You can subscribe an AWS Lambda function to an Amazon SNS topic, so that the Lambda function is called when the Snowball notification is received.