I am using AWS Websocket api-gateway. I am sending message to connected client by aws:execute-api:region:account-id:api-id/stage-name/POST/#connections.However, the message is getting delayed somehow and the UI won't be able to receive the chat message immediately. I don't why any suggestion where getting something wrong????
One thing very exceptional, when I am sending message, last message getting time.
Example : If I sent one message that take much time. If I send 1st message, as I sent second message, First message will received to client immediately but now 2nd second message take much time. SO basically when I am sending nth message, then every nth message will deliver to client when (n+1)th message sent and (n+1)th messages takes much time to deliver.
Your help much appreciated!!!
This is only Code Snippet which I have
import * as AWS from 'aws-sdk';
export class AWSWebsocketGateway {
websocketInstance;
constructor() {
AWS.config.update({
accessKeyId: <accessKeyId>,
secretAccessKey: <secretAccessKey>,
region: <region>
});
this.websocketInstance = new AWS.ApiGatewayManagementApi({
endpoint: <webSocketDomainName> + <webSocketStage>
});
}
sendMessage(connection, messageObject, callback: Function) {
messageObject = JSON.stringify(messageObject);
this.websocketInstance.postToConnection({ ConnectionId: connection, Data: JSON.stringify(messageObject) }, function (error, data) {
console.log('Error', error, 'Send Message...', data);
return callback(error, data);
});
}
}
I retrieving connection Id from DB in one query and calling this function in loop to send message.
For anyone else who get's bitten by this one ->
You need to wait for the API call to return it's result before finishing the lambda's execution.
E.g. this won't work
module.exports.ping = async (event) => {
... //setting up api gateway, etc
apig.postToConnection({ ConnectionId: connectionId, Data: JSON.stringify(message) }, (err, res) => {})
return {
statusCode: 200
}
}
But this will:
module.exports.ping = async (event) => {
... //setting up api gateway, etc
await apig.postToConnection({ ConnectionId: connectionId, Data: JSON.stringify(message) }).promise();
return {
statusCode: 200
}
}
Any async code that is executing after the return of the lambda appears to be paused til that handler is run again, which is why you see the delayed n+ issue you outlined above.
Related
I'm using GCP PubSub (push subscription to be precisely), and Cloud Run to execute subscribed messages.
Recently I've noticed that Cloud Run executes a same message for five times.
I see there is a retry policy for a subscription which its message has been unhandled (or mishandled), but Cloud Run clearly is giving 200 OK response.
So it seems that sending 200 OK response is not enough, but I cannot find a way to send ack sign properly.
Here's the code to publish message.
function publish(message: string) {
const pubsub = new PubSub({ projectId: 'my-project' });
const dataBuffer = Buffer.from(message);
const topicName = 'my-topic';
pubsub
.topic(topicName)
.publish(dataBuffer)
.then((messageId) => {
console.log(`Message ${messageId} published`);
})
.catch((err) => {
console.log(err);
});
}
publish(JSON.stringify({ foo : 'bar' }));
And here's the code of Cloud Run.
// express app
app.post('/run', async (req, res) => {
try {
const body = req.body.message ? Buffer.from(req.body.message.data, 'base64').toString() : req.body;
// req.body came out to be "{"foo":"bar"}"
const deliveredMessage = JSON.parse(body);
// do something
// Do I have to do something like message.ack() here?
return res.status(200).end();
} catch (e) {
return res.status(502).end();
}
});
These are example responses of Cloud Run retrying.
Here's a graph of unacked messages at that time.
Is your cloud run app running when you did the post?
Because there is a timeout for the messages waiting in the queue, maybe pub/sub is resending the messages before you start to consuming it.
Another thing you can try is putting the response at the first line, like:
app.post('/run', async (req, res) => {
try {
res.status(200)
...
I have a SeedIndicatorInformationSQS-dev.fifo queue (FiFo) that connects to a SeedIndicatorInformationSQS-dev-d13dfe0 lambda. I'd like to send a message within the SeedIndicatorInformationSQS-dev-d13dfe0 lambda to the EvaluationConfigSQS-dev standard queue. But no messages are being sent/received. Whereas if I try sending it from a non-SQS connected lambda (via AppSync) it works.
The SeedIndicatorInformationSQS-dev-d13dfe0 lambda has the following permissions:
I have checked:
That the lambda has the right access to send SQS messages (you can see it there).
That the EvaluationConfigSQS-devstandard queue is correctly configured as I've successfully sent messages to it from another lambda (non-SQS).
That the SQS URL is correct.
That no errors are shown in the console.
That async/await are correctly placed (I've tried both with and without them)
Here's the CloudWatch log for the SeedIndicatorInformationSQS-dev-d13dfe0lambda trying to dispatch the content: Successfully sent to the right URL, JSON parsed to string, but nothing.
Here's the CloudWatch Log: You can see. SeedIndicatorInformationSQS-dev-d13dfe0 successfully receives the message from another lambda function and processes it, but no further messages are sent.
No errors reported within SeedIndicatorInformationSQS-dev-d13dfe0
No logs within EvaluationConfigSQS-dev
But, if I try to send it within a non-SQS lambda, it works.
Received event:
This is the classes-dev-eefa2af lambda that sends successfully to EvaluationConfigSQS-dev (and coincidentally is the one which triggers the SeedIndicatorInfromationSQS-dev.fifo SQS.
Here are the permissions for EvaluationConfigSQS-dev-6da8b90 (lambda that the EvaluationConfigSQS-dev standard queue triggers)
By any chance, do I need to add special permissions to the SeedIndicatorInformatioNSQS-dev.fifo queue?
Here's the JS that gets dispatched (I'm using a mediator pattern, and it's successfully getting dispatched, you can see it in the logs above "Dispatching CREATED_INSTITUTION_CLASS". I have also managed to print the URL and verified that it's actually the one that corresponds to it.
export async function institutionClassCreatedEventHandler(
evt: InstitutionClassCreatedEvent
) {
const json = JSON.stringify({
...evt,
type: "CLASS_CREATED",
});
sqsDispatchMessage(
"InstitutionClassCreatedEvent",
evt.tenantId + evt.subject.id,
json,
Config.SQS.evaluationConfigSQS.url,
false
);
}
Here's the sqsDispatchMessage function. As you can see, there's a catch block that will print me whenever there's an error (and it works). But so far, no error has been recorded.
export async function sqsDispatchMessage(
eventName: string,
uniqueId: string,
jsonObjStringifiedToSend: string,
sqsURL: string,
isFifoQueue: boolean = true
) {
try {
await sqs
.sendMessage({
MessageAttributes: {
EventName: {
DataType: "String",
StringValue: eventName,
},
},
...(isFifoQueue && { MessageGroupId: eventName }),
MessageBody: jsonObjStringifiedToSend,
QueueUrl: sqsURL,
...(isFifoQueue && { MessageDeduplicationId: uniqueId }),
})
.promise();
} catch (e) {
console.error(`Error While Sending the ${eventName}`);
console.error(e.message);
console.log(jsonObjStringifiedToSend);
}
}
Any ideas? Is it even possible?
The problem was in my dispatcher:
It used to be like this:
export async function dispatchOfEvents({
type,
evtArgs,
}: MediatorEvents): Promise<void> {
logTime(type);
(events as any)[type].forEach((evt: Function) => {
evt(evtArgs);
});
}
I changed it to:
export async function dispatchOfEvents({
type,
evtArgs,
}: MediatorEvents): Promise<void> {
logTime(type);
const evts: Promise<any>[] = [];
for (const evt of (events as any)[type]) {
evts.push(evt(evtArgs));
}
await Promise.all(evts);
}
We are using Cloud Function to process images and sending a notification to PubSub once processing is complete. The function is built using nodejs 8 [beta]
Here's the snippet:
const PubSubMessage = require('#google-cloud/pubsub');
const pubsub = new PubSubMessage();
const htmlData = JSON.stringify({ url: url, html: Buffer.from(html).toString('base64') });
const htmlDataBuffer = Buffer.from(htmlData);
pubsub
.topic("projects/<project-id>/topics/<name>")
.publisher()
.publish(htmlDataBuffer)
.then(messageId => {
console.log('Message ${messageId} published.');
})
.catch(err => {
console.error('ERROR in sending to pubsub:', err);
console.log('ERROR in sending to pubsub:', err);
});
The message does get published to the PubSub queue, however, it is pushed ~90 seconds after the image processing is complete.
Any idea why pushing message to PubSub might be taking so long?
-anurag
I have created a FIFO SQS queue.
When sending a message to the queue using the below params,
var params= {
MessageBody: payload,
QueueUrl: sqsURL + body.device + ".fifo"
}
sqs.sendMessage(params, function(err, res) {
err ? callback(err) : callback(null, res);
});
I get a warning that says "missing parameter messagegroupid"
I then proceed to add in MessageGroupId into my params.
var params= {
MessageBody: payload,
MessageGroupId: "posts",
QueueUrl: sqsURL + body.device + ".fifo"
}
sqs.sendMessage(params, function(err, res) {
err ? callback(err) : callback(null, res);
});
SQS then throws me "UnexpectedParameter: Unexpected key 'MessageGroupId' found in params".
Am confused by this set of conflicting instructions.
I'm using the same SDK region in both my createQueue and sendMessage command.
var sqs= new aws.SQS({ region: "us-east-2" });
I ran into this same error using Lambda (hence finding your post). Wondering if they forgot to refresh the native SDK, I uploaded the latest version of the SDK(2.7.7) with my function and the error went away.
Since uploading the SDK removes the ability to edit in Lambda, I then deleted it and re-uploaded the function the error came back :(.
So for now, if you want to use it with Lambda, you have to include the SDK with your deployment package.
You've missed passing "MessageGroupId" parameter. You should pass Message group id as a parameter. It is mandatory for FIFO.
Please refer the below-mentioned link,
https://docs.aws.amazon.com/aws-sdk-php/v3/api/api-sqs-2012-11-05.html#sendmessage
I have been searching all over the web and nothing gives a clear answer to confirm the subscription request from amazon SNS. I already send the subscription from the amazon console to my website, but what's next? I am using amazon EC2 as my server with PHP.
Before you even configure the HTTP/HTTPS endpoint subscription through AWS management console, you need to make sure that the HTTP or HTTPS endpoint of your PHP web site has the capability to handle the HTTP POST requests that Amazon SNS generates. There are several types of SNS messages: SubscriptionConfirmation, Notification and UnsubscribeConfirmation. Your PHP code needs to get the header x-amz-sns-message-type from request and process it based on the message type. For SubscriptionConfirmation message, your PHP application needs to process the POST message body, which is a JSON document. In order to subscribe the topic, your PHP code needs to visit the "SubscriberURL" specified in the JSON body. Optionally, you should verify the signature to make sure the authenticity of message before subscribing the topic.
You can find more details on AWS documentation: http://docs.aws.amazon.com/sns/latest/dg/SendMessageToHttp.html
Here is an express application (Node.js) which confirms the SNS subscription:
const express = require('express')
const request = require('request')
// parse urlencoded request bodies into req.body
const bodyParser = require('body-parser')
const app = express()
const port = 8080
app.use(bodyParser.urlencoded({ extended: false }))
app.use(bodyParser.json())
app.post('/', (req, res) => {
let body = ''
req.on('data', (chunk) => {
body += chunk.toString()
})
req.on('end', () => {
let payload = JSON.parse(body)
if (payload.Type === 'SubscriptionConfirmation') {
const promise = new Promise((resolve, reject) => {
const url = payload.SubscribeURL
request(url, (error, response) => {
if (!error && response.statusCode == 200) {
console.log('Yess! We have accepted the confirmation from AWS')
return resolve()
} else {
return reject()
}
})
})
promise.then(() => {
res.end("ok")
})
}
})
})
app.listen(port, () => console.log('Example app listening on port ' + port + '!'))
To use it one needs to install required packages:
yarn add express request body-parser
Once you confirm the subscription AWS will send a POST request to the server with the following content:
{
"Type": "SubscriptionConfirmation",
"MessageId": "XXXXXXXX-1ee3-4de3-9c69-XXXXXXXXXXXX",
"Token": "SECRET_TOKEN",
"TopicArn": "arn:aws:sns:us-west-2:XXXXXXXXXXXX:ses-test",
"Message": "You have chosen to subscribe to the topic arn:aws:sns:us-west-2:XXXXXXXXXXXX:ses-test. To confirm the subscription, visit the SubscribeURL included in this message.",
"SubscribeURL": "https://sns.us-west-2.amazonaws.com/?Action=ConfirmSubscription&TopicArn=arn:aws:sns:us-west-2:XXXXXXXXXXXX:ses-test&Token=SECRET_TOKEN",
"Timestamp": "2018-11-21T19:48:08.170Z",
"SignatureVersion": "1",
"Signature": "SECRET",
"SigningCertURL": "https://sns.us-west-2.amazonaws.com/SimpleNotificationService-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.pem"
}
The payload contains SubscribeURL which is requested by the server.
The end point you have specified will get data from AWS SNS endpoint verification service, The same end point will be used to verify the end point and to get notifications from aws,
Simply dump the input sent by AWS SNS into one text file like,
$json_write_to_text = json_decode(file_get_contents("php://input"));
You will find all data sent by AWS SNS, but just find SubscriptionUrl (which will be specific for endpoint having valid token), Open this in browser you will have SubscriptionConfirmation status. That's it
Enjoy.
Spring cloud SNS subscription with annotation
spring cloud AWS has support for auto confirmation of subscriber, you just need to put this annotation "#NotificationSubscriptionMapping"
#Controller
#RequestMapping("/topicName")
public class NotificationTestController {
#NotificationSubscriptionMapping
public void handleSubscriptionMessage(NotificationStatus status) throws IOException {
//We subscribe to start receive the message
status.confirmSubscription();
}
#NotificationMessageMapping
public void handleNotificationMessage(#NotificationSubject String subject, #NotificationMessage String message) {
// ...
}
#NotificationUnsubscribeConfirmationMapping
public void handleUnsubscribeMessage(NotificationStatus status) {
//e.g. the client has been unsubscribed and we want to "re-subscribe"
status.confirmSubscription();
}
}
http://cloud.spring.io/spring-cloud-aws/spring-cloud-aws.html#_sns_support
I solved this using NodeJS backend. Lets say you have an API like this in HapiJS (Well it doesnt matter you can have another tech)
{
method: 'POST',
path: '/hello',
handler: ( request, reply ) => {
reply( Hello.print(request.payload) );
},
config: {
tags: ['api']
}
}
Just pass the payload you receive, on to your business logic.
In the business logic process it like this
'use strict';
const request = require('request');
exports.print = (payload) => {
payload = JSON.parse(payload);
if(payload.Type === 'SubscriptionConfirmation'){
return new Promise((resolve, reject) => {
const url = payload.SubscribeURL;
request(url, (error, response) => {
if (!error && response.statusCode == 200) {
console.log('Yess! We have accepted the confirmation from AWS');
return resolve();
}
else
return reject();
});
});
}
I am using request module from NPM to automatically accept such requests.
Another way would be to print the contents of payload and then click on the URL given in payload.SubscribeURL.
Once AWS accepts it you check the confirmation on the Subscriptions page where Subscription ARN would be changed from Pending Confirmation to a complex name-cum-SHA having your Topic name.