I am trying to publish multiple message at time (around 50) but Pub/Sub is giving is Deadline Exceeded at /user_code/node_modules/#google-cloud/pubsub/node_modules/grpc/src/client.js:55 error.
const pubsub = PubSub();
const topic = pubsub.topic('send_wishes');
const publisher = topic.publisher();
//data is dictionary object
Object.keys(data).forEach(function(key){
var userObj = data[key];
const dataBuffer = Buffer.from(JSON.stringify(userObj));
const publisher = topic.publisher();
publisher.publish(dataBuffer)
.then((results) => {
const messageId = results[0];
console.log(`Message ${messageId} published.`);
return;
});
})
For single message it's working fine. For batching I try the batch configuration of publisher but it is also not working
const publisher = topic.publisher({
batching: {
maxMessages: 15,
maxMilliseconds: 2000
}
});
Once creating subscription please change Acknowledgement Deadline of subscription time for default 10 sec to 100 sec.
Related
I am using Lambda. I want to send a subsegment to Xray with an custom end_time. Xray is enabled in my Lambda.
When I use the aws-xray-sdk-core and addNewSubsegment('postgres') I do not find the possibility to add an end_time. It looks like the end_time is being set when you close() the Segment.
To try and solve this limitation I base myself on the following to send a custom segment to the Xray Daemon using UDP.
Use UDP to send segment to XRay
Below code is not sending a SubSegment to Xray. I am not receiving any errors when sending the segment with client.send(...).
Does someone knows more about this limitation of setting an custom end_time/ knows if it's possible with UDP inside a Lambda?
import AWSXRay from 'aws-xray-sdk-core'
const traceDocument = {
trace_id,
id: generateRandomHex(16),
name: 'postgres',
start_time,
end_time,
type: 'subsegment',
parent_id,
sql: { sanitized_query: query },
}
const client = createSocket('udp4')
const udpSegment = `{"format": "json", "version": 1}\n${traceDocument}`
client.send(udpSegment, 2000, '127.0.0.1', (e, b) => {
console.log(e, b)
})
Managed to find out the solution myself
used the X-Ray SDK with a combination of
addAttribute('in_progress', false) and
streamSubsegments() to send the subsegments to X-Ray
export const onQueryEvent = async (e) => {
try {
const segment = AWSXRay.getSegment()
if (segment) {
const paramsArr = JSON.parse(e.params)
const query = getQueryWithParams(e.query, paramsArr) // X-Ray wants the time in seconds -> ms * 1e-3
const start_time = e.timestamp.valueOf() * 1e-3
const end_time = (e.timestamp.valueOf() + e.duration) * 1e-3
// Add a new Subsegment to parent Segment
const subSegment = segment.addNewSubsegment('postgres') // Add data to the segment
subSegment.addSqlData({ sanitized_query: query })
subSegment.addAttribute('start_time', start_time)
subSegment.addAttribute('end_time', end_time) // Set in_progress to false so subSegment
// will be send to xray on streamSubsegments()
subSegment.addAttribute('in_progress', false)
subSegment.streamSubsegments()
}
} catch (e) {
console.log(e)
}
}
I have a GC Function that triggers cron-job. Almost everything works well, except I get error in Logs "Service unavailable" even though it works (runs API call).
This my code:
exports.helloPubSub = (event, context) => {
const message = event.data
? Buffer.from(event.data, 'base64').toString()
: 'Function';
console.log("message:", message);
const url = `example.com`
const targetAudience = '12345';
const {GoogleAuth} = require('google-auth-library');
const auth = new GoogleAuth();
async function request() {
const client = await auth.getIdTokenClient(targetAudience);
const res = await client.request({
'url': url,
'method': 'post',
'data': {
'token': 'myToken'
}});
console.info(res.data);
}
request().catch(err => {
console.error(err.message);
process.exitCode = 1;
});
};
The error "Service unavailable" appears after 5 minutes as the function starts. The function is performed for 7 minutes every 10 minutes, so there is time gap between next starting.
The question is, is it something with the function or with time-limit in GC? Any thoughts?
Updated:
This is my logs:
According to the official documentation:
HTTP status and error codes for JSON
503—Service Unavailable is a backendError (internal error), and you are encouraged to try again using truncated exponentioal backoff.
This is my setup.
Subscription A is a push subscription that POSTs messages to a cloud Run deployment.
That deployment exposes an HTTP endpoint, processes the message, posts the result to Topic B, and responds 200 to subscription A's POST request. The whole process takes ~1.5 seconds.
Therefore, for every message in subscription A, I should end up with 1 message in Topic B.
This is how my code looks like
My app started an Express server
const express = require('express');
const bodyParser = require('body-parser');
const _ = require('lodash');
const startBrowser = require('./startBrowser');
const tab = require('./tab');
const createMessage = require('./publishMessage');
const domain = 'https://example.com';
require('dotenv').config();
const app = express();
app.use(bodyParser.json());
const port = process.env.PORT || 8080;
app.listen(port, async () => {
console.log('Listening on port', port);
});
The endpoint where all the magic happens
app.post('/', async (req, res) => {
// Define the success and fail functions, that respond status 200 and 500 respectively
const failed = () => res.status(500).send();
const completed = async () => {
const response = await res.status(200).send();
if (response && res.writableEnded) {
console.log('successfully responded 200');
}
};
//Process the data coming from Subscription A
let pubsubMessage = decodeBase64Json(req.body.message.data);
let parsed = await processor(pubsubMessage);
//Post the processed data to topic B
let messageId = await postParsedData(parsed);
if (messageId) {
// ACK the message once the data has been processed and posted to topic B.
completed();
} else {
console.log('Didnt get a message id');
// failed();
}
});
//define the functions that post data to Topic B
const postParsedData = async (parsed) => {
if (!_.isEmpty(parsed)) {
const topicName = 'topic-B';
const messageIdInternal = await createMessage(parsed, topicName);
};
return messageId;
} else {
console.log('Parsed is Empty');
return null;
}
};
function decodeBase64Json(data) {
return JSON.parse(Buffer.from(data, 'base64').toString());
}
Execution time takes about ~1.5 seconds and I can see the successful responses logged on Cloud run every ~1.5seconds. That adds up to about ~2400messages/hour (per cloud run instance).
Topic B is getting new messages at ~2400messages/hour, Subscription A's acknowledgement rate is ~200messages/hour, which leads to the messages being re-delivered many times.
Subscription A's Acknowledgement deadline is 600 seconds.
The request timeout period in Cloud run is 300 seconds.
I've tried ACKing messages before they're published to topic-B or even before parsing, but I'm getting the same result.
Edit: added screenshot of the pending messages and processed messages. Many more messages processed than ACKed pending messages. Should be 1:1
Thanks for your help
Solution This error could not be reproduced by GCP support. It didn't happen with large amounts of Cloud Run VMs. The solution is just to increase the number of worker instances
You need to await your complete(); function call. like this
....
if (messageId) {
// ACK the message once the data has been processed and posted to topic B.
await completed();
} else {
console.log('Didnt get a message id');
// failed();
}
The Acknowledgment Deadline is 10 seconds. When I use asynchronous-pull way to process the message, I don't call message.ack() and message.nack(), wait for the message ack deadline and expect Pub/Sub redeliver this message.
After waiting over 10 seconds, the subscriber doesn't receive the message again. Here is my code:
subscriber:
import { pubsubClient, IMessage, parseMessageData } from '../../googlePubsub';
import { logger } from '../../utils';
const topicName = 'asynchronous-pull-test';
const subName = 'asynchronous-pull-test';
const subscription = pubsubClient.topic(topicName).subscription(subName);
const onMessage = (message: IMessage) => {
const { data, ...rest } = message;
const jsonData = parseMessageData(data);
logger.debug('received message', { arguments: { ...rest, data: jsonData } });
const publishTime = new Date(message.publishTime).getTime();
const republishTimestamp = Date.now() - 5 * 1000;
if (publishTime < republishTimestamp) {
logger.info('message acked');
message.ack();
} else {
logger.info('push message back to MQ');
}
};
logger.info('subscribe the MQ');
subscription.on('message', onMessage).on('error', (err: Error) => logger.error(err));
publisher:
const topicName = 'asynchronous-pull-test';
async function main() {
const messagePayload = { email: faker.internet.email(), campaignId: '1' };
await pub(topicName, messagePayload);
}
main();
I am using "#google-cloud/pubsub": "^0.19.0",
I expect the subscriber will receive the message again at the ack deadline 10 seconds later. Which means my subscriber receives and processes the message every 10 seconds. Am I wrong?
The Google Cloud Pub/Sub client libraries automatically call modifyAckDeadline for messages that are neither acked or nacked for a configurable period of time. In node.js, this is configured via the maxExtension property:
const options = {
flowControl: {
maxExtension: 60, // Specified in seconds
},
};
const subscription = pubsubClient.topic(topicName).subscription(subName, options);
In general, it is not a good practice to not ack/nack a message as a means to delay its redelivery. This will result in the message still counting against the flow control max outstanding messages, meaning it could prevent the delivery of future messages until the originally received messages are acked or nacked. At this time, Cloud Pub/Sub does not have a means by which to delay message redelivery, but it is something under consideration.
You should nack the message to tell Pub/Sub to redeliver it. Documentation
I am trying to create HTTP API in Cloud Function - that eventually published a message t PubSub. Understood, that there is PubSub REST API - but it enforced me to set up the authentication (in client side) - that I would like to skip and move it to the server side.
Below code is deployed as Google Cloud Function with this command gcloud functions deploy helloGET --runtime nodejs8 --trigger-http
But while tested in browser, it is errored out Error: could not handle the request
Any suggestion is appreciated, thanks!
"use strict";
// [START functions_pubsub_setup]
const { PubSub } = require("#google-cloud/pubsub");
// Instantiates a client
const pubsub = new PubSub();
// [END functions_pubsub_setup]
const Buffer = require("safe-buffer").Buffer;
exports.helloGET = (req, res) => {
const topic = pubsub.topic("projects/myproject/topics/openit");
const message = {
data: {
message: "req.body.message"
}
};
// Publishes a message
res.send(
topic
.publish(message)
.then(() => res.status(200).send("Message published."))
.catch(err => {
err = `Catch block ... ${err}`;
console.error(err);
res.status(500).send(err);
return Promise.reject(err);
})
);
};
Below code will work. But it will take around 30 seconds or plus for the subscriber to receive the event - it is way too slow for my used case :S
"use strict";
const { PubSub } = require("#google-cloud/pubsub");
const pubsub = new PubSub();
const Buffer = require("safe-buffer").Buffer;
exports.helloGET = async (req, res) => {
var toPublish = `hello ${Date.now()}`;
publishMessage("_REPLACE_WITH_TOPIC_NAME_", toPublish);
res.send(`Published ${toPublish}`);
};
async function publishMessage(topicName, data) {
console.log("=> publishMessage, data = ", data);
const dataBuffer = Buffer.from(data);
const topic = pubsub.topic(topicName);
const publisher = topic.publisher();
publisher.publish(dataBuffer, { a: "XYZ" }, function() {
console.log("Published eventually ...");
});
}