Google Cloud Tasks not executing tasks randomly - google-cloud-platform

Google Cloud Tasks seems to be unreliable some times. I've added a bunch of tasks and sometimes those are not getting executed.
When I checked the console, it shows '11 Tasks in Queue'. However, those tasks are not listed below (usually it shows correctly).
Here is how I've created the queue (upsert):
const client = require('./client');
module.exports = async (queue_name, concurrent) => {
return client.updateQueue({
queue: {
name: client.queuePath(
process.env.GCP_PROJECT,
process.env.GCP_QUEUE_LOCATION,
queue_name,
),
rateLimits: {
maxConcurrentDispatches: concurrent,
maxDispatchesPerSecond: concurrent,
maxBurstSize: 100,
},
retryConfig: {
maxAttempts: 3,
unlimitedAttempts: false,
maxRetryDuration: {
seconds: 3600,
},
minBackoff: {
seconds: 60,
},
maxBackoff: {
seconds: 300,
},
maxDoublings: 3,
},
},
});
};
and here is how I'm adding tasks:
const client = require('./client');
module.exports = async (queue_name, url, config) => {
return client.createTask({
parent: client.queuePath(
process.env.GCP_PROJECT,
process.env.GCP_QUEUE_LOCATION,
queue_name,
),
task: {
httpRequest: {
httpMethod: 'POST',
headers: {
'Content-Type': 'application/json',
},
url: `${process.env.OPTIMIZER_URL}/optimize-page`,
body: Buffer.from(JSON.stringify({ url, config })).toString(
'base64',
),
},
},
});
};
The same code works sometimes and sometimes it just got stuck at executing!

Related

Configuring Pub/Sub destination for ad-hoc Transcoder API job requests in Google Cloud

Using the Node JS client, I can't get the Transcoder API to accept the pub/sub destination.
The documentation variously says to configure the pubsub_destination field or the pubsubDestination field, but neither appear to work when creating a job using the createJob method (following and extending the example for ad-hoc job creation).
const config = {
parent: client.locationPath(projectId, location),
job: {
inputUri: inputUri,
outputUri: outputUri,
config: {
elementaryStreams: [
{
key: 'video-stream0',
videoStream: {
h264: {
heightPixels: 360,
widthPixels: 640,
bitrateBps: 550000,
frameRate: 60
}
}
},
{
key: 'audio-stream0',
audioStream: {
codec: 'aac',
bitrateBps: 64000
}
}
],
muxStreams: [
{
elementaryStreams: ['video-stream0', 'audio-stream0'],
key: 'sd',
filename: 'sd.mp4',
container: 'mp4'
}
]
},
pubsubDestination: { topic: `projects/${projectId}/topics/transcoder` }
}
};
return await client.createJob(request);
The job gets created and completes successfully, but the config field of the created job shows "pubsubDestination": null.
What am I doing wrong?
I have tried creating a job using an ad-hoc configuration template with a pubsubDestination field in the jobConfig and successfully received the job status updates from the Transcoder API to Pub/Sub topic. Try the following code for your requirement:
projectId = 'my-project-id';
location = 'us-central1';
inputUri = 'gs://my-bucket/my-video-file';
outputUri = 'gs://my-bucket/my-output-folder/';
// Imports the Transcoder library
const {TranscoderServiceClient} =
require('#google-cloud/video-transcoder').v1;
// Instantiates a client
const transcoderServiceClient = new TranscoderServiceClient();
async function createJobFromAdHoc() {
// Construct request
const request = {
parent: transcoderServiceClient.locationPath(projectId, location),
job: {
inputUri: inputUri,
outputUri: outputUri,
config: {
elementaryStreams: [
{
key: 'video-stream0',
videoStream: {
h264: {
heightPixels: 360,
widthPixels: 640,
bitrateBps: 550000,
frameRate: 60,
},
},
},
{
key: 'video-stream1',
videoStream: {
h264: {
heightPixels: 720,
widthPixels: 1280,
bitrateBps: 2500000,
frameRate: 60,
},
},
},
{
key: 'audio-stream0',
audioStream: {
codec: 'aac',
bitrateBps: 64000,
},
},
],
muxStreams: [
{
key: 'sd',
container: 'mp4',
elementaryStreams: ['video-stream0', 'audio-stream0'],
},
{
key: 'hd',
container: 'mp4',
elementaryStreams: ['video-stream1', 'audio-stream0'],
},
],
pubsubDestination: {
topic: 'projects/{project-ID}/topics/{topic-ID}'
},
},
},
};
// Run request
const [response] = await transcoderServiceClient.createJob(request);
console.log(`Job: ${response.name}`);
}
createJobFromAdHoc();
Result:

Stubbing / Spy on global.fetch in Deno

I'm just getting into Deno, one of the things I'm a little unsure about is how to stub or create a spy for the global fetch function?
One solution is to simply wrap the fetch in a function which itself can by stubbed or spied on, but that seems like an unnecessary abstraction.
Any help would be much appreciated.
With denock you can mock the return object of the fetch call. Maybe not what you want but now you can test without a real call to the server.
https://deno.land/x/denock#0.2.0
import { assertEquals } from "https://deno.land/std/testing/asserts.ts";
import { denock } from "https://deno.land/x/denock/mod.ts";
// function to test
async function fetchFromServer() {
const urlObject = new URL("https://jsonplaceholder.typicode.com/todos");
const response = await fetch(urlObject, {
method: "POST",
headers: new Headers({
"content-type": "application/json",
}),
body: JSON.stringify({
userId: 2,
id: 23024,
title: "delectus aut autem",
completed: false,
}),
});
return await response.json();
}
// mock return
denock({
method: "POST",
protocol: "https",
host: "jsonplaceholder.typicode.com",
headers: [
{
header: "content-type",
value: "application/json",
},
],
path: "/todos",
requestBody: {
userId: 2,
id: 23024,
title: "delectus aut autem",
completed: false,
},
replyStatus: 201,
responseBody: { example: "My mocked response" },
});
// test
Deno.test("fetch", async () => {
const actual = await fetchFromServer();
assertEquals({ example: "My mocked response" }, actual);
});

chart js not dispalying data array that comes from an axios request

I have an API end point that returns an array of 24 values that I want to use in my chartjs within a vue component.
when the page loads I get no errors but the bars on the charts just don't show and I don't know why.
EDIT: I noticed that the async function returns a promise instead of the actual data:
async filterData() {
await this.$axios.get('/api/data_app/job_count_by_hour/')
.then(response => {
return this.chart_data = response.data;
})
}
here is the data return code, I have a function that populates the chart_data array :
data(){
return {
form:{
day: 'select day',
workspace:'',
machine_family: [],
duration: []
},
res: [],
total:[],
chart_data: [],
url: '/api/jobs/job_count_by_hour/',
days: ["Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday", "sunday"],
barChart2: {
labels: ["6h", "7h", "8h", "9h","10h","11h", "12h", "13h", "14h", "15h", "16h", "17h", "18h", "19h", "20h", "21h","22h", "23h", "00h"],
datasets: [{
label: ["popularity"],
backgroundColor:"#f93232" ,
data: this.chart_data
},],
},
}
},
methods: {
async filterData() {
let _url = `${this.url}`
await this.$axios.get(_url)
.then(response => {
this.chart_data = response.data;
})
return this.chart_data
},
},
mounted() {
this.filterData()
}
}
this is the chart component:
<script>
import { Line } from 'vue-chartjs'
export default {
extends: Line,
props: {
chartdata: {
type: Object,
default: null
},
options: {
type: Object,
default: null
}
},
mounted () {
this.renderChart(this.chartdata, this.options)
}
}
in the parent component It looks like this:
en <BarChart :labels="barChart2.labels"
:datasets="barChart2.datasets"
:height="100"
>
</BarChart>ter code here
Turns out that when you try to update nested data, the component doesn't re-render.
This is how I solved it, I put the entire object in an update function and call that function when i get my data from the back end, I hope this helps!:
methods: {
onInput(value) {
this.filterData()
},
updateChart(data) {
this.datasets = [{
label: ["popularity"],
backgroundColor:"#f93232",
data: data
}]
},
async loadData() {
await this.$axios.get(this.url)
.then(response => {
this.updateChart(response.data)
})
},
},
mounted() {
this.loadData()
},

Cross origin resource sharing in AWS lambda proxy integration

Been browsing through SO for the past hours to find a fix for my issue, but no progress yet, I'm getting
No 'Access-Control-Allow-Origin' header is present on the requested resource.
Now typically, this could be fixed with adding appropriate headers to the code, and it would work, however it's not the case for me, since I've tried to configure cors through API Gateway on AWS.
Screenshot:
Some research on google mentioned, that if the function is using lambda proxy integration, we would have to modify the lambda itself, and add the headers by our own, e.g
headers: {
'Access-Control-Allow-Origin': '*',
},
However this doesn't make much difference, is there anything I'm missing?
My actual code for the lambda (forgot to add):
const rp = require('request-promise')
const sendEmail = require('./sendEmail')
module.exports.run = async (event, context, callback) => {
const body = JSON.parse(event.body)
const { name, email, budget, message, attachment } = body
if (!name) {
return callback(null, {
statusCode: 400,
body: JSON.stringify({ message: 'Name is required' }),
})
}
if (!email) {
return callback(null, {
statusCode: 400,
body: JSON.stringify({ message: 'Email address is required' }),
})
}
if (!message) {
return callback(null, {
statusCode: 400,
body: JSON.stringify({ message: 'Message is required' }),
})
}
return Promise.all([
sendEmail({
to: 'Example <user#example.com>',
subject: 'New enquiry received!',
data:
`Name: ${name}\n` +
`Email: ${email}\n` +
`Budget: ${budget || 'n/a'}\n` +
`Attachment: ${attachment || 'n/a'}\n` +
`\n${message}`,
}),
sendEmail({
to: `${name} <${email}>`,
subject: 'Your message was delivered at ',
data:
'Thanks for reaching out!\n' +
'Somebody at our office will get back to you as soon as possible.\n' +
'\n' +
'While you wait, check out our Handbook (/) and get acquainted with how we do things around here.\n' +
'We have a lot of content there so feel free to explore as you please.\n' +
'\n' +
'Speak soon,\n' +
'\n',
}),
rp({
method: 'POST',
uri: `https://hooks.slack.com/services/${process.env.SLACK_PATH}`,
json: true,
body: {
text: `<!channel> New enquiry received`,
attachments: [
{
fallback: 'Information:',
pretext: 'Information:',
color: '#FF5050',
fields: [
{ title: 'Name', value: name, short: false },
{ title: 'Email', value: email, short: false },
{ title: 'Budget', value: budget || 'n/a', short: false },
{ title: 'Attachment', value: attachment || 'n/a', short: false },
{ title: 'Message', value: message || 'n/a', short: false },
],
},
],
},
}),
])
.then(() => {
return callback(null, {
statusCode: 200,
headers: {
'Access-Control-Allow-Origin': '*',
},
body: JSON.stringify({ message: 'Great success' }),
})
})
.catch(err => {
return callback(null, {
statusCode: 500,
body: JSON.stringify({
message: 'Oh no :( Message not delivered',
error: err
}),
})
})
}
No 'Access-Control-Allow-Origin' header is present on the requested resource.
This is saying that the resource you requested, your Lambda via API Gateway, is not returning an Access-Control-Allow-Origin header in its response; the browser is expecting the CORS headers in the response from the API (possibly because of an OPTIONS request), but the response doesn’t have them.
To solve your issue, add a Access-Control-Allow-Origin: * header to the response your Lambda returns. Using the first item you're returning:
if (!name) {
return callback(null, {
statusCode: 400,
headers: {
'Access-Control-Allow-Origin': '*',
// any other required headers
},
body: JSON.stringify({ message: 'Name is required' }),
})
}
Worth noting that you'll have to add those headers to every response.

Updating location data into the server is not working using background geo locaion in ionic 3

I am using https://github.com/mauron85/cordova-plugin-background-geolocation. It gives location details in the foreground and gives debug messages when the app is closed but it does not update the location details into the server both foreground and background.
Thanks in advance
const config: BackgroundGeolocationConfig = {
desiredAccuracy: 10,
stationaryRadius: 10,
distanceFilter: 10,
debug: false,
stopOnTerminate: false,
startForeground: true,
notificationTitle: 'location tracking',
notificationText: 'Active',
interval: 60000,
url: localStorage.getItem('api_base_url')+'user/currentlocation',
syncUrl:localStorage.getItem('api_base_url')+'user/currentlocation',
httpHeaders: {
'Content-Type': 'application/json'
},
postTemplate: {
lat: '#latitude',
lon: '#longitude',
user_id: '1',
currentDate: '12-12-2019',
address: 'test',
}
};
this.backgroundGeolocation.configure(config)
.then(() => {
this.backgroundGeolocation.on(BackgroundGeolocationEvents.location).subscribe((location: BackgroundGeolocationResponse) => {
console.log(location);
});
});
this.backgroundGeolocation.start();
Background geo location post template
Keep in mind that all locations (even a single one) will be sent as an array of object(s), when postTemplate is jsonObject and array of array(s) for jsonArray!
In server-side, I changed the JSON object to an array of object.
For example,
{
"user_id": "1",
"lon": "13.2",
"lat": "82.3"
}
I changed the above JSON object to following
[{
"user_id": "1",
"lon": "13.2",
"lat": "82.3"
}]