Do we know what the warmup/cold-start time is for a function on the new (still beta as of now) Google Cloud Functions? Or the timeout before it cools down again?
I've been trying Azure functions in the per-use mode, and it's been ridiculously bad - seemed inconsistent but I've seen 30 seconds, and it seems to be about 5 minutes of no-use before it cools down again.
I'm assuming Google's functions have the same issue, but I don't see even preliminary documentation on those time periods. The answer seems to be a "ping" every X minutes to keep it alive, but knowing what that X should be would make a difference in billing.
There is an interesting blog post by Mikhail Shilkov, published in August 2018. At this time and based on his experiments, the minimum alive period observed for Google Cloud Functions (GCF) is 3 minutes:
As a consequence, using X=2 should be enough to keep a single function warm and ready to serve a request. However, in that case, you should ask yourself whether it is still relevant to use a serverless solution such as GCF.
Related
I am interested to know what the best solution would be to the following problem:
I have telemetry data published to pubsub at 10 second intervals.
The data changes in cycles (ie. One particular parameter goes from 400 to 1700 and resets back to 400 ...)
I would like to trigger a Cloud Function or Cloud Run API when the parameter is at 800 and 1200 and 1500.
So I was wondering if there is such a thing whereby we can trigger a cloud function/cloud run if the pubsub message says that the parameter is at those values?
One way to do it of course is to write a Cloud Function to always activate when a new message is published and use an "if" condition to determine whether to proceed. However, I'm thinking this is a very expensive method because the cloud function would be triggered every 10 seconds when these cycles are particularly long and don't need to be executed frequently.
So I'm wondering if there is a more cost effective method.
The naive solution is fine even if not "elegant". We yet have to define what "very expensive" is. A single function up is not an important point of spending. Especially if you consider the free tier usage.
That being said, since June PubSub beta released a filtering feature that allows you to put delivery conditions based on your message attributes. It may helps you. I say it may bc as far as i can tell, there is no comparaison operator on the payload content. Only attributes. So you would have to forge messages with attributes that allow you to deliver or not according your business rule.
There is an Elastic Container Service cluster running an application internally referred to as Deltaload. It checks the data in Oracle production database and in dev database in Amazon RDS and loads whatever is missing into RDS. A CloudWatch rule is set up to trigger this process every hour.
Now, for some reason, every 20-30 hours there is one interval of a different length. Normally, it is a ~25 min gap, but on other occasions it can be 80-90 min instead of 60. I could understand a difference of 1-2 minutes, but being off by 30 min from an hourly schedule sounds really problematic, especially given the full run takes ~45 min. Does anyone have any ideas on what could be the reason for that? Or at least how can I figure out why it is so?
The interesting part is that this glitch in schedule either breaks or fixes the Deltaload app. What I mean is, if it is running successfully every hour for a whole day and then the 20 min interval happens, it will then be crashing every hour for the next day until the next glitch arrives, after which it will work again (the very same process, same container, same everything). It crashes, because the connection to RDS times out. This 'day of crashes, day of runs' thing has been going on since early February. I am not too proficient with AWS. This Deltaload app is written in C#, which I don't know. The only thing I managed to do is to increase the RDS connection timeout to 10 min, which did not fix the problem. The guy that wrote the app has left the company a time ago and is unavailable. There are no other developers on this project, as everyone got fired, because of corona. So far, the best alternative I see, is to just rewrite the whole thing in Python (which I know). If anyone has any other thoughts on how understand/fix it, I'd greatly appreciate any input.
To restate my actual question: why is it that CloudWatch rule drops in irregular intervals in a regular schedule? How to prevent this from happening?
Hello Dear Informatica admin/platform experts,
I have a workflow that i need to scheduled say Monday-Friday and Sunday. All the 6 days the job should at a specific time say 10 times in a day, but the timing is not uniform but at a predefined time say(9 AM, 11 AM, 1:30 PM etc), so the difference in the timing is not uniform. so we had 10 different scheduling workflows for each schedule/run that triggers a shell script that uses pmcmd command.
It looked a bit weird for me, so what i did was, have a single workflow that triggers the pmcmd shell script, and have a link between the start and the shell script where i specified a condition of the time and scheduled it to run monday-friday and sunday every 30 minutes.
So what happens is, it runs 48 times in a day but actually triggers the "actual" workflow only 10 times. and the remaining 38 times it just runs but does nothing.
one of my informatica admin colleague says that running this 38 times(which does actually nothing) consumes informatica resources. Though i was quite sure it does not, but as i am just an informatica developer and not an expert, thought of posting it here, to check whether it is really true?
Thanks.
Regards
Raghav
Well... it does consume some resources. Each time workflow starts, it does quite a few operations on the Repository. It also allocates some memory on Integration Service as well as creates log file for the Workflow. Even if there are no sessions executed at all.
So, there is an impact. Multiply that by the number of workflows, times the number of executions - and there might be a problem.
Not to mention there are some limitations regarding the number of Workflow being executed at the same time.
I don't know your platform and setup. But this look like a field for improvement indeed. A cron scheduler should help you a lot.
There are a few pieces of my app that cannot afford the additional 1-2 second delay caused by the "freeze-thaw" cycle that Lambda functions go through when they're new or unused for some period of time.
How can I keep these Lambda functions warm so AWS doesn't have to re-provision them all the time? This goes for both 1) infrequently-used functions and 2) recently-deployed functions.
Ideally, there is a setting that I missed called "keep warm" that increases the cost of the Lambda functions, but always keeps a function warm and ready to respond, but I'm pretty sure this does not exist.
I suppose an option is to use a CloudWatch timer to ping the functions every so often... but this feels wrong to me. Also, I don't know the interval that AWS uses to put Lambda functions on ice.
UPDATE DEC 2019
AWS now also offers 'Provisioned Concurrency'. https://aws.amazon.com/blogs/aws/new-provisioned-concurrency-for-lambda-functions/
Basically you pay around 10$/month (for a 1GB Lambda) per instance that you want to keep 'warm'.
BBC has published a nice article on iPlayer engineering where they describe a similar issue.
They have chosen to call the function every few minutes using CloudWatch Scheduled Events.
So in theory, it should just stay there, except it might not. So we have set up a scheduled event to keep the container ‘warm’. We invoke the function every couple of minutes not to do any processing, but to make sure we’ve got the model ready. This accounts for a very small percentage of invocations but helps us mitigate race conditions in the model download.(We also limit artificially how many lambdas we invoke in parallel as an additional measure).
I am planning to get an app developed but the developer has told me that there is a limit of 600 calls per 600 seconds per IP. The app has plenty of scenarios in which this would not suffice. Is there a way to increase the limit somehow? Or does Facebook offer any premium account or something probably with a yearly fee that does not have such a limit?
Thanks.
If you exceed, or plan to exceed, any of the following thresholds please contact us as you may be subject to additional terms: (>5M MAU) or (>100M API calls per day) or (>50M impressions per day).
Pulled from : https://developers.facebook.com/policy/
100M API Queries per day should be for a single app. So that should restrict you, but I dont think that matters.
Another thing, what you mentioned in your question, and I have read that elsewhere as well.
I've found 600 calls per 600 seconds, per token & per IP to be about where they stop you.
Pulled from : http://www.quora.com/Whats-the-Facebook-Open-Graph-API-rate-limit
Note, it is per token. Every other user has a different access token and IP as well. If it happens to be a cron running from the server, still I dont think they would catch you for the IP as long as you keep changing the tokens.
Another thing to implement is the Real time updates API, which will ping you when something changes so you dont have to run a 24*7 monitoring script.
P.S : Real Time Updates is Buggy! Have experienced it myself.