Looking at implementing Elmah but I cant see anyway to limit the number of errors logged if they rise above a certain threshold?
Basically thinking about a scenario where some automated process (unintended, or malicious) generates a ton of errors in a short period of time.
We will be logging to a db (probably MySQL), rather than sending emails, but couldnt see any info on anyway to set any max limits - ideally I'd like to say x thousand in 5 min interval rather than just rely on crashing my error db connection.
The server we log errors to will probably also run a few other servioces / functions we wont want to lose should a ton of errors get generated
Is this something others have considered / solved, or ius the threshold effectively so high it is irrelevant?
I imagine it would be vital if you leave the error reporting on?
These are the closest things I could find...
And here is an example of doing it via subclass of the ELMAH logger:
Related
We are running simple GCP Functions (pure, no Firebase, or any other layer added) that just handle HTTP requests using Node.js engine (previously version 8, now 10) and return some "simple JSON response". What we see is that sometimes (but not rarely) there is a huge latency when the request is "accepted by GCP" and before it gets to our function code. If I say huge I'm not speaking ms but units of seconds! And it is not a cold start (we have separate log messages on the global scope so we know when cold start occurs). Functions have currently 256 or 512 mb and run in close region.
We log at the very first line of the GCP function, for example:
or
Does anyone also experience that? And is that normal that sometimes this delay may take up to 5s (or rarely even more)?
By the way, sometimes the same thing happens on the output side as well. So if unlucky, it may take up to 10s. Thanks in advance for any reply, no matter if you have or have not similar experience.
All such problems I have seen have been related with cold start or it was not possible to prove that they are not related with code start.
This question could be even to broad to stackoverflow. We do not have any chance to reproduce it without example at least functions and number of the executions, however I will try to answer.
It seems that latency analyzes are done mainly on logs. I think you should try to use "Trace" functionality that is available in GCP (direct link and documentation). This should give you data to be able to track the issue.
Example i have used it on helloworld cloud function and was curl'ing it from bush script. It seems that over few hundreds of invocations there was one execution with latency 10 times greater than usually.
I hope it will help somehow :)!
My project is to download extremly big number of ID-s from twitter.
Also known as, that the average user have small number of followers(100-200).
I use for this streaming the Twython package, and here is the main part of my program:
while(next_cursor):
follower_id=twitter.get_followers_ids(user_id=ids,cursor=next_cursor)
time.sleep(60)
next_cursor=follower_id['next_cursor']
This is a really simple cod, and works also, but really slow, for big number of ID-s, becouse the function tw.get_follower_id()-s rate limit is 5000 id/minute, thats why the time sleep function is in the code.
My question, is there any possibilites of speed up this code?
Perhaps so that the program does not pause after each query, only when it really need.
Could somebody help with this?
Twitter provide rate-limit info in the headers sent with every API response. SO you could check that, and hence call at the maximum rate allowed. You can also request your rate-limit status from Twitter via a specific rate-limit API call, and it doesn't reduce the rate-limit to check. I don't use Twython myself, so I can't advise on how to do so within Twython.
It won't gain you much extra -- maybe a few %.
Alternatively, it doesn't hurt to bump into the rate-limit occasionally -- you'll get an error message. As long as it isn't too frequent, Twitter won't mind.
The basic rate-limit speed cap -- no way round that. Perhaps Gnip have a paid service that will let you download this data faster?
I'm aware that a web application should render it's pages as fast as possible, with few database requests only in milliseconds. What are the guidelines about this response time (like Microsoft guidelines for UI or something like that)?
What is the absolute maximum time a webpage should respond under?
Are there any "limits" or general guidelines for this?
When should I put jobs into task queues (like Python celery for example)?
My concrete problem is that I have to parse a bunch of text files, which users submits. The average time these can be parsed are 2-3 seconds (response times are 3-4sec with database inserts) but if the file is very big, it takes 8sec to parse (10sec to respond).
Is it okay to let the user without feedback for this time? If not, what is the best way for handling these kinds of situations?
Is it even okay to put these in the request-response cycle ?
Is there any difference if I provide a REST API vs a website form ? Are these "allowed" slower to respond?
I think this is really hard to answer. Different guidelines exist.
When I was at university during interface / interaction design courses I learned that no user should be left with response times over 50 ms.
If that is exceeded, something like a loading icon etc should be displayed.
Also users are educated enough to expect right loading times form websites... So the user will accept 2 seconds loading time for a ticket booking page but not accept more than 300 ms from a search engine.
The limits I hear about during this days are 0.1 sec, 1 sec and 10 sec.
0.1 feels instantly to the user on websites
1 sec is slow but no interruption
10 sec is the maximum for the user to endure before loosing attention (for example light a smoke, check facebook feed in the meantime, etc...)
There is a nice article along with a lot of useful comments which I lately read which I would like to point you to:
http://www.nngroup.com/articles/response-times-3-important-limits/
I think it answers your questions well.
Please understand that this is all purely subjective but I think this is a very subjective topic...
We would like to be able to create intermediate releases of our software that would time-bomb or expire after a certain fixed time or number of uses that would not easily be manipulated. We are using Visual C++ with mixed native and managed assemblies.
I imagine we may need to rely on a registry tag but this seems to be insecure.
Can anyone offer some advice on how to do this?
I was working on a "trial-ware" solution a while back and it used a combination of registry keys, information stored in a flat-file at a certain position surrounded with junk data, and then also had an option to reach out to a webservice that would verify it back with the software creators.
However, as FrustratedWithFormsDesigner stated, there is no 100% fool-proof way to do this. There is always a way that a hacker can get around whatever precautions you put in place.
If you are using a database for the application, then it might be better to store a install (datetime) and a numberofusers (int) and then make code that checks those fields when the program is starting / loading / initing. If they are past a certain number or time (this could also be in the db) then exit the program.
This is very hard if not impossible to do in a foolproof way. In any event, there's nothing to stop somebody removing and reinstalling the software (you do support that, right?).
If you cannot limit the function of these intermediate releases (a much better incentive for people to move to official bits), it might be more trouble than it's worth to implement such a scheme.
Set a variable to a specific date in the program then every time the program is run access the system date and check if that date is equal to or greater than the specified date. If true then start the expiry process and display a message or alert panel to the user.
Have the binary download a tiny bit of code on startup from one of your servers.
Keep track of the activation counter on the server, when the counter reaches the limit, return a piece of code that displays the 'sorry!' message.
You could deploy it as a ClickOnce application with a certificate that expires at a certain date. If I recall correctly, the app will err on startup after that date.
A couple caveats:
The only option for the user may be to uninstall the app, which is a jerk move.
You will end up maintaining a ton of different deployments.
It will be a shock to the user as it will just happen without warning.
I currently have a django site, and it's kind of slow, so I want to understand what's going on. How can I profile it so to differentiate between:
effect of the network
effect of the hosting I'm using
effect of the javascript
effect of the server side execution (python code) and sql access.
any other effect I am not considering due to the massive headache I happen to have tonight.
Of course, for some of them I can use firebug, but some effects are correlated (e.g. javascript could appear slow because it's doing slow network access)
Thanks
client side:
check with firebug if/which page components take long to load, and how long the browser needs to render the page after loading is completed. If everything is fast but rendering takes its time, then probably your html/css/js is the problem, otherwise it's server side.
server side (i assume you sit on some unix-alike server):
check the web server with a small static content (a small gif or a little html page), using apache bench (ab, part of the apache webserver package) or httperf, the server should be able to answerat least 100 requests per second (of course this depends heavily on the size of your test content, webserver type, hardware and other stuff, so dont take that 100 to seriously). if that looks good,
test django with ab or httperf on a "static view" (one that doesnt use a database object), if thats slow it's a hint that you need more cpu power. check cpu utilization on the server with top. if thats ok, the problem might be in the way the web server executes the python code
if serving semi-static content is ok, your problem might be the database or IO-bound. Database problems are a wide field, here is some general advice:
check i/o throughput with iostat. if you see lot's of writes then you have get a better disc subsystem, faster raid, SSD hard drives .. or optimize your application to write less.
if its lots of reads, the host might not have enough ram dedicated as file system buffer, or your database queries might not be optimized
if i/o looks ok, then the database might be not be suited for your workload or not correctly configured. logging slow queries and monitoring database activity, locks etc might give you some idea
if you let us know what hardware/software you use i might be able to give more detailed advice
edit/PS: forgot one thing: of course your app might have a bad design and does lots of unnecessary/inefficient things ...
Take a look at the Django debug toolbar - that'll help you with the server side code (e.g. what database queries ran and how long they took); and is generally a great resource for Django development.
The other non-Django specific bits you could profile with yslow.
There are various tools, but problems like this are not hard to find because they are big.
You have a problem, and when you remove it, you will experience a speedup. Suppose that speedup is some factor, like 2x. That means the program is spending 50% of its time waiting for the slow part. What I do is just stop it a few times and see what it's waiting for. In this case, I would see the problem 50% of the times I stop it.
First I would do this on the client side. If I see that the 50% is spent waiting for the server, then I would try stopping it on the server side. Then if I see it is waiting for SQL queries, I could look at those.
What I'm almost certain to find out is that more work is being requested than is actually needed. It is not usually something esoteric like a "hotspot" or an "algorithm". It is usually something dumb, like doing multiple queries when one would have been sufficient, so as to avoid having to write the code to save the result from the first query.
Here's an example.
First things first; make sure you know which pages are slow. You might be surprised. I recommend django_dumpslow.