We are seeing a random error that seems to be caused by two requests' data getting mixed up. We receive a request for quoting shipping costs on an Order, but the request fails because the requested Order is not accessible by the requesting account. I'm looking for anyone who can provide an inkling on what might be happening here, I haven't found anything on google, the official flask help channels, or SO that looks like what we're experiencing.
We're deployed on AWS, with apache, mod_wsgi, 1 process, 15 threads, about 10 instances.
Here's the code that sends the email:
msg = f"Order ID {self.shipping.order.id} is not valid for this Account {self.user.account_id}"
body = f"Error:<br/>{msg}<br/>Request Data:<br/>{request.data}<br/>Headers:<br/>{request.headers}"
send_email(msg, body, "devops#*******.com")
request_data = None
The problem is that in that scenario we email ourselves with the error and the request data, and the request data we're getting, in many cases, would've never landed in that particular piece of code. It can be a request from the frontend to get the current user's settings, for example, that make no reference to any orders, nevermind trying to get a shipping quote for it.
Comparing the application logs with apache's access_log, we see that, in all cases, we got two requests on the same instance, one requesting the quoting, and another which is the request that is actually getting logged. We don't know whether these two requests are processed by the same thread in rapid succession, or by different threads, but they come so close together that I think the latter is much more probable. We have no way of univocally tying the access_log entries with the application logging, so far, so we don't know which one of the requests is logging the error, but the fact is that we're getting routed to a view that does not correspond to the request's content (i.e., we're not sure whether the quoting request is getting the wrong request object, or if the other one is getting routed to the wrong view).
Another fact that is of interest is that we use graphql, so part of the routing is done after flask/werkzeug do theirs, but the body we get from flask.request at the moment the error shows up does not correspond with the graphql function/mutation that gets executed. But this also happens in views mapped directly through flask. The user is looked up by the flask-login workflow at the very beginning, and it corresponds to the "bad" request (i.e., the one not for quoting).
The actual issue was a bug on one of python-graphql's libraries (promise), not on Flask, werkzeug or apache. It was not the request data that was "moving" to a different thread, but a different thread trying to resolve the promise for a query that was supposed to be handled elsewhere.
Related
Following is the screenshot of the server activity log.I can see that many requests are automatically raised in the server.How can I avoid this.?
It looks like someone is fuzzing your website and scanning to find any common file names or extensions that commonly have security vulnerabilities. One way to limit this behaviour is to implement rate limiting whereby you might limit the number of requests a user makes that result in HTTP 404 Not Found during some time period before giving them a temporary ban. Note: this solution doesn't stop this from happening but it does buy you time and may deter the attacker or researcher
I working on a service which scrapes specific links from blogs. The service makes calls to different sites which pulls in and stores the data.
I'm having troubles specifying the url for updating the data on the server where I now use the verb update to pull in the latest links.
I currently use the following endpoints:
GET /user/{ID}/links - gets all previously scraped links (few milliseconds)
GET /user/{ID}/links/update - starts scraping and returned the scraped data (few seconds)
What would be a good option for the second url? some examples I came up with myself.
GET /user/{ID}/links?collection=(all|cached|latest)
GET /user/{ID}/links?update=1
GET /user/{ID}/links/latest
GET /user/{ID}/links/new
Using GET to start a process isn't very RESTful. You aren't really GETting information, you're asking the server to process information. You probably want to POST against /user/{ID]/links (a quick Google for PUT vs POST will give you endless reading if you're curious about the finer points there). You'd then have two options:
POST with background process: If using a background process (or queue) you can return a 202 Accepted, indicating that the service has accepted the request and is about to do something. 202 generally indicates that the client shouldn't wait around, which makes sense when performing time dependent actions like scraping. The client can then issue GET requests on the first link to retrieve updates.
Creative use of Last-Modified headers can tell the client when new updates are available. If you want to be super fancy, you can implement HEAD /user/{ID}/links that will return a Last-Modified header without a response body (saving both bandwidth and processing).
POST with direct processing: If you're doing the processing during the request (not a great plan in the grand scheme of things), you can return a 200 OK with a response body containing the updated links.
Subsequent GETs would perform as normal.
More info here
And here
And here
I've got a django site that is producing a csv download. The content of the csv is dictated by user defined parameters. It's possible that users will set parameters that require significant thinking time on the server. I need a way of sustaining the http connection so the browser doesn't kick up an error message. I heard that it's possible to send intermittent http headers to do this. Can anyone point me in the right direction to set this up on a django site?
(unfortunatly I'm stuck with the possibility of slow reports - improving my sql won't mitigate this)
Don't do it online. Trigger an offline task, use a bit of Javascript to repeatedly call a view that checks if the task has finished, and redirect to the finished file when it's ready.
Instead of blocking the user and it's browser for 20 minutes (which is not a good idea) do the time-consuming task in the background. When the task will finish and generate the result simply notify the user so that he/she will just need to download the ready result.
I'm creating a web app for handling various surveys. An admin can create his own survey and ask users to fill it up. Users are defined by target groups assigned to the survey (so only user in survey's target group can fill the survey).
One of methods to define a target group is a "Token target group". An admin can decide to generate e.g. 25 tokens. After that, the survey can be accessed by anyone who uses a special link (containing the token of course).
So now to the main question:
Every token might have an e-mail address associated with itself. How can I safely send e-mails containing the access link for the survey? I might need to send a few thousand e-mails (max. 10 000 I believe). This is an extreme example and such huge mailings would be needed only occasionally.
But I also would like to be able to keep track of the e-mail message status (was it send or was there any error?). I would also like to make sure that the SMTP server doesn't block this mailing. It would also be nice if the application remained responsive :) (The task should run in background).
What is the best way to handle that problem?
As far as I'm concerned, the standard Django mailing feature won't be much help here. People report that setting up a connection and looping through messages calling send() on them takes forever. It wouldn't run "in background", so I believe that this could have negative impact on the application responsiveness, right?
I read about django-mailer, but as far as I understood the docs - it doesn't allow to keep track of the message status. Or does it?
What are my other options?
Not sure about the rest, but regardless for backgrounding the task (no matter how you eventually do it) you'll want to look for Celery
The key here is to reuse connection and to not open it again for each email. Here is a documentation on the subject.
I have a legacy VB6 application that needs to make asynchronous calls to a web service. The web service provides a search method allows end-users to query a central database and view the results from within the application. I'm using the MSXML2.XMLHTTP to make the requests, and have written a SearchWebService class that encapsulates the web service call and code to handle the response asychronously.
Currently, the SearchWebService raises one of two events to the caller: SearchCompleted and SearchFailed. A SearchCompleted event is raised that contains the search results in a parameter to the event if the call completes successfully. A SearchFailed is raised when any type of failure is detected, which can be anything from an improperly-formatted URL (this is possible because the URL is user-configurable), to low-level network errors such as "Host not found", to HTTP errors such as internal server errors. It returns a error message string to the end-user (which is extracted from the web service response body, if present, or from the HTTP status code text if the response has no body, or translated from the network error code if a network error occurs).
Because of various security requirements, the calling application does not access the web service directly, but instead accesses it through a proxy web server running at the customer site, which in turn accesses the actual web service through via a VPN. However, the SearchWebService doesn't know that the calling application is accessing the web service through a proxy: it's just given a URL and told to make the request. The existence of the proxy is a application-level requirement.
The problem is that from an end-user perspective, it's important that the calling application be able to distinguish between low-level network errors versus HTTP errors from the web service, and to distinguish proxy errors from remote web server errors. For example, the application needs to know if a request failed because the proxy server is down, or because the remote web service that the proxy is accessing is down. An application-specific message needs to be presented to the end-user in each case, such as "Search web service proxy server appears to be down. The proxy server may need to be restarted" versus "The proxy is currently running but the remote web server appears to be unavailable. Please contact (name of person in charge of the remote web server)." I could handle this directly in the SearchWebService class, but it seems wrong to generate these application-specific error messages from such a generic class (and the class might be used in environments that don't require a proxy, where the error messages would no longer make sense).
This distinction is important for troubleshooting: a proxy server problem can usually be resolved by the customer, but a remote web server error has to handled by a third party.
I was thinking one way to handle this would be to have the SearchWebService class detect different types of errors and raise different events in each case. For example, instead of a single SearchFailed event, I could have a NetworkError event for low-level network errors (which would indicate a problem accessing the proxy server), a ConfigurationError event for invalid properties on the SearchWebService class (such as passing an improperly-formatted URL), and a ServiceError for errors that occur on the remote web server (implying that the proxy is working properly but the remote server returned an error).
Now that I think about it, there is also an additional error scenario: it could be possible that the proxy server is running properly, but the remote web server is down, or the proxy server has been misconfigured.
Is the approach of using multiple error events to classify different classes of error a reasonable solution to this problem? For the last scenario (the proxy is running but the remote server cannot be reached), I'm guessing I may have to set up the proxy to return a specific HTTP error code so that client can detect this situation (i.e. something more specific than a 500 response).
Originally I kept the single SearchFailed event and simply added an additional errorCode parameter to the event, but that got messy quickly, especially in cases where there wasn't a logical error code to use (such as if the VB6 raises a "real" error, i.e. if the XMLHTTP class isn't registered).
I think that some ideas I've used with Java exceptions may apply here.
Having a large number of different Exceptions gets pretty messy, yet we need to give enough detail to the user so we don't want to lose information.
Hence I have a small number of specific Exceptions, which I guess would correspond to your Events:
InvalidRequestEvent: Used when the user specifies bad information
TransientErrorEvent: used when there's infrastructure issues when a retry might work.
I tend to work in environments where we have clusters of servers so if a user request hits a dying server then if he resubmits he'll probably get a good one, hence from his perspective a simple retry often works. However sometimes the error is with a service such as the Network or Database and in which case the user needs diagnostic information to report to the helpdesk. Hence we need to decide on the extra information to put into the exception. This is (if I understand you correctly) your question.
In the case of InvalidRequestException we would bet giving some information about the problems with the input. It could be on the lines of "Mismatched parenthese" or "Unknown column CUTSOMER in table ORDER". In the case of TransientErrorException it could be "Proxy server is down".
Now depending upon your exact requirments you may not actually choose to put that text in the Exception, but rather an error number which the presentation layer converts to a locale-specific string (English, French ...).
So either Exception might contain something like this (sorry for that Java syntax, but I hope the idea is clear):
BaseException {
String ErrorText; // the error text itself
// OR if you want to allow for internationaliation
int ErrorCode; // my application specific code, corresponds to text held by the UI
String[] params; // specific parameters to be substitued in the error text
// CUTSOMER and ORDER in my example above
int SystemErrorCode; // If you have an underlying error code it goes here
String SystemErrorText; // any further diagnoistic you might need to give to
// the user so that they can report the problem to the
// help desk.
// OR instead of the text (this is something I've seen done)
int SystemErrorTag; // A unique id for this particular error problem.
// This server systems will label their message in the
// server logs. Users just tell the help desk this number
// they don't need to read detailed server error text.
}