We've got a Django-based web application that is used for receiving POST data from iOS devices (push notification tokens).
All in all, the application seems to be working fine, and we're receiving a 1000-2000 POSTs with valid data every hour. However, I'm occasionally receiving error logs from Django with the following data:
Traceback (most recent call last):
File "/opt/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 111, in get_response
response = callback(request, *callback_args, **callback_kwargs)
File "/opt/local/lib/python2.7/site-packages/django/views/decorators/vary.py", line 19, in inner_func
response = func(*args, **kwargs)
File "/opt/local/lib/python2.7/site-packages/django_piston-0.2.3-py2.7.egg/piston/resource.py", line 160, in __call__
request.data = request.POST
File "/opt/local/lib/python2.7/site-packages/django/core/handlers/wsgi.py", line 180, in _get_post
self._load_post_and_files()
File "/opt/local/lib/python2.7/site-packages/django/http/__init__.py", line 372, in _load_post_and_files
self._post, self._files = QueryDict(self.body, encoding=self._encoding), MultiValueDict()
File "/opt/local/lib/python2.7/site-packages/django/http/__init__.py", line 328, in body
self._body = self.read()
File "/opt/local/lib/python2.7/site-packages/django/http/__init__.py", line 384, in read
return self._stream.read(*args, **kwargs)
File "/opt/local/lib/python2.7/site-packages/django/core/handlers/wsgi.py", line 98, in read
result = self.buffer + self._read_limited()
File "/opt/local/lib/python2.7/site-packages/django/core/handlers/wsgi.py", line 92, in _read_limited
result = self.stream.read(size)
UnreadablePostError: request data read error
And the WSGIRequest dump says POST: <could not parse>
I've been trying to find more information on this error, and a lot of what I'm seeing points to this error being caused by a user canceling a POST request before the post completes. Is this an error that I should be concerned about, or should I just set up the server to filter out these error messages? I'd say that I get maybe 8-10 automated emails per day about this.
I'm not sure whether you're still waiting for the answer but basically the comments contain most of the information. So, just to close the question up...
These errors mean that a malformed request arrived to the server. Someone might have cancelled their request or it got mangled on its way (e.g. bad internet connection), etc.
You can't solve these errors directly. However you can take a look at the page (if it's always the same one) and check if for example the loading doesn't take too long. You can also try and cause the error manually to see when exactly it happens and why. Unless you find something relevant, I don't think you need to worry about it much.
Related
I have a Google Cloud Function in Python 3.7 reading from a Pub/Sub subscription in synchronous pull mode.
After running fine 1/hour for 24 hours, it threw this exception stack trace:
Traceback (most recent call last): File
"/env/local/lib/python3.7/site-packages/google/api_core/grpc_helpers.py",
line 57, in error_remapped_callable
return callable_(*args, **kwargs) File "/env/local/lib/python3.7/site-packages/grpc/_channel.py", line 824,
in call
return _end_unary_response_blocking(state, call, False, None) File "/env/local/lib/python3.7/site-packages/grpc/_channel.py", line
726, in _end_unary_response_blocking
raise _InactiveRpcError(state) grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status =
StatusCode.DEADLINE_EXCEEDED details = "Deadline Exceeded"
debug_error_string =
"{"created":"#1580454091.145703535","description":"Error received from
peer
ipv4:74.125.202.95:443","file":"src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Deadline
Exceeded","grpc_status":4}"
The above exception was the direct cause of the following exception:
Traceback (most recent call last): File
"/env/local/lib/python3.7/site-packages/google/cloud/functions/worker.py",
line 346, in run_http_function
result = _function_handler.invoke_user_function(flask.request) File
"/env/local/lib/python3.7/site-packages/google/cloud/functions/worker.py",
line 217, in invoke_user_function
return call_user_function(request_or_event) File "/env/local/lib/python3.7/site-packages/google/cloud/functions/worker.py",
line 210, in call_user_function
return self._user_function(request_or_event) File "/user_code/main.py", line 39, in iteration
response = sub.pull(sub_path, MAX_MESSAGES) File "/env/local/lib/python3.7/site-packages/google/cloud/pubsub_v1/_gapic.py",
line 40, in
fx = lambda self, *a, **kw: wrapped_fx(self.api, *a, **kw) # noqa File
"/env/local/lib/python3.7/site-packages/google/cloud/pubsub_v1/gapic/subscriber_client.py",
line 1005, in pull
request, retry=retry, timeout=timeout, metadata=metadata File "/env/local/lib/python3.7/site-packages/google/api_core/gapic_v1/method.py",
line 143, in call
return wrapped_func(*args, **kwargs) File "/env/local/lib/python3.7/site-packages/google/api_core/retry.py",
line 286, in retry_wrapped_func
on_error=on_error, File "/env/local/lib/python3.7/site-packages/google/api_core/retry.py",
line 184, in retry_target
return target() File "/env/local/lib/python3.7/site-packages/google/api_core/timeout.py",
line 214, in func_with_timeout
return func(*args, **kwargs) File "/env/local/lib/python3.7/site-packages/google/api_core/grpc_helpers.py",
line 59, in error_remapped_callable
six.raise_from(exceptions.from_grpc_error(exc), exc) File "", line 3, in raise_from
google.api_core.exceptions.DeadlineExceeded: 504 Deadline Exceeded
What is this about? Is it to be expected or a result of some configuration problem? If to be expected, how should it be handled?
The documentation ( view-source:https://googleapis.dev/python/pubsub/latest/subscriber/api/client.html ) on pull has nothing about this being a possible exception.
I ack the messages immediately after the pull completes. I only permit one function execution at a time. I have a 600 second acknowledgement deadline. A block of messages pulled at one time seem to be less than 100 in number. If this is about failing to ack a message, it seems like the error could be done much better.
This exception is raised by the client when there's no messages to read in the subscription. It is a known issue from the latest PubSub library versions >= 1.0.0. If necessary, you can downgrade to the version 0.45.0 where this issue was not present.
However, as a workaround you can catch the DeadlineExceeded exception and retry the operation again. Also, based on the comment of Hemang, here's a small monkeypatch that you can add to your running code, which might help to get the same behavior as in version 0.45.0.
from google.cloud.pubsub_v1.gapic import subscriber_client_config as sub_config
sub_config.config['interfaces']['google.pubsub.v1.Subscriber']['retry_params']['messaging']['initial_rpc_timeout_millis'] = 25000
Finally, keep in mind that when using synchronous pull, having many outstanding pull requests helps lower the delivery latency, which in turn might result in higher latency pull requests (and DeadlineExceeded errors). Although, if latency is crucial for the application, you could consider using StreamingPull
I have been trying to get the Elasticsearch to work in a Django application. It has been a problem because of the mess of compatibility considerations this apparently involves. I follow the recommendations, but still get an error when I actually perform a search.
Here is what I have
Django==2.1.7
Django-Haystack==2.5.1
Elasticsearch(django)==1.7.0
Elasticsearch(Linux app)==5.0.1
There is also DjangoCMS==3.7 and aldryn-search=1.0.1, but I am not sure how relevant those are.
Here is the error I get when I submit a search query via the basic text form.
GET /videos/modelresult/_search?_source=true [status:400 request:0.001s]
Failed to query Elasticsearch using '(video)': TransportError(400, 'parsing_exception')
Traceback (most recent call last):
File "/home/user-name/miniconda3/envs/project-web/lib/python3.7/site-packages/haystack/backends/elasticsearch_backend.py", line 524, in search
_source=True)
File "/home/user-name/miniconda3/envs/project-web/lib/python3.7/site-packages/elasticsearch/client/utils.py", line 69, in _wrapped
return func(*args, params=params, **kwargs)
File "/home/user-name/miniconda3/envs/project-web/lib/python3.7/site-packages/elasticsearch/client/__init__.py", line 527, in search
doc_type, '_search'), params=params, body=body)
File "/home/user-name/miniconda3/envs/project-web/lib/python3.7/site-packages/elasticsearch/transport.py", line 307, in perform_request
status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
File "/home/user-name/miniconda3/envs/project-web/lib/python3.7/site-packages/elasticsearch/connection/http_urllib3.py", line 93, in perform_request
self._raise_error(response.status, raw_data)
File "/home/user-name/miniconda3/envs/project-web/lib/python3.7/site-packages/elasticsearch/connection/base.py", line 105, in _raise_error
raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
elasticsearch.exceptions.RequestError: TransportError(400, 'parsing_exception')
Could someone tell me if this is an issue with compatibility or is there something else going on? How can I fix it?
The combination that appears to have worked for my setup is as follows. I believe the key was to drastically downgrade the Elasticsearch.
Elasticsearch=1.7.6 (with Java 8)
Django==2.1.7
Django-Haystack==2.8.1
elasticsearch==1.7.0
The two items below may or may not be relevant. I have not changed them.
DjangoCMS==3.7.0
aldryn-search==1.0.1
I am playing around with Flask. I have created an API using Flask-Restful and Flask-JWT. When Debug=True in Flask, and I do not send the Authorization Header, I get the response as However, when the debug=False, the response returned is Internal Server Error with this stack trace,
[2017-01-19 19:43:10,753] ERROR in app: Exception on /api_0_1/deals [GET]
Traceback (most recent call last):
File "C:\Users\ARFATS~1\Desktop\Dealflow\venv\lib\site-packages\flask\app.py", line 1612, in full_dispatch_request
rv = self.dispatch_request()
File "C:\Users\ARFATS~1\Desktop\Dealflow\venv\lib\site-packages\flask\app.py", line 1598, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "C:\Users\ARFATS~1\Desktop\Dealflow\venv\lib\site-packages\flask_restful\__init__.py", line 477, in wrapper
resp = resource(*args, **kwargs)
File "C:\Users\ARFATS~1\Desktop\Dealflow\venv\lib\site-packages\flask_jwt\__init__.py", line 176, in decorator
_jwt_required(realm or current_app.config['JWT_DEFAULT_REALM'])
File "C:\Users\ARFATS~1\Desktop\Dealflow\venv\lib\site-packages\flask_jwt\__init__.py", line 155, in _jwt_required
headers={'WWW-Authenticate': 'JWT realm="%s"' % realm})
JWTError: Authorization Required. Request does not contain an access token
I would like Flask-JWT to respond with the response which is there when Debug=True. However, I cannot use debug on Production servers. One way is to use my own jwt_required decorator. Is there any other way?Also, I would be happy to know what I am missing, if any. Thanks
You will need to add this setting to your flask app:
app.config['PROPAGATE_EXCEPTIONS'] = True
When debug is true, PROPAGATE_EXCEPTIONS is also set to true by default.
Perhaps consider checking out flask-jwt-extended instead (https://github.com/vimalloc/flask-jwt-extended), it takes care of the PROPAGATE_EXCEPTIONS for you. It aims to replace the abandoned flask-jwt library, and add some conviences when working with JWTs (such as refresh tokens, easily adding custom data to the JWTs, fresh vs non-fresh tokens, and more). Full disclosure, I'm the author of that extension.
Cheers.
So I'm trying to use Python to automate 508 compliance checking. There're a few hundred pages on our site, and at the moment a person is actually going through the site every week and tries to enter all the URLs by hand. The UIUC link below checks the request for the referer header and then returns the evaluation of that site. I can't get the request to actually work. I've looked all through SO and can't find anything that helps. The code that is screwy is below and below that the error message.
def fae(urltofae):
opener = urllib2.build_opener()
opener.addheaders = [('User-agent', 'Mozilla/5.0')]
#[('Referer': urltofae)]
r = opener.open('http://www.fae.cita.uiuc.edu/evaluate/link/')
print r
fae("http://www.example.com/")
And the Error:
File "<stdin>", line 1, in <module>
File "<stdin>", line 4, in fae
File "/Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/urllib2.py", line 400, in open
response = self._open(req, data)
File "/Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/urllib2.py", line 418, in _open
'_open', req)
File "/Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/urllib2.py", line 378, in _call_chain
result = func(*args)
File "/Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/urllib2.py", line 1207, in http_open
return self.do_open(httplib.HTTPConnection, req)
File "/Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/urllib2.py", line 1177, in do_open
raise URLError(err)
urllib2.URLError: <urlopen error [Errno 8] nodename nor servname provided, or not known>
And when I attempt to try to change the referer header (instead of the User-agent) I get formatting errors instead of it even getting to the request even though the format is identical to the one that it didn't complain about for the user-agent.
I'm still very much a new programmer so if I'm missing something blatant then I'm terribly sorry, but I have tried everything I can think of.
Thanks in advance, cheers.
OK so I switched my strategy, and it worked. Unfortunately, I have no idea why the below code worked, and the stuff above kept erroring me, but I have seen a couple of similar-ish questions (no specific answers) around the google so I figured I should post it.
vlz, appreciate the help, cheers.
def faeRequest2(urltofae):
r = urllib2.Request('http://fae.cita.illinois.edu/evaluate/link/', headers={'User-agent':'Mozilla/5.0', 'Referer':urltofae})
c = urllib2.urlopen(r)
print c.read()
I do not see any errory there. Is the url correct? Try using
'http://fae.cita.uiuc.edu/evaluate/link/'
instead of
'http://www.fae.cita.uiuc.edu/evaluate/link/'
The latter seems not to lead anywhere.
I have a function that will read data from a website, process it, and then load it into MongoDB. When I run this without threading it works fine but as soon as I set up celery tasks that just call this one function I frequently get the following error: "OperationFailure: database error: unauthorized db:dbname lock type:-1"
It's somewhat odd because if I run the non-celery version on multiple terminals, I do not get this error at all.
I suspect it has something to do with there not being an open connection to Mongo although in my code I'm opening one up right before every Mongo call.
The exact exception is below:
Task twitter[a974bfcc-d6ca-4baf-b36f-cae9143ce2d9] raised exception: OperationFailure(u'database error: unauthorized db:data lock type:-1 client:68.193.49.9',)
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/celery/execute/trace.py", line 36, in trace
return cls(states.SUCCESS, retval=fun(*args, **kwargs))
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/celery/app/task/__init__.py", line 232, in __call__
return self.run(*args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/celery/app/__init__.py", line 172, in run
return fun(*args, **kwargs)
File "/djangoblog/network/tasks.py", line 40, in twitter
n_twitter.GetTweetsTwitter(user)
File "/djangoblog/network/twitter.py", line 255, in GetTweetsTwitter
id = SaveTweet(user, network, tweet)
File "/djangoblog/network/twitter.py", line 150, in SaveTweet
if mmo.Moment.objects(user=user.id,source_id=id,network=network.id).count() == 0:
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/mongoengine/queryset.py", line 933, in count
return self._cursor.count(with_limit_and_skip=True)
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/mongoengine/queryset.py", line 563, in _cursor
self._cursor_obj = self._collection.find(self._query,
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/mongoengine/queryset.py", line 493, in _collection
if self._collection_obj.name not in db.collection_names():
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/pymongo/database.py", line 361, in collection_names
names = [r["name"] for r in results]
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/pymongo/cursor.py", line 703, in next
if len(self.__data) or self._refresh():
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/pymongo/cursor.py", line 666, in _refresh
self.__uuid_subtype))
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/pymongo/cursor.py", line 628, in __send_message self.__tz_aware)
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/pymongo/helpers.py", line 101, in _unpack_response error_object["$err"])
OperationFailure: database error: unauthorized db:data lock type:-1 client:68.193.49.9
Sorry for the formatting but if you look at the line that starts with mmo.Moment there's a connection being opened right before that's called.
Doing a bit of research it looks as if it has something to do with the way threading is handled in PyMongo - http://api.mongodb.org/python/1.5.1/faq.html#how-does-connection-pooling-work-in-pymongo - I may need to start closing the connections but I'd expect MongoEngine to be doing this..
This is likely due to the fact that you are not calling db.authenticate() when you start the new connection and are using auth on MongoDB.
Regarding the closing of threads, I would recommend making sure you are using connection pooling and letting the driver manage the pools (calling close() or similar manually can lead to a lot of pain).
For more info see the note in the pymongo documentation about using authenticate() in a multi-threaded environment.