Could someone please clarify me why this url http://www.nacolmeia.com.br/do/Home/oferta/EnER is not being accepted in a form generated from URLField's Django?
:)
Thanks
Are you hosting the site from the same server you are trying to validate it on? docs
Note that when you're using the
single-threaded development server,
validating a URL being served by the
same server will hang. This should not
be a problem for multithreaded
servers.
It doesn't look like its failing validation at the form level
>>> from django import forms
>>> f = forms.URLField()
>>> f.clean('http://www.nacolmeia.com.br/do/Home/oferta/EnER')
u'http://www.nacolmeia.com.br/do/Home/oferta/EnER'
>>> f.clean('sadfas')
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "/home/dev/.virtualenvs/thepidb/lib/python2.7/site-packages/django/forms/fields.py", line 171, in clean
self.run_validators(value)
File "/home/dev/.virtualenvs/thepidb/lib/python2.7/site-packages/django/forms/fields.py", line 160, in run_validators
raise ValidationError(errors)
ValidationError: [u'Enter a valid URL.']
>>>
If you don't need to validate that the website doesn't return a 404, in your models.py
url = models.URLField(verify_exists=False)
edit:
after some digging around in the django source code here and some messing around with the shell, I'm still not sure why the URL with caps is causing a redirect loop.
>>> from django.core.validators import URLValidator
>>> u = URLValidator(verify_exists=True)
>>> u.__call__('http://www.nacolmeia.com.br/do/Home/oferta/EnER')
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "/home/dev/.virtualenvs/thepidb/lib/python2.7/site-packages/django/core/validators.py", line 105, in __call__
raise broken_error
ValidationError: [u'This URL appears to be a broken link.']
>>> u.__call__('http://www.nacolmeia.com.br/do/home/oferta/ener')
>>>
The actual exception being raised is an HTTPError:
File "/usr/lib/python2.7/urllib2.py", line 606, in http_error_302
return self.parent.open(new, timeout=req.timeout)
File "/usr/lib/python2.7/urllib2.py", line 398, in open
response = meth(req, response)
File "/usr/lib/python2.7/urllib2.py", line 511, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib/python2.7/urllib2.py", line 430, in error
result = self._call_chain(*args)
File "/usr/lib/python2.7/urllib2.py", line 370, in _call_chain
result = func(*args)
File "/usr/lib/python2.7/urllib2.py", line 606, in http_error_302
return self.parent.open(new, timeout=req.timeout)
File "/usr/lib/python2.7/urllib2.py", line 398, in open
response = meth(req, response)
File "/usr/lib/python2.7/urllib2.py", line 511, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib/python2.7/urllib2.py", line 430, in error
result = self._call_chain(*args)
File "/usr/lib/python2.7/urllib2.py", line 370, in _call_chain
result = func(*args)
File "/usr/lib/python2.7/urllib2.py", line 596, in http_error_302
self.inf_msg + msg, headers, fp)
HTTPError: HTTP Error 302: The HTTP server returned a redirect error that would lead to an infinite loop.
The last 30x error message was:
Found
>>>
here are some posts talking about the HTTPError: here and here
seems like it has something to do with cookies, but I'm not able to offer a good explanation, I'll leave that to some one else.
A workaround that might work if you don't want to turn off validation but don't care about the capitalization of your urls is to override the clean_field method of your forms.
def clean_your_url_field(self):
return self.cleaned_data['your_url_field'].lower()
I think I found the issue. When you open this URL:
http://www.nacolmeia.com.br/do/Home/oferta/EnER
...it re-directs to this URL:
http://www.nacolmeia.com.br/do/Home/oferta/EnER/piracicaba/a-pascoa-chegou-na-planet-chokolate!-50-off-para-1-caixa-com-16-bombons-recheados--1-pao-de-mel-recheado-ou-1-caixa-com-16-trufas-recheadas--1-pao-de-mel-recheado-de-rs-47.10-por-rs-23.55.
The first URL is fine, but the re-directed one is 247 characters long. This "shouldn't" be a problem, except that models.fields.URLField has max_length which defaults to 200 characters. So it fails validation because it's too long.
Instead, increase the max_length and it should work: models.URLField(max_length=255) For info on the longest URL possible, see this SO question. It's definitely longer than 200 characters though.
EDIT: It only re-directs to the second URL when setting a cookie! If you re-visit the same page again with an existing cookie, it just displays the shorter URL.
But what about the lowercase URL? It appears your web-server is case-sensitive regarding URLs, and the lowercase version:
http://www.nacolmeia.com.br/do/home/oferta/ener
...displays a generic error page. It doesn't re-direct to the 247 character URL. So that passes validation since the only thing models.URLField cares about is; does it load a webpage or not?
Related
I'm testing an API endpoint that is supposed to raise a ValidationError in a Django model (note that the exception is a Django exception, not DRF, because it's in the model).
from rest_framework.test import APITestCase
class TestMyView(APITestCase):
# ...
def test_bad_request(self):
# ...
response = self.client.post(url, data)
self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)
However, my test errors out with an exception instead of passing. It doesn't even fail getting a 500 instead of 400, it doesn't get there at all. Isn't DRF's APIClient supposed to handle every exception? I've search online but found nothing. I've read that DRF doesn't handle Django's native ValidationError, but still that doesn't explain why I am not even getting a 500. Any idea what I'm doing wrong?
Full stack trace:
E
======================================================================
ERROR: test_cannot_create_duplicate_email (organizations.api.tests.test_contacts.TestContactListCreateView)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/code/organizations/api/tests/test_contacts.py", line 98, in test_cannot_create_duplicate_email
response = self.jsonapi_post(self.url(new_partnership), data)
File "/code/config/tests/base.py", line 166, in jsonapi_post
url, data=json.dumps(data), content_type=content_type)
File "/usr/local/lib/python3.7/site-packages/rest_framework/test.py", line 300, in post
path, data=data, format=format, content_type=content_type, **extra)
File "/usr/local/lib/python3.7/site-packages/rest_framework/test.py", line 213, in post
return self.generic('POST', path, data, content_type, **extra)
File "/usr/local/lib/python3.7/site-packages/rest_framework/test.py", line 238, in generic
method, path, data, content_type, secure, **extra)
File "/usr/local/lib/python3.7/site-packages/django/test/client.py", line 422, in generic
return self.request(**r)
File "/usr/local/lib/python3.7/site-packages/rest_framework/test.py", line 289, in request
return super(APIClient, self).request(**kwargs)
File "/usr/local/lib/python3.7/site-packages/rest_framework/test.py", line 241, in request
request = super(APIRequestFactory, self).request(**kwargs)
File "/usr/local/lib/python3.7/site-packages/django/test/client.py", line 503, in request
raise exc_value
File "/usr/local/lib/python3.7/site-packages/django/core/handlers/exception.py", line 34, in inner
response = get_response(request)
File "/usr/local/lib/python3.7/site-packages/django/core/handlers/base.py", line 115, in _get_response
response = self.process_exception_by_middleware(e, request)
File "/usr/local/lib/python3.7/site-packages/django/core/handlers/base.py", line 113, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/usr/local/lib/python3.7/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
return view_func(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/django/views/generic/base.py", line 71, in view
return self.dispatch(request, *args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/rest_framework/views.py", line 495, in dispatch
response = self.handle_exception(exc)
File "/usr/local/lib/python3.7/site-packages/rest_framework/views.py", line 455, in handle_exception
self.raise_uncaught_exception(exc)
File "/usr/local/lib/python3.7/site-packages/rest_framework/views.py", line 492, in dispatch
response = handler(request, *args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/rest_framework/generics.py", line 244, in post
return self.create(request, *args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/rest_framework/mixins.py", line 21, in create
self.perform_create(serializer)
File "/usr/local/lib/python3.7/site-packages/rest_framework/mixins.py", line 26, in perform_create
serializer.save()
File "/usr/local/lib/python3.7/site-packages/rest_framework/serializers.py", line 214, in save
self.instance = self.create(validated_data)
File "/code/organizations/api/serializers.py", line 441, in create
'partnership': self.context['partnership']
File "/usr/local/lib/python3.7/site-packages/rest_framework/serializers.py", line 943, in create
instance = ModelClass._default_manager.create(**validated_data)
File "/usr/local/lib/python3.7/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 422, in create
obj.save(force_insert=True, using=self.db)
File "/code/organizations/models.py", line 278, in save
self.full_clean()
File "/usr/local/lib/python3.7/site-packages/django/db/models/base.py", line 1203, in full_clean
raise ValidationError(errors)
django.core.exceptions.ValidationError: {'__all__': ['Supplier contact emails must be unique per organization.']}
Question: Isn't DRF's APIClient supposed to handle every exception?
Answer: No. It's a test client, it won't handle any uncaught exceptions, that's how test clients work. Test clients propagate the exception so that the test fails with a "crash" when an exception isn't caught. You can test that exceptions are raised and uncaught with self.assertRaises
Question: The APIView should return HTTP_400_BAD_REQUEST when I raise a ValidationError but the exception isn't caught.
Answer:
You should look at the source code for APIView.
Inside the dispatch() method, all exceptions raised while creating the response object are caught and the method handle_exception() is called.
Your exception is a ValidationError. The crucial lines are:
exception_handler = self.get_exception_handler()
context = self.get_exception_handler_context()
response = exception_handler(exc, context)
if response is None:
self.raise_uncaught_exception(exc)
If you haven't changed settings.EXCEPTION_HANDLER, you get the default DRF exception handler, source code here.
If handles Http404, PermissionDenied and APIException. The APIView itself actually also handles AuthenticationFailed and NotAuthenticated. But not ValidationError. So it returns None and therefore the view raises your ValidationError which stops your test.
You see that in your traceback:
File "/usr/local/lib/python3.7/site-packages/rest_framework/views.py", line 455, in handle_exception
self.raise_uncaught_exception(exc)
You can decide to handle more exceptions than the default ones handled by DRF, you can read this on custom exception handling.
EDIT: You can also raise rest_framework.exceptions.ValidationError instead of the standard Django ValidationError. That is an APIException and therefore will be handled by DRF as a HTTP400_BAD_REQUEST. [1]
Side note: Luckily DRF doesn't catch every single exception! If there's a serious flaw in your code you actually want your code to "crash" and produce an error log and your server to return a HTTP 500. Which is what happens here. The response would be an HTTP 500 if this wasn't the test client.
[1]https://github.com/encode/django-rest-framework/blob/3.9.0/rest_framework/exceptions.py#L142
Something in your code is causing a python error which is halting execution before your POST request can return a valid HTTP response. Your code doesn't even reach the line self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST) because there is no response.
If you're calling your tests in the normal way with ./manage.py test then you should see the traceback and be able to narrow down what caused the error.
Django >3.0
Starting with Django 3.0, Client constructor takes raise_request_exception parameter. Set it False and response 500 will returned instead of raising an exception.
Source:
https://docs.djangoproject.com/en/3.2/releases/3.0/#tests
self.client = Client(raise_request_exception=False)
Django <3.0
There is a workaround for this for Django 2.2 and earlier.
TL;DR:
Just mock the store_exc_info method of Client class.
from unittest import mock
class TestMyView(APITestCase):
...
def test_server_error(self):
mock.patch("django.test.client.Client.store_exc_info")
self.client.post("/url/, data)
...
Now, the client will return regular response object with status code = 500.
Explanation:
Django's Test client always raises exceptions, even if they were wrapped in 500 Server error response.
Client connects its method called store_exc_info to the got_request_exception Django's signal (source: https://github.com/django/django/blob/3.2.5/django/test/client.py#L712)
got_request_exception.connect(self.store_exc_info, dispatch_uid=exception_uid)
This method saves an exception info as self.exc_info. This attribute is later tested if its None. If it is, then error is raised. If not, response is normally returned (source: https://github.com/django/django/blob/2.2.24/django/test/client.py#L500)
Mocking the self.store_exc_info makes it do nothing, so exception info is not stored - therefore it's None later :)
I'm trying to do some functional testing on Flask view functions.
Currently I'm using login, logout from Flask Security module and when I try to follow the login and logout guide from flask's documentation(http://flask.pocoo.org/docs/0.12/testing/#logging-in-and-out), the 'post' of login seems to not working. I've been getting this same error when I try to post using requests module too.
My Flask-Security's login endpoint is /login_test/
Below are piece of my unit test code.
class TestUser(unittest.TestCase):
#run before each test
def setUp(self):
self.client = app.test_client()
db.create_all()
def tearDown(self):
#db.session.remove()
#DropEverything().drop_db()
pass
def login(self, email, password):
return self.client.post('/login_test/', data=dict(
email=email,
password=password
), follow_redirects=False)
def logout(self):
return self.client.get('/logout', follow_redirects=True)
def test_login_logout(self):
response = self.client.post('/login_test', data=dict(
email='admin',
password='admin'
), follow_redirects=False)
self.assertIn(b'You logged in', response.data)
The error message that I got after hitting test_login_logout is like below. The below is when I hit the url with '/login_test'
Ran 1 test in 0.187s
FAILED (failures=1)
Failure
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/unittest/case.py", line 58, in testPartExecutor
yield
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/unittest/case.py", line 600, in run
testMethod()
File "/Users/genom003dm/PycharmProjects/sample_accessioning_dev/app/tests/user_management_testing.py", line 38, in test_login_logout
), follow_redirects=False)
File "/Users/genom003dm/sample_accessioning_dev_virtual_env/lib/python3.5/site-packages/werkzeug/test.py", line 801, in post
return self.open(*args, **kw)
File "/Users/genom003dm/sample_accessioning_dev_virtual_env/lib/python3.5/site-packages/flask/testing.py", line 127, in open
follow_redirects=follow_redirects)
File "/Users/genom003dm/sample_accessioning_dev_virtual_env/lib/python3.5/site-packages/werkzeug/test.py", line 764, in open
response = self.run_wsgi_app(environ, buffered=buffered)
File "/Users/genom003dm/sample_accessioning_dev_virtual_env/lib/python3.5/site-packages/werkzeug/test.py", line 677, in run_wsgi_app
rv = run_wsgi_app(self.application, environ, buffered=buffered)
File "/Users/genom003dm/sample_accessioning_dev_virtual_env/lib/python3.5/site-packages/werkzeug/test.py", line 884, in run_wsgi_app
app_rv = app(environ, start_response)
File "/Users/genom003dm/sample_accessioning_dev_virtual_env/lib/python3.5/site-packages/flask/app.py", line 1997, in __call__
return self.wsgi_app(environ, start_response)
File "/Users/genom003dm/sample_accessioning_dev_virtual_env/lib/python3.5/site-packages/flask/app.py", line 1985, in wsgi_app
response = self.handle_exception(e)
File "/Users/genom003dm/sample_accessioning_dev_virtual_env/lib/python3.5/site-packages/flask/app.py", line 1540, in handle_exception
reraise(exc_type, exc_value, tb)
File "/Users/genom003dm/sample_accessioning_dev_virtual_env/lib/python3.5/site-packages/flask/_compat.py", line 33, in reraise
raise value
File "/Users/genom003dm/sample_accessioning_dev_virtual_env/lib/python3.5/site-packages/flask/app.py", line 1982, in wsgi_app
response = self.full_dispatch_request()
File "/Users/genom003dm/sample_accessioning_dev_virtual_env/lib/python3.5/site-packages/flask/app.py", line 1614, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/Users/genom003dm/sample_accessioning_dev_virtual_env/lib/python3.5/site-packages/flask/app.py", line 1517, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/Users/genom003dm/sample_accessioning_dev_virtual_env/lib/python3.5/site-packages/flask/_compat.py", line 33, in reraise
raise value
File "/Users/genom003dm/sample_accessioning_dev_virtual_env/lib/python3.5/site-packages/flask/app.py", line 1612, in full_dispatch_request
rv = self.dispatch_request()
File "/Users/genom003dm/sample_accessioning_dev_virtual_env/lib/python3.5/site-packages/flask/app.py", line 1590, in dispatch_request
self.raise_routing_exception(req)
File "/Users/genom003dm/sample_accessioning_dev_virtual_env/lib/python3.5/site-packages/flask/app.py", line 1576, in raise_routing_exception
raise FormDataRoutingRedirect(request)
flask.debughelpers.FormDataRoutingRedirect: b'A request was sent to this URL (http://localhost/login_test) but a redirect was issued automatically by the routing system to "http://localhost/login_test/". The URL was defined with a trailing slash so Flask will automatically redirect to the URL with the trailing slash if it was accessed without one. Make sure to directly send your POST-request to this URL since we can\'t make browsers or HTTP clients redirect with form data reliably or without user interaction.\n\nNote: this exception is only raised in debug mode'
If I change the URL to /login_test/ then I get HTTP 400 errors. I'm assuming that this is happening due to the fact that I'm missing form object for login? (but in this case I don't have form object because I'm trying just trying to login with post api).
I want to know is there a way to login using flask-security's /login_test/ url.
Thanks
Ok, I found an answer. The reason why I was only seeing HTTP 400 errors instead of the specifics of HTTP 400 errors are because I put the error handling on Flask app and it just showed me 400 rather than what the actual error was. Once I removed the HTTP 400 error handling, it was saying that the CSRF token was missing. So what I did was to WTF_CSRF_ENABLED = False in app config file.
I try to read this url - http://malc0de.com/bl/BOOT in Python
import urllib2
threats = urllib2.urlopen("http://malc0de.com/bl/BOOT")
But I got this error:
Traceback (most recent call last):
File "C:\Android\android_workspace\pro2\test.py", line 2, in <module>
threats = urllib2.urlopen("http://malc0de.com/bl/BOOT")
File "C:\Python27\lib\urllib2.py", line 154, in urlopen
return opener.open(url, data, timeout)
File "C:\Python27\lib\urllib2.py", line 437, in open
response = meth(req, response)
File "C:\Python27\lib\urllib2.py", line 550, in http_response
'http', request, response, code, msg, hdrs)
File "C:\Python27\lib\urllib2.py", line 475, in error
return self._call_chain(*args)
File "C:\Python27\lib\urllib2.py", line 409, in _call_chain
result = func(*args)
File "C:\Python27\lib\urllib2.py", line 558, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 403: Forbidden
What can I do to fix it?
This is a HTTP error unrelated to python or urllib. It says that, for some reason, you are not allowed to view this particular page.
It seems to me that the site owner filters access by bots/crawlers, because I can open it in Firefox, but not via urllib. It might filter based on user agent, which may be changed, see Changing user agent on urllib2.urlopen, although this might be bad etiquette.
Unable to create a response with this api.I am unable to call the function locu_search('new york'). I get the following error shown below. I am using Komodo as my IDE, this started when I created a new python shell.
import urllib2
import json
local_api = '0d5897aae41eeafbd62ad0815af15cc42b2ed7c0'
def locu_search(query):
api_key = local_api
url = 'https://api.locu.com/v1_0/venue/search/?api_key=' + api_key
locality = query.replace('','%20')
final_url = url + "&locality=" + locality + "&category=restaurant"
json_obj = urllib2.urlopen(final_url)
data = json.load(json_obj)
for item in data['objects']:
print item['name'],item['phone']
locu_search('new york')
The error is listed below:
**Traceback (most recent call last):
File "<console>", line 0, in <module>
File "<console>", line 0, in locu_search
File "c:\python27\lib\urllib2.py", line 154, in urlopen
return opener.open(url, data, timeout)
File "c:\python27\lib\urllib2.py", line 437, in open
response = meth(req, response)
File "c:\python27\lib\urllib2.py", line 550, in http_response
'http', request, response, code, msg, hdrs)
File "c:\python27\lib\urllib2.py", line 475, in error
return self._call_chain(*args)
File "c:\python27\lib\urllib2.py", line 409, in _call_chain
result = func(*args)
File "c:\python27\lib\urllib2.py", line 558, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 400: BAD_REQUEST**
400 Bad Request should give you a headsup about the problem , this is basically due to a malformed request and I strongly suspect the culprit is in th line url = 'https://api.locu.com/v1_0/venue/search/?api_key=' + api_key , check if api_key token is invalid or no longer valid.
I'm able to open the webpage by simply entering the URL link in my chrome browser
But when i move this URL link to below code, it will prompt me the error message:
CODE:
import urllib2
url = 'http://www.klse.info/companies/listed-companies/alphabet/A'
page = urllib2.urlopen(url).read()
ERROR:
File "C:\Python27\lib\urllib2.py", line 127, in urlopen
return _opener.open(url, data, timeout)
File "C:\Python27\lib\urllib2.py", line 410, in open
response = meth(req, response)
File "C:\Python27\lib\urllib2.py", line 523, in http_response
'http', request, response, code, msg, hdrs)
File "C:\Python27\lib\urllib2.py", line 448, in error
return self._call_chain(*args)
File "C:\Python27\lib\urllib2.py", line 382, in _call_chain
result = func(*args)
File "C:\Python27\lib\urllib2.py", line 531, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 403: Forbidden
Anyone have idea with this?
I tried to change the URL link to other link address and it does work.
Is the website set restriction or anything i should take care for?
How to get rid from HTTP Error 403: Forbidden?
Please refer to the below code...
url = 'http://www.klse.info/companies/listed-companies/alphabet/A'
req = urllib2.Request(url, headers={'User-Agent' : "Magic Browser"})
page = urllib2.urlopen(req).read()