I am receiving a large number of request for PHP files that does not exist in my wordpress.
They show up in nginx error logs as following two examples:
2019/06/24 03:16:43 [error] 4201#4201: *17573871 FastCGI sent in stderr: "Unable to open primary script: /var/www/html/vendor/phpunit/phpunit/src/Util/PHP/eval-stdin.php (No such file or directory)" while reading response header from upstream, client: 172.68.189.50, server: mywebsite.net, request: "GET /vendor/phpunit/phpunit/src/Util/PHP/eval-stdin.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php/php7.2-fpm.sock:", host: "mywebsite.net"
2019/06/24 03:16:43 [error] 4201#4201: *17573871 FastCGI sent in stderr: "Unable to open primary script: /var/www/html/vendor/phpunit/phpunit/src/Util/PHP/eval-stdin.php (No such file or directory)" while reading response header from upstream, client: 172.68.189.50, server: mywebsite.net, request: "POST /vendor/phpunit/phpunit/src/Util/PHP/eval-stdin.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php/php7.2-fpm.sock:", host: "mywebsite.net"
I have tried making a noscript filter.
In file /etc/fail2ban/jail.local I put:
[nginx-noscript]
enabled = true
port = http,https
filter = nginx-noscript
logpath = /var/log/nginx/error.log
maxretry = 2
In File /etc/fail2ban/filter.d/nginx-noscript.conf I put:
[Definition]
failregex = \[error\] \d+#\d+: \*\d+ (FastCGI sent in stderr: "Unable to open primary script:)
ignoreregex =
But this filter is not catching these type of 404s. After systemctl restart fail2ban the fail2ban logs shows these error messages.
2019-06-24 16:11:05,548 fail2ban.filter [6182]: ERROR No failure-id group in '\[error\] \d+#\d+: \*\d+ (FastCGI sent in stderr: "Unable to open primary script:)'
2019-06-24 16:11:05,548 fail2ban.transmitter [6182]: WARNING Command ['set', 'nginx-noscript', 'addfailregex', '\\[error\\] \\d+#\\d+: \\*\\d+ (FastCGI sent in stderr: "Unable to open primary script:)'] has failed. Received RegexException('No failure-id group in \'\\[error\\] \\d+#\\d+: \\*\\d+ (FastCGI sent in stderr: "Unable to open primary script:)\'',)
2019-06-24 16:11:05,549 fail2ban [6182]: ERROR NOK: ('No failure-id group in \'\\[error\\] \\d+#\\d+: \\*\\d+ (FastCGI sent in stderr: "Unable to open primary script:)\'',)
What am I doing wrong. What will be the full regex for such nginx error logs.
This should work (for fail2ban >= 0.10):
failregex = ^\s*\[error\] \d+#\d+: \*\d+ FastCGI sent in stderr: "Unable to open primary script: [^"]*" while reading response header from upstream, client: <ADDR>
If you have older versions (0.9 or below), use <HOST> instead of <ADDR> (and better disable DNS-lookup for jail with usedns = no).
Related
I'd like to ban via fail2ban anyone generating both these type of lines in my nginx error.log file :
2019/12/15 20:12:12 [error] 640#640: *6 open() "/data/xxxxxx.com/www/50x.html" failed (2: No such file or directory), client: 35.187.45.148, server: xxxxxx.com, request: "GET /external.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: x.x.x.x
2019/12/16 13:42:59 [crit] 647#647: *41 connect() to unix:/var/run/php5-fpm.sock failed (2: No such file or directory) while connecting to upstream, client: 35.233.78.55, server: xxxxxx.com, request: "GET /external.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "x.x.x.x"
I thought these lines would work :
open() .* client: < HOST >
connect() to .* client: < HOST >
But they apparently don't (tested with fail2ban-regex). Here's my complete filter :
[Definition]
failregex = open() .* client: < HOST >
connect() to .* client: < HOST >
FastCGI sent in stderr: "Primary script unknown" while reading response header from upstream, client: < HOST >
datepattern = {^LN-BEG}
Note : the last one (FastCGI...) does work. Would something be wrong with ".*" ?
Parentheses () are both regex metacharacters, meaning they have a special meaning in regex. For example, here is what your first regex is actually matching:
open .* client:
That is, the () are actually a zero-width capture group, and so are the same as matching nothing at all. Since you are trying to match open followed by a space, therefore you are failing to get a match. Here is the corrected version:
[Definition]
failregex = open\(\) .* client: < HOST >
connect\(\) to .* client: < HOST >
FastCGI sent in stderr: "Primary script unknown" while reading response header from upstream, client: < HOST >
datepattern = {^LN-BEG}
Note that if we want to match literal parentheses, we should escape them with backslash.
I'm running django application with uwsgi + Nginx with crontab command given below
* * * * * /usr/local/bin/lockrun --lockfile /path/to/lock_file -- uwsgi --close-on-exec -s /path/to/socket_file --chdir /path/to/project/settings/folder/ --pp .. -w project_name.wsgi -C666 -p 3 -H /path/to/virtualenv/folder/ 1>> /path/to/log_file 2>> /path/to/error_log
but nginx error log file shows the error
*83 connect() to unix:///path/to/socket_file failed (111: Connection refused) while connecting to upstream, client: xxx.xxx.xx.xxx, server:
localhost, request: "GET /auth/status/ HTTP/1.1", upstream:
"uwsgi://unix:///path/to/socket_file:", host: "xx.xxx.xx.xxx",
referrer: "http://xx.xxx.xx.xxx/"
Check if .sock file is created:
ls /path/to/socket_file
If it is created, check permissions:
ls -l /path/to/socket_file
Check where is the socket file created, may be due to chdir command, it has created somewhere else.
lsof | grep socket_file
If they are fine, can you share ngnix config details?
It's simple and better to use a file to supply configuration parameters.
Sample uwsgi.ini file
[uwsgi]
print = -------------------------------------------------
print = Hello UWSGI is ready to start...
print = -------------------------------------------------
# the base directory (full path)
chdir = /path/to/chdir/
#Application's wsgi file/module
module = wsgi:application
# process-related settings
master = true
# maximum number of worker processes
processes = 8
socket = /path/to/myuwsgi.sock
pidfile = /opt/uwsgi/mywsgi.pid
chmod-socket = 660
logto2 = /log/path/uwsgi.log
to apply these while uwsgi is starting up, do
uwsgi --ini uwsgi.ini
Alternatively, to direct logs to specific file:
uwsgi <your other paramerters> --logto2 /log/path/uwsgi.log
Preface: I'm running nginx + gunicorn + django on an amazon ec2 instance using s3boto as a default storage backend. I am free tier. The ec2 security group allows: http, ssh, & https.
I'm attempting to send a multipart/form-data request containing a single element: a photo. When attempting to upload the photo, the iPhone (where the request is coming from) hangs. The photo is around 9.5 MB in size.
When I check the nginx-access.logs:
"POST /myUrl/ HTTP/1.1" 400 5 "-""....
When I check the nginx-error.logs:
[error] 5562#0: *1 readv() failed (104: Connection reset by peer) while reading upstream, client: my.ip.addr.iphone, server: default, request: "POST /myUrl/ HTTP/1.1", upstream: "http://127.0.0.1:8000/myUrl/", host: "ec2-my-server-ip-addr.the-location-2.compute.amazonaws.com"
[info] 5562#0: *1 client my.ip.addr.iphone closed keepalive connection
I really cannot figure out why this is happening... I have tried changing the /etc/nginx/sites-available/default timeout settings...
server { ...
client_max_body_size 20M;
client_body_buffer_size 20M;
location / {
keepalive_timeout 300;
proxy_read_timeout 300;
}
}
Any thoughts?
EDIT: After talking on IRC a little more, his problem is the 403 itself, not the nginx error. Leaving my comments on the nginx error below, in case anyone else stumbles into it someday.
I ran into this very problem last week and spent quite a while trying to figure out what was going on. See here: https://github.com/benoitc/gunicorn/issues/872
Basically, as soon as django sees the headers, it knows that the request isn't authenticated. It doesn't wait for the large request body to finish uploading; it responds immediately, and gunicorn closes the connection right after. nginx keeps sending data, and the end result is that gunicorn sends a RST packet to nginx. Once this happens, nginx cannot recover and instead of sending the actual response from gunicorn/django, it sends a 502 Bad Gateway.
I ended up putting in a piece of middleware that acecsses a couple fields in the django request, which ensures that the entire request body is downloaded before Django sends a response:
checker = re.compile(feed_url_regexp)
class AccessPostBodyMiddleware:
def process_request(self, request):
if checker.match(request.path.lstrip('/')) is not None:
# just need to access the request info here
# not sure which one of these actually does the trick.
# This will download the entire request,
# fixing this random issue between gunicorn and nginx
_ = request.POST
_ = request.REQUEST
_ = request.body
return None
However, I do not have control of the client. Since you do (in the form of your iphone app), maybe you can find a way to handle the 502 Bad Gateway. That will keep your app from having to send the entire request twice.
I have a Django site developed using Pinax. When I deploy it in apache+mod_wsgi, it works fine. But when I deploy it in nginx+uwsgi, it nearly works fine, but the page includes a {% csrf_token %} tag. The crashed page dose not display a Django error page, but displays an Nginx 502 error page. The Nginx error log is:
2012/06/08 09:11:59 [error] 30224#0: *79 upstream sent invalid header
while reading response header from upstream, client: 211.142.12.3,
server: mysite.com, request: "GET /discuss/ HTTP/1.1", upstream:
"uwsgi://127.0.0.1:9001", host: "mysite.com", referrer:
"http://mysite.com/"
uwsgi displays:
{address space usage: 42319872 bytes/40MB} {rss usage: 22573056
bytes/21MB} [pid: 21398|app: 0|req: 1/3] 110.178.82.221 () {42 vars in
988 bytes} [Fri Jun 8 18:27:01 2012] GET /discuss/ => generated 31139
bytes in 2306 msecs (HTTP/1.1 200) 5 headers in 358 bytes (1 switches
on core 0)
The error occurs on a GET request, not a POST request. I tested this - when I delete the csrf_token token from the template, it's OK. So, there must be a relationship between the token and the error, not anything else.
What's going on?
Okay, it's solved. I had installed uwsgi by compiling the source. Now I delete that version, and reinstall it using pip install uwsgi, and everything is fine!
I have a django app hosted via Nginx and uWsgi. In a certain very simple request, I get different behaviour for GET and POST, which should not be the case.
The uWsgi daemon log:
[pid: 32454|app: 0|req: 5/17] 127.0.0.1 () {36 vars in 636 bytes} [Tue Oct 19 11:18:36 2010] POST /buy/76d4f520ae82e1dfd35564aed64a885b/a_2/10/ => generated 80 bytes in 3 msecs (HTTP/1.0 440) 1 headers in 76 bytes (0 async switches on async core 0)
[pid: 32455|app: 0|req: 5/18] 127.0.0.1 () {32 vars in 521 bytes} [Tue Oct 19 11:18:50 2010] GET /buy/76d4f520ae82e1dfd35564aed64a885b/a_2/10/ => generated 80 bytes in 3 msecs (HTTP/1.0 440) 1 headers in 76 bytes (0 async switches on async core 0)
The Nginx accesslog:
127.0.0.1 - - [19/Oct/2010:18:18:36 +0200] "POST /buy/76d4f520ae82e1dfd35564aed64a885b/a_2/10/ HTTP/1.0" 440 0 "-" "curl/7.19.5 (i486-pc-linux-gnu) libcurl/7.19.5 OpenSSL/0.9.8g zlib/1.2.3.3 libidn/1.15"
127.0.0.1 - - [19/Oct/2010:18:18:50 +0200] "GET /buy/76d4f520ae82e1dfd35564aed64a885b/a_2/10/ HTTP/1.0" 440 80 "-" "curl/7.19.5 (i486-pc-linux-gnu) libcurl/7.19.5 OpenSSL/0.9.8g zlib/1.2.3.3 libidn/1.15"
The Nginx errorlog:
2010/10/19 18:18:36 [error] 4615#0: *5 readv() failed (104: Connection reset by peer) while reading upstream, client: 127.0.0.1, server: localhost, request: "POST /buy/76d4f520ae82e1dfd35564aed64a885b/a_2/10/ HTTP/1.0", upstream: "uwsgi://unix:sock/uwsgi.sock:", host: "localhost:9201"
In essence, Nginx somewhere loses the response if I use POST, not so if I use GET.
Anybody knows something about that?
Pass --post-buffering 1 to uwsgi
This will automatically buffer all the http body > 1 byte
The problem is raised by the way nginx manages upstream disconnections
I hit the same issue, but on my case I can't disable "uwsgi_pass_request_body" as most times (but not always) my app do need the POST data.
This is the workaround I found, while this issue is not fixed in uwsgi:
http://permalink.gmane.org/gmane.comp.python.wsgi.uwsgi.general/813
import django.core.handlers.wsgi
class ForcePostHandler(django.core.handlers.wsgi.WSGIHandler):
"""Workaround for: http://lists.unbit.it/pipermail/uwsgi/2011-February/001395.html
"""
def get_response(self, request):
request.POST # force reading of POST data
return super(ForcePostHandler, self).get_response(request)
application = ForcePostHandler()
I am facing the same issues. I tried all solutions above, but they were not working. Ignoring the response body in my case is simply not an option.
Apparently it is a bug with nginx and uwsgi when dealing with POST requests whose response is smaller than 4052 bytes
What solved it for me was adding "--pep3333-input" to the parameter list of uwsgi. After that all POSTs are returned correctly.
Versions of nginx/uwsgi I'm using:
$ nginx -V
nginx: nginx version: nginx/0.9.6
$ uwsgi --version
uWSGI 0.9.7
After a lucky find in further research (http://answerpot.com/showthread.php?577619-Several%20Bugs/Page2) I found something that helped...
Supplying the uwsgi_pass_request_body off; parameter in the Nginx conf resolves this problem...