I am trying to set up a web service on AWS Elastic Beanstalk, however whenever I submit a request to the server, it responds with a 404 error.
Is there a way to view a log of the traffic to my AWS server, so I can view whether or not requests are actually being submitted, and whether or not they are properly formed so I may potentially narrow down the origin of the problem?
Log for requests sent to the server can be found under:
/var/log/nginx/access.log
You can see a snippet of my log below:
-------------------------------------
/var/log/nginx/access.log
-------------------------------------
141.163.107.3 - - [25/Nov/2016:15:50:33 +0000] "GET / HTTP/1.1" 200 2111 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36"
141.163.107.3 - - [25/Nov/2016:15:51:01 +0000] "GET /HelloWorld HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36"
Related
So when a 'ws' request hits my django server on azure app service in logs it shows
2022-05-27T03:40:41.606708018Z Not Found: /tm/123
2022-05-27T03:40:41.608467127Z 169.254.130.1 - - [27/May/2022:03:40:41 +0000] "GET /tm/123 HTTP/1.1" 404 4190 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.5005.61 Safari/537.36"
So this may be due to daphne service is not on in my server environment
when I put a startup command for daphne it gives error
My flask chat application was working fine on localhost development server. But when i hosted my app on heroku nothing is working properly and i checked console to find lots of Bad Requests.
Both GET and POST requests are getting 400 errors.
"GET /socket.io/?EIO=3&transport=polling&t=1592543494128-64&sid=6740d63740c24f6ca6bd7234325a7c21 HTTP/1.1" 400 11 "https://flack-ma.herokuapp.com/" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.106 Safari/537.36"
"POST /socket.io/?EIO=3&transport=polling&t=1592543547379-72&sid=8eadd5ac2bcf4ff1a414d878aa506839 HTTP/1.1" 400 11 "https://flack-ma.herokuapp.com/" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.106 Safari/537.36"
App worked fine on localhost.
Deployment Link
Project Repo
I dont know whats wrong.
In our code:
app = Flask(name)
blueprint = Blueprint('api',name,url_prefix='/api')
api = Api(blueprint, doc='/doc/')
app.register_blueprint(blueprint)
From the log..it's not using the prefix...swaggerui is off root.
"GET /swaggerui/bower/swagger-ui/dist/fonts/DroidSans-Bold.ttf HTTP/1.1" 200 0
"http://10.99.72.221:5002/api/doc/" "Mozilla/5.0 (Macintosh; Intel Mac OS X
10_10_5) AppleWebKit/603.3.8 (KHTML, like Gecko) Version/10.1.2
Safari/603.3.8"
It's causing issues for AWS Application Load Balancer, as the only route I can define is /api, since it's running inside a Docker container.
Is there a way to insert a path in front of /swaggerui?
I have a restful API created using Flask and Flask-Restful. Everything works fine using the development server. All routes are in a Blueprint, though the particular route we're dealing with here is not a Flask-Restful one. It's just a normal Flask route.
I am also using Flask-CORS.
To deploy, everything is dockerized and I'm using CherryPy as the WSGI host. So a CherryPy app hosts the Flask app in a container.
I'm using Traefik as a reverse proxy in another container.
If I make the following request in Chrome by pasting the URL in, the GET request works:
https://api.my-app.new/api/admin/user?_end=10&_order=DESC&_sort=id&_start=0
However, if I attemtp to make the same GET request from a React app, an OPTIONS preflight request is made and it is failing with a 404.
I've traced through it as best I can in PyCharm and the problem seems to be in the following code in Flask's app.py:
def preprocess_request(self):
bp = _request_ctx_stack.top.request.blueprint
Basically, the blueprint is not found.
What is actually getting sent is seen in this curl call:
curl 'https://api.my-app.new/api/admin/user?_end=10&_order=DESC&_sort=id&_start=0' \
-X OPTIONS -H 'access-control-request-method: GET' -H 'origin: https://admin.my-app.new' \
-H 'accept-encoding: gzip, deflate, br' -H 'accept-language: en-US,en;q=0.9' \
-H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36' \
-H 'accept: */*' -H 'referer: https://admin.my-app.new/' -H 'authority: api.my-app.new' \
-H 'access-control-request-headers: authorization,content-type' --compressed
And if I print out the headers received by the Flask-app (in #app.before_request), I get the following:
[2018-01-25 04:31:48,438] INFO - X-Forwarded-Server: cb5d56692c6d
Referer: https://admin.my-app.new/
Accept-Language: en-US,en;q=0.9
Origin: https://admin.my-app.new
X-Real-Ip: 172.19.0.1
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36
Access-Control-Request-Headers: authorization,content-type
X-Forwarded-Proto: https
Host: api.my-app.new
Accept: */*
Access-Control-Request-Method: GET
X-Forwarded-Host: api.my-app.new
X-Forwarded-For: 172.19.0.1
X-Forwarded-Port: 443
Accept-Encoding: gzip, deflate, br
[2018-01-25 04:31:48,440] INFO - Error 404:/api/admin/user?_end=10&_order=DESC&_sort=id&_start=0: 404 Not Found: The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again. ("/app/app/routes.py:87")
Now, if I make the same request with the app running without the traefik reverse proxy, it works. The only thing that is different in the curl
request is that I'm using http, instead of https.
Here are the headers that Flask receives in this case:
[2018-01-24 21:43:43,199] INFO - Referer: https://admin.my-app.new/
Origin: https://admin.my-app.new
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36
Authority: api.my-app.new
Access-Control-Request-Headers: authorization,content-type
Host: api.my-app.new:8000
Accept: */*
Access-Control-Request-Method: GET
Accept-Language: en-US,en;q=0.9
Accept-Encoding: gzip, deflate, br
[24/Jan/2018 21:43:43] "OPTIONS /api/admin/user?_end=10&_order=DESC&_sort=id&_start=0 HTTP/1.1" 200 -
I figure that there is something wrong with the headers or something that is confusing Flask. I saw another post from 2014 that mentioned needing the SERVER_NAME, but that didn't help.
Finally, I had originally used NGinx as a reserve proxy and had gotten everything working. One of the the things
that was hard to get working with NGinx was redirects and OPTION requests. I did get it working after madly chasing down various blog
posts, but as I look back on it as I right this, I notice a curious thing: I wound up writing script in nginx.conf
to automatically return 200 for all OPTIONS requests!
Any idea why the OPTION requests are failing?
As webKnjaZ notes: "It's a bug."
After digging in deeper, I discovered that the problem that Flask was getting different URL for the GET request than for the OPTIONS request, and that if CherryPy was removed the problem went away. That led me to this CherryPy issue which describes my situation exactly.
https://github.com/cherrypy/cherrypy/issues/1662
The author noted that the bug appeared when moving from CherryPy 11 to 12 (I was on 13.x) so I tried downgrading to 11.0.0 and that fixed it.
I am running a Django application on AWS Elastic Beanstalk. I keep having alerts (for days now) because the following user agent constantly tries to access some pages: "Mozilla/5.0 Jorgee"
[30/Jul/2017:13:55:55 +0000] "HEAD /pma/ HTTP/1.1" 403 - "-" "Mozilla/5.0 Jorgee"
[30/Jul/2017:13:55:55 +0000] "HEAD /db/ HTTP/1.1" 403 - "-" "Mozilla/5.0 Jorgee"
[30/Jul/2017:13:55:56 +0000] "HEAD /admin/ HTTP/1.1" 403 - "-" "Mozilla/5.0 Jorgee"
[30/Jul/2017:13:55:56 +0000] "HEAD /mysql/ HTTP/1.1" 403 - "-" "Mozilla/5.0 Jorgee"
[30/Jul/2017:13:55:56 +0000] "HEAD /database/ HTTP/1.1" 403 - "-" "Mozilla/5.0 Jorgee"
[30/Jul/2017:13:55:57 +0000] "HEAD /db/phpmyadmin/ HTTP/1.1" 403 - "-" "Mozilla/5.0 Jorgee"
[30/Jul/2017:13:55:57 +0000] "HEAD /db/phpMyAdmin/ HTTP/1.1" 403 - "-" "Mozilla/5.0 Jorgee"
[30/Jul/2017:13:55:57 +0000] "HEAD /sqlmanager/ HTTP/1.1" 403 - "-" "Mozilla/5.0 Jorgee"
[30/Jul/2017:13:55:58 +0000] "HEAD /mysqlmanager/ HTTP/1.1" 403 - "-" "Mozilla/5.0 Jorgee"
[30/Jul/2017:13:55:58 +0000] "HEAD /php-myadmin/ HTTP/1.1" 403 - "-" "Mozilla/5.0 Jorgee"
[30/Jul/2017:13:55:58 +0000] "HEAD /phpmy-admin/ HTTP/1.1" 403 - "-" "Mozilla/5.0 Jorgee"
[30/Jul/2017:13:55:59 +0000] "HEAD /mysqladmin/ HTTP/1.1" 403 - "-" "Mozilla/5.0 Jorgee"
[30/Jul/2017:13:55:59 +0000] "HEAD /mysql-admin/ HTTP/1.1" 403 - "-" "Mozilla/5.0 Jorgee"
[30/Jul/2017:13:55:59 +0000] "HEAD /admin/phpmyadmin/ HTTP/1.1" 403 - "-" "Mozilla/5.0 Jorgee"
It used to return 404, but I managed to block it to 403 thanks to the following line in settings.py:
DISALLOWED_USER_AGENTS = (re.compile(r'Mozilla\/5.0 Jorgee'), )
Is there a way to simply block it from even getting to the Django level? Or a way to stop getting it written to the logs? It generates Health Check alerts :-/
You could create an AWS Web Application Firewall with a rule to reject traffic using that user agent string. Then attach the WAF to the Elastic Load Balancer in your Elastic Beanstalk environment.
Alternatively, you could create a rule in the reverse-proxy running on your Elastic Beanstalk EC2 instances to block that traffic before it gets to Django. I'm not sure if Django apps on EB default to using Apache or Nginx for the reverse proxy. You'd have to figure out which one you are using and then look up how to configure that to block traffic based on a user-agent string.
It's not clear to me how this traffic is causing health check alerts in your application. If it is spamming your app with so much traffic that your server becomes overloaded and unresponsive, then I would recommend using a WAF to block it so that your server(s) will never have to see the traffic at all.
You could use AWS WAF rules to block anything you want from there. Nevertheless I assume that you've created environment with Classic Load Balancer. Migration to the Application Load Balancer is not a pain indeed. And what it does is way more than Classic Load Balancer, with some chances for upgrade in the future. If you would consider moving out to the ALB (we did this long time ago, so every new environment has it), the there's easy way to block these things out, or just move them aside so they don't interrupt.
For us, the easiest approach was to define AWS WAF rule to have string condition matching the Host header against some defined domains we publish our sites with. If there's any request, that is not using this domain, it will get 403 and won't generate things like requests to the ELB are failing with 5xx on Elastic Beanstalk. If you take a closer look at logs on ELB level, these bots tend to use direct IP adresses, not DNS names, so it generates errors when accessing website with defined server_names in Nginx, for instance.
So thanks to this, we have clearer logs and there's no more health warning on EB environment, which was a standard before.
I described this case in details here https://www.mkubaczyk.com/2017/10/10/use-aws-waf-block-bots-jorgee-500-status-elastic-beanstalk if you need some guidance or screenshots.
Try to use ModSecurity , Where you can control your traffic to apache server
https://github.com/SpiderLabs/ModSecurity/wiki/Reference-Manual
https://github.com/SpiderLabs/owasp-modsecurity-crs/tree/v3.0/master/rules
add user agent to be blocked to this file in your server
https://github.com/SpiderLabs/owasp-modsecurity-crs/blob/v3.0/master/rules/crawlers-user-agents.data