I am setting up Fail2ban on my server, recently got a lots bad bots is crawling my site cause my SQL server down
From my Apache2 logs
51.255.65.13 - - [10/Dec/2017:12:03:19 +0800] "GET /crew/nm0935095-gary-winick HTTP/1.0" 200 17985 "-" "Mozilla/5.0 (compatible; AhrefsBot/5.2; +http://ahrefs.com/robot/)"
51.255.65.30 - - [10/Dec/2017:12:03:31 +0800] "GET /movie/tt0498567-summer-time-machine-blues HTTP/1.0" 200 17658 "-" "Mozilla/5.0 (compatible; AhrefsBot/5.2; +http://ahrefs.com/robot/)"
217.182.132.190 - - [10/Dec/2017:12:03:36 +0800] "GET /movie/tt1705064-genji-monogatari:-sennen-no-nazo/ HTTP/1.0" 200 17344 "-" "Mozilla/5.0 (compatible; AhrefsBot/5.2; +http://ahrefs.com/robot/)"
how to create a failregex for "ahrefs.com" ?
Many Thanks
In order to catch anything containing "ahrefs.com", your failregex would look as follows:
failregex = ^<HOST>.*ahrefs\.com.*
Where the <HOST> tag is built-in Fail2ban as an alias for (?:::f{4,6}:)?(?P<host>\S+):
https://www.fail2ban.org/wiki/index.php/Apache
Related
Good afternoon people,
I created an environment in Elastic Beanstalk and uploaded a NODEjs application an api with express.
She's working fine, all right.
But the integrity of the environment is reported as serious, and this monitoring attempt appears in the logs.
----------------------------------------
/var/log/nginx/access.log
----------------------------------------
172.31.46.198 - - [03/Nov/2021:19:14:13 +0000] "GET / HTTP/1.1" 404 139 "-" "ELB-HealthChecker/2.0" "-"
172.31.1.181 - - [03/Nov/2021:19:14:13 +0000] "GET / HTTP/1.1" 404 139 "-" "ELB-HealthChecker/2.0" "-"
172.31.30.127 - - [03/Nov/2021:19:14:13 +0000] "GET / HTTP/1.1" 404 139 "-" "ELB-HealthChecker/2.0" "-"
172.31.46.198 - - [03/Nov/2021:19:14:28 +0000] "GET / HTTP/1.1" 404 139 "-" "ELB-HealthChecker/2.0" "-"
172.31.1.181 - - [03/Nov/2021:19:14:28 +0000] "GET / HTTP/1.1" 404 139 "-" "ELB-HealthChecker/2.0" "-"
172.31.30.127 - - [03/Nov/2021:19:14:28 +0000] "GET / HTTP/1.1" 404 139 "-" "ELB-HealthChecker/2.0" "-"
172.31.46.198 - - [03/Nov/2021:19:14:43 +0000] "GET / HTTP/1.1" 404 139 "-" "ELB-HealthChecker/2.0" "-"
172.31.30.127 - - [03/Nov/2021:19:14:43 +0000] "GET / HTTP/1.1" 404 139 "-" "ELB-HealthChecker/2.0" "-"
172.31.1.181 - - [03/Nov/2021:19:14:43 +0000] "GET / HTTP/1.1" 404 139 "-" "ELB-HealthChecker/2.0" "-"
172.31.30.127 - - [03/Nov/2021:19:14:58 +0000] "GET / HTTP/1.1" 404 139 "-" "ELB-HealthChecker/2.0" "-"
172.31.1.181 - - [03/Nov/2021:19:14:58 +0000] "GET / HTTP/1.1" 404 139 "-" "ELB-HealthChecker/2.0" "-"
172.31.46.198 - - [03/Nov/2021:19:14:58 +0000] "GET / HTTP/1.1" 404 139 "-" "ELB-HealthChecker/2.0" "-"
172.31.30.127 - - [03/Nov/2021:19:15:13 +0000] "GET / HTTP/1.1" 404 139 "-" "ELB-HealthChecker/2.0" "-"
Does anyone know how I can fix this, without turning off the monitoring?
Good night people,
I found the problem, I didn't have anything set in my API's root on "/", so EB tried to monitor the api state and took a 404.
I set up a HealthCheck on the root "/" and normalized the 404 errors and integrity issue in the environment.
I'm using Laravel 5.4 and after applying this package to my project and deploying it to my Elastic Beanstalk environment, messages in my sqs stay always in flight. I've done everything that the readme file says...
I followed every step but still got error 500 on POST /worker/queue requests.
Here's my worker log:
127.0.0.1 (-) - - [21/Jun/2017:01:36:59 +0000] "POST /worker/schedule HTTP/1.1" 200 92 "-" "aws-sqsd/2.3"
127.0.0.1 (-) - - [21/Jun/2017:01:37:59 +0000] "POST /worker/schedule HTTP/1.1" 200 92 "-" "aws-sqsd/2.3"
127.0.0.1 (-) - - [21/Jun/2017:01:38:59 +0000] "POST /worker/schedule HTTP/1.1" 200 92 "-" "aws-sqsd/2.3"
127.0.0.1 (-) - - [21/Jun/2017:01:39:59 +0000] "POST /worker/schedule HTTP/1.1" 200 92 "-" "aws-sqsd/2.3"
127.0.0.1 (-) - - [21/Jun/2017:01:39:54 +0000] "POST /worker/queue HTTP/1.1" 500 84128 "-" "aws-sqsd/2.3"
127.0.0.1 (-) - - [21/Jun/2017:01:40:59 +0000] "POST /worker/schedule HTTP/1.1" 200 92 "-" "aws-sqsd/2.3"
127.0.0.1 (-) - - [21/Jun/2017:01:41:13 +0000] "POST /worker/queue HTTP/1.1" 500 84128 "-" "aws-sqsd/2.3"
127.0.0.1 (-) - - [21/Jun/2017:01:41:13 +0000] "POST /worker/queue HTTP/1.1" 500 84128 "-" "aws-sqsd/2.3"
127.0.0.1 (-) - - [21/Jun/2017:01:41:13 +0000] "POST /worker/queue HTTP/1.1" 500 84128 "-" "aws-sqsd/2.3"
127.0.0.1 (-) - - [21/Jun/2017:01:41:13 +0000] "POST /worker/queue HTTP/1.1" 500 84128 "-" "aws-sqsd/2.3"
127.0.0.1 (-) - - [21/Jun/2017:01:41:13 +0000] "POST /worker/queue HTTP/1.1" 500 84128 "-" "aws-sqsd/2.3"
I have also been experiencing similar issues, and while I would need you to post some code, and more details as to what your doing, I can describe the steps I took to resolve my own issues.
You will need to be able to log into your instance in order to do some faster troubleshooting, so I suggest you set up eb ssh on your machines, or if you are using a VPC, create a Bastion host, to use to connect to the internal EC2s. Log into the app environment, go to /var/www/html, run php artisan tinker, and manually dispatch a job. If you are successful you will get a response with the id of the queued message, if not you will receive an error output, which you can further use to troubleshoot the issue. I also suggest checking whether the app is picking up the environmental values for the queues, so check whether the queue you set in the deployment configuration, matches the one your app is trying to send the request to.
If the message is successfully being sent to the queue, but you still have failing jobs, I suggest you SSH into the worker environment EC2 machine, and try to dispatch a job from there as well. In my personal scenario the worker environment was in the wrong security group, so it didn't have access to the database, resulting in a 500 error, due to an internal server issue.
Instructions on creating bastion hosts:
https://vaughanj10.github.io/creating-a-bastion-host-for-aws/
I use AWS Elastic Beanstalk worker environment with SQS and cronjobs to do what I want.
But sometimes, my environment bug and stop to work (it needs to be restarted manually) because it received some unknown requests (not send by me of course) :
196.52.43.55 (-) - - [09/Jun/2017:00:33:11 +0000] "GET / HTTP/1.1" 400 226 "-" "-"
81.196.3.208 (-) - - [09/Jun/2017:01:45:30 +0000] "GET / HTTP/1.0" 200 4576 "-" "-"
195.154.214.162 (-) - - [09/Jun/2017:03:43:21 +0000] "GET //recordings/modules/phonefeatures.module HTTP/1.1" 404 471 "-" "python-requests/2.6.0 CPython/2.6.6 Linux/2.6.32-696.1.1.el6.x86_64"
195.154.214.162 (-) - - [09/Jun/2017:04:54:27 +0000] "GET //recordings/modules/phonefeatures.module HTTP/1.1" 404 471 "-" "python-requests/2.6.0 CPython/2.6.6 Linux/2.6.32-696.1.1.el6.x86_64"
Example of cron job I executed every minute
127.0.0.1 (-) - - [09/Jun/2017:00:14:59 +0000] "POST /workers/cron/search/detailsHTTP/1.1" 200 - "-" "aws-sqsd/2.3"
127.0.0.1 (-) - - [09/Jun/2017:00:14:59 +0000] "POST /workers/cron/positions HTTP/1.1" 200 60 "-" "aws-sqsd/2.3"
127.0.0.1 (-) - - [09/Jun/2017:00:15:01 +0000] "POST /queue/received HTTP/1.1" 200 10 "-" "aws-sqsd/2.3"
Do you have a solution for me? Do I need to change my VPC and/or EC2 group security?
My architecture is one Elasticsearch Application and one Elasticsearch Worker.
Thank you very much
While installing the certificate of the WSO2 agent I am getting the following error "Cannot Install Profile-Safari could not install a profile due to an unknown error".
The logs which I am getting are
<ip>- - [20/Jun/2016:16:47:54 +0530] "GET /ios-enrollment/ca HTTP/1.1" 302 - "-" "Jakarta Commons-HttpClient/3.1"
<ip>- - [20/Jun/2016:16:47:54 +0530] "GET /carbon/admin/login.jsp HTTP/1.1" 200 15541 "-" "Jakarta Commons-HttpClient/3.1"
172.17.242.31 - - [20/Jun/2016:16:47:54 +0530] "GET /emm-web-agent/enrollment/ios/download-certificate HTTP/1.1" 200 15541 "https://<ip>:9443/emm-web-agent/enrollments/ios/download-agent" "Mozilla/5.0 (iPad; CPU OS 7_0_2 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A501 Safari/9537.53"
I have successfully got the credentials on my mail id for enrolling my device through WSO2 EMM. But when i am signing in using the mail credentials its giving an error which "Enrollment failed -Please contact administrator "
The logs were:
Chrome/44.0.2403.133 Mobile Safari/537.36"
172.17.29.121 - - [16/Jun/2016:12:34:53 +0530] "GET /emm-web-agent/public/asset-download-agent-android/asset/android-agent.apk HTTP/1.1" 200 2896941 "-" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1)"
172.17.29.121 - - [16/Jun/2016:12:35:32 +0530] "GET /emm-web-agent/public/asset-download-agent-android/asset/android-agent.apk HTTP/1.1" 200 590411 "-" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1)"
172.17.29.121 - - [16/Jun/2016:12:40:13 +0530] "GET /emm-web-agent/public/asset-download-agent-android/asset/android-agent.apk HTTP/1.1" 200 590411 "-" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1)
172.17.29.121 - - [16/Jun/2016:12:42:45 +0530] "POST /dynamic-client-web/register HTTP/1.1" 201 148 "-" "Mozilla/5.0 ( compatible ), Android"
172.17.29.121 - - [16/Jun/2016:12:42:45 +0530] "POST /oauth2/token HTTP/1.1" 200 160 "-" "Apache-HttpClient/UNAVAILABLE (java 1.4)"
172.17.29.121 - - [16/Jun/2016:12:42:45 +0530] "GET /mdm-android-agent/device/license HTTP/1.1" 401 23 "-" "Mozilla/5.0 ( compatible ), Android"
According to the conversation the problem is with the permission you have provided to the given user role.
You can use an existing role with device management permission as sashika has suggested.
There is a login permission as the very last permission entry in the permission management UI, please include that permission to the related role in order to overcome the situation.
You need to add permissions to the role - specifically the "enroll" role.