Elastic Beanstalk Laravel 5.4 Worker HTTP 500 - amazon-web-services

I'm using Laravel 5.4 and after applying this package to my project and deploying it to my Elastic Beanstalk environment, messages in my sqs stay always in flight. I've done everything that the readme file says...
I followed every step but still got error 500 on POST /worker/queue requests.
Here's my worker log:
127.0.0.1 (-) - - [21/Jun/2017:01:36:59 +0000] "POST /worker/schedule HTTP/1.1" 200 92 "-" "aws-sqsd/2.3"
127.0.0.1 (-) - - [21/Jun/2017:01:37:59 +0000] "POST /worker/schedule HTTP/1.1" 200 92 "-" "aws-sqsd/2.3"
127.0.0.1 (-) - - [21/Jun/2017:01:38:59 +0000] "POST /worker/schedule HTTP/1.1" 200 92 "-" "aws-sqsd/2.3"
127.0.0.1 (-) - - [21/Jun/2017:01:39:59 +0000] "POST /worker/schedule HTTP/1.1" 200 92 "-" "aws-sqsd/2.3"
127.0.0.1 (-) - - [21/Jun/2017:01:39:54 +0000] "POST /worker/queue HTTP/1.1" 500 84128 "-" "aws-sqsd/2.3"
127.0.0.1 (-) - - [21/Jun/2017:01:40:59 +0000] "POST /worker/schedule HTTP/1.1" 200 92 "-" "aws-sqsd/2.3"
127.0.0.1 (-) - - [21/Jun/2017:01:41:13 +0000] "POST /worker/queue HTTP/1.1" 500 84128 "-" "aws-sqsd/2.3"
127.0.0.1 (-) - - [21/Jun/2017:01:41:13 +0000] "POST /worker/queue HTTP/1.1" 500 84128 "-" "aws-sqsd/2.3"
127.0.0.1 (-) - - [21/Jun/2017:01:41:13 +0000] "POST /worker/queue HTTP/1.1" 500 84128 "-" "aws-sqsd/2.3"
127.0.0.1 (-) - - [21/Jun/2017:01:41:13 +0000] "POST /worker/queue HTTP/1.1" 500 84128 "-" "aws-sqsd/2.3"
127.0.0.1 (-) - - [21/Jun/2017:01:41:13 +0000] "POST /worker/queue HTTP/1.1" 500 84128 "-" "aws-sqsd/2.3"

I have also been experiencing similar issues, and while I would need you to post some code, and more details as to what your doing, I can describe the steps I took to resolve my own issues.
You will need to be able to log into your instance in order to do some faster troubleshooting, so I suggest you set up eb ssh on your machines, or if you are using a VPC, create a Bastion host, to use to connect to the internal EC2s. Log into the app environment, go to /var/www/html, run php artisan tinker, and manually dispatch a job. If you are successful you will get a response with the id of the queued message, if not you will receive an error output, which you can further use to troubleshoot the issue. I also suggest checking whether the app is picking up the environmental values for the queues, so check whether the queue you set in the deployment configuration, matches the one your app is trying to send the request to.
If the message is successfully being sent to the queue, but you still have failing jobs, I suggest you SSH into the worker environment EC2 machine, and try to dispatch a job from there as well. In my personal scenario the worker environment was in the wrong security group, so it didn't have access to the database, resulting in a 500 error, due to an internal server issue.
Instructions on creating bastion hosts:
https://vaughanj10.github.io/creating-a-bastion-host-for-aws/

Related

High number of aws target group health checks

I have an application load balancer with several registered target groups (and 6 availability zones in case it is important to mention).
There is one ec2 instance which is the registered target for all target groups. On the ec2 instance there is an nginx running.
For each target group I defined a health check with a custom url and with an interval of 60 seconds.
When I look at the nginx logs I expect to see the health check url for a particular target group every 60 seconds. But to my surprise I see that in 60 seconds there are groups of 8 calls like this:
172.31.25.32 - - [14/Feb/2022:16:00:29 +0000] "GET /path/target-group-X/ HTTP/1.1" 200 4 "-" "ELB-HealthChecker/2.0" rt=0.118 uct="0.000" uht="0.120" urt="0.120"
172.31.89.13 - - [14/Feb/2022:16:00:35 +0000] "GET /path/target-group-X/ HTTP/1.1" 200 4 "-" "ELB-HealthChecker/2.0" rt=0.080 uct="0.000" uht="0.080" urt="0.080"
172.31.75.210 - - [14/Feb/2022:16:00:43 +0000] "GET /path/target-group-X/ HTTP/1.1" 200 4 "-" "ELB-HealthChecker/2.0" rt=0.050 uct="0.000" uht="0.052" urt="0.052"
172.31.88.219 - - [14/Feb/2022:16:00:44 +0000] "GET /path/target-group-X/ HTTP/1.1" 200 4 "-" "ELB-HealthChecker/2.0" rt=0.059 uct="0.000" uht="0.060" urt="0.060"
172.31.9.236 - - [14/Feb/2022:16:00:51 +0000] "GET /path/target-group-X/ HTTP/1.1" 200 4 "-" "ELB-HealthChecker/2.0" rt=0.059 uct="0.000" uht="0.060" urt="0.060"
172.31.15.138 - - [14/Feb/2022:16:01:02 +0000] "GET /path/target-group-X/ HTTP/1.1" 200 4 "-" "ELB-HealthChecker/2.0" rt=0.010 uct="0.000" uht="0.008" urt="0.008"
172.31.49.23 - - [14/Feb/2022:16:01:07 +0000] "GET /path/target-group-X/ HTTP/1.1" 200 4 "-" "ELB-HealthChecker/2.0" rt=0.062 uct="0.000" uht="0.064" urt="0.064"
172.31.47.189 - - [14/Feb/2022:16:01:13 +0000] "GET /path/target-group-X/ HTTP/1.1" 200 4 "-" "ELB-HealthChecker/2.0" rt=0.094 uct="0.000" uht="0.092" urt="0.092"
172.31.25.32 - - [14/Feb/2022:16:01:29 +0000] "GET /path/target-group-X/ HTTP/1.1" 200 4 "-" "ELB-HealthChecker/2.0" rt=0.050 uct="0.000" uht="0.048" urt="0.048"
172.31.89.13 - - [14/Feb/2022:16:01:35 +0000] "GET /path/target-group-X/ HTTP/1.1" 200 4 "-" "ELB-HealthChecker/2.0" rt=0.049 uct="0.000" uht="0.048" urt="0.048"
172.31.75.210 - - [14/Feb/2022:16:01:43 +0000] "GET /path/target-group-X/ HTTP/1.1" 200 4 "-" "ELB-HealthChecker/2.0" rt=0.280 uct="0.000" uht="0.280" urt="0.280"
172.31.88.219 - - [14/Feb/2022:16:01:44 +0000] "GET /path/target-group-X/ HTTP/1.1" 200 4 "-" "ELB-HealthChecker/2.0" rt=0.050 uct="0.000" uht="0.048" urt="0.048"
172.31.9.236 - - [14/Feb/2022:16:01:52 +0000] "GET /path/target-group-X/ HTTP/1.1" 200 4 "-" "ELB-HealthChecker/2.0" rt=0.508 uct="0.000" uht="0.508" urt="0.508"
172.31.15.138 - - [14/Feb/2022:16:02:02 +0000] "GET /path/target-group-X/ HTTP/1.1" 200 4 "-" "ELB-HealthChecker/2.0" rt=0.176 uct="0.000" uht="0.172" urt="0.172"
172.31.49.23 - - [14/Feb/2022:16:02:07 +0000] "GET /path/target-group-X/ HTTP/1.1" 200 4 "-" "ELB-HealthChecker/2.0" rt=0.061 uct="0.000" uht="0.060" urt="0.060"
172.31.47.189 - - [14/Feb/2022:16:02:13 +0000] "GET /path/target-group-X/ HTTP/1.1" 200 4 "-" "ELB-HealthChecker/2.0" rt=0.057 uct="0.000" uht="0.056" urt="0.056"
There are 8 different local IP-s from which the calls are coming. If I take each such IP separately (e.g. 172.31.25.32), then indeed the health checks calls from that IP are arriving after exactly 60 seconds. But what is about the other calls? Why are so many?
I think at a minimum the target group is going to do a health check from each availability zone, or maybe each VPC subnet. You can probably map those IPs back to specific subnets in your VPC.
It definitely seems excessive, but you have to realize that behind the scenes a multi-az load balancer is really multiple servers, and each one is doing its own health check against your target server(s).

Error 4xx AWS Elastic Beanstalk - Severe integrity

Good afternoon people,
I created an environment in Elastic Beanstalk and uploaded a NODEjs application an api with express.
She's working fine, all right.
But the integrity of the environment is reported as serious, and this monitoring attempt appears in the logs.
----------------------------------------
/var/log/nginx/access.log
----------------------------------------
172.31.46.198 - - [03/Nov/2021:19:14:13 +0000] "GET / HTTP/1.1" 404 139 "-" "ELB-HealthChecker/2.0" "-"
172.31.1.181 - - [03/Nov/2021:19:14:13 +0000] "GET / HTTP/1.1" 404 139 "-" "ELB-HealthChecker/2.0" "-"
172.31.30.127 - - [03/Nov/2021:19:14:13 +0000] "GET / HTTP/1.1" 404 139 "-" "ELB-HealthChecker/2.0" "-"
172.31.46.198 - - [03/Nov/2021:19:14:28 +0000] "GET / HTTP/1.1" 404 139 "-" "ELB-HealthChecker/2.0" "-"
172.31.1.181 - - [03/Nov/2021:19:14:28 +0000] "GET / HTTP/1.1" 404 139 "-" "ELB-HealthChecker/2.0" "-"
172.31.30.127 - - [03/Nov/2021:19:14:28 +0000] "GET / HTTP/1.1" 404 139 "-" "ELB-HealthChecker/2.0" "-"
172.31.46.198 - - [03/Nov/2021:19:14:43 +0000] "GET / HTTP/1.1" 404 139 "-" "ELB-HealthChecker/2.0" "-"
172.31.30.127 - - [03/Nov/2021:19:14:43 +0000] "GET / HTTP/1.1" 404 139 "-" "ELB-HealthChecker/2.0" "-"
172.31.1.181 - - [03/Nov/2021:19:14:43 +0000] "GET / HTTP/1.1" 404 139 "-" "ELB-HealthChecker/2.0" "-"
172.31.30.127 - - [03/Nov/2021:19:14:58 +0000] "GET / HTTP/1.1" 404 139 "-" "ELB-HealthChecker/2.0" "-"
172.31.1.181 - - [03/Nov/2021:19:14:58 +0000] "GET / HTTP/1.1" 404 139 "-" "ELB-HealthChecker/2.0" "-"
172.31.46.198 - - [03/Nov/2021:19:14:58 +0000] "GET / HTTP/1.1" 404 139 "-" "ELB-HealthChecker/2.0" "-"
172.31.30.127 - - [03/Nov/2021:19:15:13 +0000] "GET / HTTP/1.1" 404 139 "-" "ELB-HealthChecker/2.0" "-"
Does anyone know how I can fix this, without turning off the monitoring?
Good night people,
I found the problem, I didn't have anything set in my API's root on "/", so EB tried to monitor the api state and took a 404.
I set up a HealthCheck on the root "/" and normalized the 404 errors and integrity issue in the environment.

Suddenly I am getting 502 on an End point, rest are working fine

I am serving django project with gunicorn, It running fine but after some time on one specific endpoint start giving 502. Other api end point still okay and giving proper response.
I already tried with gunicorn service settings
Current setting
ExecStart=/var/virtualenv/d/bin/gunicorn --workers 5 proj.wsgi:application -b :9008 --threads 8 -k gthread --timeout 120
47.247.243.245 - - [14/Jun/2019:15:02:11 +0000] "GET /api/user/1263/league/81/ HTTP/1.1" 200 291 "-" "okhttp/3.12.1"
157.32.16.35 - - [14/Jun/2019:15:02:11 +0000] "GET /api/v1/xyc-leaderboard/?contest=4973 HTTP/1.1" 200 43714 "-" "okhttp/3.12.1""
157.32.16.35 - - [14/Jun/2019:15:02:16 +0000] "GET /api/v1/xy/xy-team/15543 HTTP/1.1" 301 5 "-" "okhttp/3.12.1"
157.32.16.35 - - [14/Jun/2019:15:02:16 +0000] "GET /api/v1/xy/xy-team/15543/ HTTP/1.1" 502 5517 "-" "okhttp/3.12.1"
171.76.167.124 - - [14/Jun/2019:14:39:56 +0000] "GET /api/v1/xy/xy-team/15343/ HTTP/1.1" 502 182 "-" "okhttp/3.12.1"

AWS elasticbeanstalk worker received not allowed requests and stop to work. It need to be restart manually

I use AWS Elastic Beanstalk worker environment with SQS and cronjobs to do what I want.
But sometimes, my environment bug and stop to work (it needs to be restarted manually) because it received some unknown requests (not send by me of course) :
196.52.43.55 (-) - - [09/Jun/2017:00:33:11 +0000] "GET / HTTP/1.1" 400 226 "-" "-"
81.196.3.208 (-) - - [09/Jun/2017:01:45:30 +0000] "GET / HTTP/1.0" 200 4576 "-" "-"
195.154.214.162 (-) - - [09/Jun/2017:03:43:21 +0000] "GET //recordings/modules/phonefeatures.module HTTP/1.1" 404 471 "-" "python-requests/2.6.0 CPython/2.6.6 Linux/2.6.32-696.1.1.el6.x86_64"
195.154.214.162 (-) - - [09/Jun/2017:04:54:27 +0000] "GET //recordings/modules/phonefeatures.module HTTP/1.1" 404 471 "-" "python-requests/2.6.0 CPython/2.6.6 Linux/2.6.32-696.1.1.el6.x86_64"
Example of cron job I executed every minute
127.0.0.1 (-) - - [09/Jun/2017:00:14:59 +0000] "POST /workers/cron/search/detailsHTTP/1.1" 200 - "-" "aws-sqsd/2.3"
127.0.0.1 (-) - - [09/Jun/2017:00:14:59 +0000] "POST /workers/cron/positions HTTP/1.1" 200 60 "-" "aws-sqsd/2.3"
127.0.0.1 (-) - - [09/Jun/2017:00:15:01 +0000] "POST /queue/received HTTP/1.1" 200 10 "-" "aws-sqsd/2.3"
Do you have a solution for me? Do I need to change my VPC and/or EC2 group security?
My architecture is one Elasticsearch Application and one Elasticsearch Worker.
Thank you very much

AWS elastic beanstalk cannot return custom response code using resteasy?

I'm working on a web service using RESTEASY to set the response status code when get some exception.
First I tried resteasy exception mapper which works fine locally. The mapper code attached below. However, when I upload that WS into elastic beanstalk, that always return 500 (internal server error).
#Provider
public class LoadGridTileFailedExceptionMapper extends BaseExceptionMapper implements ExceptionMapper<LoadGridTileFailedException>
{
#Override
public Response toResponse(LoadGridTileFailedException e)
{
log(e.getMessage(), e);
return printMsg(e.getMessage(), DtmWebServiceReturnStatus.LOAD_GRID_TILE_FAILED_EXCEPTION_CODE);
}
}
Then I try just throw exception WebApplicationException(ex, DtmWebServiceReturnStatus.LOAD_GRID_TILE_FAILED_EXCEPTION_CODE) to get around exception mapping. The result is that I got a response status 498(LOAD_GRID_TILE_FAILED_EXCEPTION_CODE) wrapped in status code 500.
Apache Tomcat/7.0.27 - Error report HTTP Status 498 - type Status reportmessage description http.498Apache Tomcat/7.0.27
It seems that elastic beanstalk wrapped all exceptions throw out in the server side with status code 500?The question is how can I get around that feature and return the status code I set in response? Thank you.
UPDATE
Try more requests this morning and find something interesting:
Get the right return status in elastic beanstalk log snapshot
/var/log/tomcat7/localhost_access_log.txt
127.0.0.1 - - [09/Jan/2013:15:06:28 +0000] "GET /published/tile/003331330031 HTTP/1.1" 498 22
127.0.0.1 - - [09/Jan/2013:15:06:31 +0000] "GET /published/tile/003331330031 HTTP/1.1" 498 22
127.0.0.1 - - [09/Jan/2013:15:06:34 +0000] "GET /published/tile/003331330031 HTTP/1.1" 498 22
127.0.0.1 - - [09/Jan/2013:15:06:37 +0000] "GET /published/tile/003331330031 HTTP/1.1" 498 22
127.0.0.1 - - [09/Jan/2013:15:06:39 +0000] "GET /published/tile/003331330031 HTTP/1.1" 498 22
127.0.0.1 - - [09/Jan/2013:15:06:41 +0000] "GET /published/tile/003331330031 HTTP/1.1" 498 22
127.0.0.1 - - [09/Jan/2013:15:06:44 +0000] "GET /published/tile/003331330031 HTTP/1.1" 498 22
127.0.0.1 - - [09/Jan/2013:15:06:48 +0000] "GET /published/tile/003331330031 HTTP/1.1" 498 22
127.0.0.1 - - [09/Jan/2013:15:06:51 +0000] "GET /published/tile/003331330031 HTTP/1.1" 498 22
127.0.0.1 - - [09/Jan/2013:15:06:54 +0000] "GET /published/tile/003331330031 HTTP/1.1" 498 22
127.0.0.1 - - [09/Jan/2013:15:06:57 +0000] "GET /published/tile/003331330031 HTTP/1.1" 498 22
/var/log/httpd/elasticbeanstalk-access_log
10.28.215.233 (65.167.11.254, 10.28.215.233) - - [09/Jan/2013:15:06:28 +0000] "GET /published/tile/003331330031 HTTP/1.1" 498 22 "-" "-"
10.28.215.233 (65.167.11.254, 10.28.215.233) - - [09/Jan/2013:15:06:31 +0000] "GET /published/tile/003331330031 HTTP/1.1" 498 22 "-" "-"
10.28.215.233 (65.167.11.254, 10.28.215.233) - - [09/Jan/2013:15:06:34 +0000] "GET /published/tile/003331330031 HTTP/1.1" 498 22 "-" "-"
10.28.215.233 (65.167.11.254, 10.28.215.233) - - [09/Jan/2013:15:06:37 +0000] "GET /published/tile/003331330031 HTTP/1.1" 498 22 "-" "-"
10.28.215.233 (65.167.11.254, 10.28.215.233) - - [09/Jan/2013:15:06:39 +0000] "GET /published/tile/003331330031 HTTP/1.1" 498 22 "-" "-"
10.28.215.233 (65.167.11.254, 10.28.215.233) - - [09/Jan/2013:15:06:41 +0000] "GET /published/tile/003331330031 HTTP/1.1" 498 22 "-" "-"
10.28.215.233 (65.167.11.254, 10.28.215.233) - - [09/Jan/2013:15:06:44 +0000] "GET /published/tile/003331330031 HTTP/1.1" 498 22 "-" "-"
10.28.215.233 (65.167.11.254, 10.28.215.233) - - [09/Jan/2013:15:06:48 +0000] "GET /published/tile/003331330031 HTTP/1.1" 498 22 "-" "-"
10.28.215.233 (65.167.11.254, 10.28.215.233) - - [09/Jan/2013:15:06:51 +0000] "GET /published/tile/003331330031 HTTP/1.1" 498 22 "-" "-"
10.28.215.233 (65.167.11.254, 10.28.215.233) - - [09/Jan/2013:15:06:54 +0000] "GET /published/tile/003331330031 HTTP/1.1" 498 22 "-" "-"
10.28.215.233 (65.167.11.254, 10.28.215.233) - - [09/Jan/2013:15:06:57 +0000] "GET /published/tile/003331330031 HTTP/1.1" 498 22 "-" "-"
However in client side, still got 500 :-(
printMsg method:
protected Response printMsg(String msg, int intStatus)
{
// Need this due to the Resteasy bug
ServiceDataCollector.processRequest(true);
ResponseBuilder builder = Response.status(intStatus);
builder.type("text/plain");
builder.entity("ERROR: " + msg);
Response rep = builder.build();
LOG.error(rep.getStatus() + ":" + rep.toString());
return rep;
}
Some one help me to work the problem out. I had the httpd deployed in my AMI before tomcat server at 80. So the load balancer will interact with httpd server, which change the status code from tomcat to 500. Disable that httpd server will solve the problem. Thx for everyone's help.