Intro
I have a worker environment setup with AWS Elastic Beantstalk. It is using the Apache proxy on a Linux 2 AMI.
My worker environment occasionaly has to handle longer running tasks (around 120s).
Problem
I've noticed longer lasting tasks get resend to the SQS queue and thus are handled more then once.
I've already taken care of the Visibility timeout and Inactivity timeout so that this would not happen, but the issue persists.
After checking my logs at /var/log/httpd/access_log I noticed 504 errors on exaclty 60 seconds, telling me I somewhere hit another timeout.
Hacky solution
I was able to solve it by increasing the httpd Timeout from 60s to 120s by manually changing the value in /etc/httpd/conf/httpd.conf and restarting the httpd service.
The issue still comes back each time autoscaling happens or the server is restarted. The value gets reset to 60 seconds.
What is needed is a way to set the timeout through configuration, bringing me to me actual question: How can i configure the httpd timeout setting with Elastic Beanstalk in a maintable way?
This article brought me so far, but the solution provided (using .ebextensions) seems to be for the deprecated Linux AMI (using linux 2).
What I've tried
I've read op on configuring the proxy here.
I've tried all the following configurations, but it appears the configuration isn't being picked up at all.
Initial try:
~/workspace/my-app/
|-- .ebextensions
| -- ....
|-- .platform
| `-- httpd
| `-- conf.d
| `-- timeout.conf
`-- index.php
.platform/httpd/conf.d/timeout.conf
Timeout 120
Added 00_reload.config
based of this.
~/workspace/my-app/
|-- .ebextensions
| -- ....
|-- .platform
| `-- httpd
| `-- conf.d
| `-- timeout.conf
| `-- 00_reload.config
`-- index.php
.platform/httpd/00_reload.config
container_commands:
01_reload_httpd:
command: "service httpd reload"
Overwrite the default apache config completely
if possible I would not like to do this as it becomes a maintenance nightmare.
~/workspace/my-app/
|-- .ebextensions
| -- .....
|-- .platform
| `-- httpd
| `-- conf
| `-- httpd.conf
`-- index.php
.platform/httpd/conf/httpd.conf
It's a complete copy of the default provided by AWS, but changed the Timout setting to 120.
# Managed by Elastic Beanstalk
PidFile /run/httpd.pid
Listen 80
User apache
Group apache
LogLevel warn
TraceEnable off
EnableSendfile on
AddDefaultCharset UTF-8
# Enable TCP keepalive
Timeout 120
KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 60
Include conf.modules.d/*.conf
.....
It appears beanstalk assumes the .platform folder to be at the absolute root of your project. Mine was located inside my document root: /public/.platform/
The confusing started because the documentation showed it to be at the same level as the .ebextensions folder, which does need to be inside your document root (/public for me). There's also no mention about this anywhere.
Once I figured that out, the documentation of AWS was working perfectly. My final config below.
~/workspace/my-app/
|-- public
| -- .ebextensions
| -- index.php
|-- .platform
| `-- httpd
| `-- conf.d
| `-- increase_timeout.conf
.platform/httpd/conf.d/timeout.conf
Timeout 120
I hope my struggles can be of help to someone down the line.
Related
I have a golang web server running on AWS elastic beanstalk in this machine: Go 1 running on 64bit Amazon Linux 2;
The error occurs when I make a request to the server and it executes several processes, so it takes around 90 seconds to complete the entire process, so after 60 seconds the AWS beanstalk server ends the client connection and returns the following error, although the process ends completely after 90 seconds.
It sends the following error:
<html>
<head>
<title>504 Gateway Time-out</title>
</head>
<body>
<center>
<h1>504 Gateway Time-out</h1>
</center>
</body>
</html>
Source application files:
.
├── application.go
├── cron.yaml
└── public
└── index.html
When I tested it on my local machine It works well and It take 90 seconds, only when It runs on Beanstalk has this issue.
How can I fix it ??
504 Gateway Timeout indicates that the nginx proxy is waiting too long for a response from the upstream app. If this happens for an endpoint that usually returns after a few seconds, it is very likely that the nginx proxy is trying to reach a port that your app is not listening on (or that the app has crashed).
By default, Elastic Beanstalk configures the nginx proxy to forward requests to your application on port 5000. You can override the default port by setting the PORT environment property to the port on which your main application listens. More info: AWS Reverse Proxy Docs
Make sure that your application code listens on the right port.
http.ListenAndServe(":5000", nil);
Another reason for this could be a crash in the app code. Check the last 100 log lines in Elastic Beanstalk. You can also retrieve the logs by SSH'ing into your server and running cat /var/log/eb-engine.log.
Reverse Proxy
On a separate note, it looks like you're trying to statically host /public/index.html. You can do this in 2 separate ways.
Using Go (not recommended)
http.Handle("/", http.FileServer(http.Dir("./static")))
http.ListenAndServe(":3000", nil)
Using Nginx (recommended)
By extending your nginx config you can use the proxy server to statically host files and redirect routes before they hit your server application.
~/workspace/my-app/
|-- .platform
| `-- nginx
| `-- conf.d
| `-- myconf.conf
This question already has answers here:
AWS Elastic Beanstalk chown PythonPath error
(2 answers)
Closed 10 months ago.
Following AWS instructions for (https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/create-deploy-python-django.html)
Currently trying to run "eb deploy" with the following output:
Creating application version archive "app-220413_141345835306".
Uploading: [##################################################] 100% Done...
2022-04-13 20:14:00 INFO Environment update is starting.
2022-04-13 20:14:05 INFO Deploying new version to instance(s).
2022-04-13 20:14:09 ERROR Instance deployment failed. For details, see 'eb-engine.log'.
2022-04-13 20:14:11 ERROR [Instance: i-0fb1ab203cc59f0d1] Command failed on instance. Return code: 1 Output: Engine execution has encountered an error..
2022-04-13 20:14:11 INFO Command execution completed on all instances. Summary: [Successful: 0, Failed: 1].
2022-04-13 20:14:11 ERROR Unsuccessful command execution on instance id(s) 'i-0fb1ab203cc59f0d1'. Aborting the operation.
2022-04-13 20:14:11 ERROR Failed to deploy application.
ERROR: ServiceError - Failed to deploy application.
In the logs I see:
2022/04/13 20:00:19.230141 [INFO] finished extracting /opt/elasticbeanstalk/deployment/app_source_bundle to /var/app/staging/ successfully
2022/04/13 20:00:19.231105 [ERROR] An error occurred during execution of command [app-deploy] - [StageApplication]. Stop running the command. Erro
r: chown /var/app/staging/venv/bin/python: no such file or directory
Any ideas what to do? I have no idea where that path is coming from
Th second answer here worked for me: AWS Elastic Beanstalk chown PythonPath error
I added a file caled .ebignore into my main project directory:
~/ebdjango/
|-- .ebignore
|-- .ebextensions
| `-- django.config
|-- ebdjango
| |-- __init__.py
| |-- settings.py
| |-- urls.py
| `-- wsgi.py
|-- db.sqlite3
|-- manage.py
`-- requirements.txt
the contents of the file are only one line:
venv
I have the same issue as 404
and the answers is correct and when I add
location / {
try_files $uri $uri/ /index.php?$query_string;
gzip_static on;
}
The problem is solved.
How can I do such a thing in .ebextensions?
To customize nginx on Amazon Linux 2 you should use .platform/nginx/conf.d/, not .ebextensions. The .ebextensions` were used for Amazon Linux 1.
The AWS docs provide an example of how to add configuration options to nginx:
~/workspace/my-app/
|-- .platform
| `-- nginx
| `-- conf.d
| `-- myconf.conf
I want to config the nginx config with the ebextension at Amazon Elastic Beanstalk,
The content of the conf as below:
files:
"/etc/nginx/conf.d/custom-nginx-proxy.conf" :
mode: "000755"
owner: "root"
group: "root"
content: |
client_max_body_size 60M;
contatiner_commands:
reload_nginx:
command: "sudo service nginx reload"
but always got the error about the content,
nginx: [emerg] unknown directive "files:" in /var/elasticbeanstalk/staging/nginx/conf.d/custom-nginx-proxy.conf:7
I put this file at
ROOT.war
|- ROOT/
|- .ebextensions
|- nginx
|- conf.d
|- custom-nginx-proxy.conf
|- Others content
Your extension should be placed at .ebextensions/01-custom-nginx-proxy.config. Also, the second block should start with container_commands:, not contatiner_commands:, and you don't need sudo as the deployment will already be running as root. If that doesn't help, try using two spaces per indent level. EB extensions are written in YAML, which is very sensitive to whitespace.
If you're still stuck, please post the contents of /var/elasticbeanstalk/staging/nginx/conf.d/custom-nginx-proxy.conf - the file that gets created after your deployment.
I have been struggling for the past few days now to increase the upload limit on nginx, which is apart my elastic beanstalk instance. From my understanding i need an ebextensions file to set the client_max_body_size. Which I have tried several different ways to configure, like directly overwriting nginx.conf to inserting client_max_body_size in nginx.conf on the live server. None of these methods have worked. I was sure to manually reload the file just after each attempt. As it stands, this is what my config file looks like at the moment(after several iteration):
.ebextensions/01_nginx.config
files:
/etc/nginx/conf.d/proxy.conf:
content: |
client_max_body_size 2G;
If anyone can help me that would be awesome.
You can either extend it or override it.
Elastic Beanstalk provides a default nginx configuration that you can
either extend or override completely with your own configuration
Overriding
~/workspace/my-app/
|-- .ebextensions
| `-- nginx
| `-- conf.d
| `-- myconf.conf
Source
Your Config looks right from your question compared to the example from the docs. Verify location of your nginx conf by sshing into the
$ eb ssh environment-name
$ sudo service nginx configtest
nginx: the configuration file /${LOCATION}/nginx.conf syntax is ok
nginx: configuration file /${LOCATION}/nginx.conf test is successful
Doc Example
"/home/ec2-user/myfile2" :
content: |
# this is my file
# with content
Also try adding the nginx reload to the .ebextensions if you have the right path to your nginx config.
files:
/etc/nginx/conf.d/proxy.conf:
mode: "000644"
owner: root
group: root
content: |
client_max_body_size 2G;
commands:
01_reload_nginx:
command: "sudo service nginx reload"
Source
Helpful blog post that runs thru most of this: "Getting to Know and Love AWS Elastic Beanstalk Configuration Files"(.ebextensions)
This might be helpful as it worked for me. Taking reference form aws ebs docs
To extend the Elastic Beanstalk default nginx configuration, add .conf configuration files to a folder named .platform/nginx/conf.d/ in your application source bundle. The Elastic Beanstalk nginx configuration includes .conf files in this folder automatically.
~/workspace/my-app/
|-- .platform
| `-- nginx
| `-- conf.d
| `-- myconf.conf
`-- other source files
myconf.conf
client_max_body_size 50M;
Note
When you add or edit an nginx .conf configuration file, be sure to encode it as UTF-8.