HTTP 500 Deploying Elixir/Phoenix to AWS Elastic Beanstalk - amazon-web-services

Question

I'm having trouble with the elixir/phoenix config that I need for a deployment to AWS/Elastic Beanstalk. (Following the guide found here: https://thoughtbot.com/blog/deploying-elixir-to-aws-elastic-beanstalk-with-docker - my Dockerfile looks similar except for updated libraries).
I can run in eb local run, but am having trouble pushing to production.
However, when I try and deploy to EB, I get the following warning, and it crashes:
Environment health has transitioned from Degraded to Severe.
100.0 % of the requests are failing with HTTP 5xx.
Command failed on all instances.
Incorrect application version found on all instances. Expected version "app-8412-171116_115503" (deployment 5).
ELB processes are not healthy on all instances.
100.0 % of the requests to the ELB are erroring with HTTP 4xx.
Insufficient request rate (0.5 requests/min) to determine application health (5 minutes ago).
ELB health is failing or not available for all instances.
I was wondering if someone could let me know if my configs look right.
I've been trying a bunch of things, but I think I've gotten confused, as I'm just guessing at this point.
config.exs
use Mix.Config
config :newsly,
ecto_repos: [Newsly.Repo]
config :logger, :console,
format: "$time $metadata[$level] $message\n",
metadata: [:request_id]
import_config "#{Mix.env}.exs"
prod.exs
use Mix.Config
config :logger, :console, format: "[$level] $message\n"
config :phoenix, :stacktrace_depth, 5
import_config "prod.secret.exs"
prod.secret.exs
use Mix.Config
config :ex_aws,
access_key_id: System.get_env("AWS_ACCESS_KEY_ID"),
secret_access_key: System.get_env("AWS_SECRET_ACCESS_KEY"),
bucket_name: System.get_env("BUCKET_NAME"),
s3: [
scheme: "https://",
host: System.get_env("BUCKET_NAME"),
region: "us-west-2"
]
config :newsly, Newsly.Repo,
adapter: Ecto.Adapters.Postgres,
username: System.get_env("USERNAME"),
password: System.get_env("PASSWORD"),
database: System.get_env("DATABASE"),
hostname: System.get_env("DBHOST"),
# sometimes hostname is db (like in the docker-compose method - play with this one)
pool_size: 10
config :newsly, Newsly.Endpoint,
http: [port: 4000],
debug_errors: true,
code_reloader: false,
url: [scheme: "http", host: System.get_env("HOST"), port: 4000],
secret_key_base: System.get_env("SECRET_KEY_BASE"),
pubsub: [adapter: Phoenix.PubSub.PG2, pool_size: 5, name: Newsly.PubSub],
check_origin: false,
watchers: [node: ["node_modules/brunch/bin/brunch", "watch", "--stdin",
cd: Path.expand("../", __DIR__)]]
And in my Dockerfile I set my environment variables like the following;
ENV AWS_ACCESS_KEY_ID=nottelling
ENV AWS_SECRET_ACCESS_KEY=nottelling
ENV BUCKET_NAME=s3 storage bucket (not eb related)
ENV SECRET_KEY_BASE=nottelling
ENV HOST=host name of my eb instance im uploading to
ENV DBHOST=AWS rds host that holds postgres
ENV USERNAME=nottelling
ENV PASSWORD=nottelling
My health report on the instance fails to red with the following warning:
Environment health has transitioned from Warning to Severe. 100.0 % of the requests are failing with HTTP 5xx. ELB processes are not healthy on all instances. ELB health is failing or not available for all instances.
NGINX seems to be choking with the lines
2017/11/16 17:59:46 [error] 28815#0: *99 connect() failed (113: No route to host) while connecting to upstream, client: 172.31.20.108, server: , request: "GET / HTTP/1.1", upstream: "http://172.17.0.2:4000/", host: "172.31.38.244"
in nginx logs.
If I look at eb-activity log I have
duplicate MIME type "text/html" in /etc/nginx/sites-enabled/elasticbeanstalk-nginx-docker-proxy.conf:11
which seems to be killing nginx
[2017-11-16T18:02:33.927Z] INFO [29355] - [Application update app-8412-171116_115503#5/AppDeployStage1/AppDeployEnactHook/01flip.sh] : Completed activity. Result:
nginx: [warn] duplicate MIME type "text/html" in /etc/nginx/sites-enabled/elasticbeanstalk-nginx-docker-proxy.conf:11
Stopping nginx: [ OK ]
Starting nginx: nginx: [warn] duplicate MIME type "text/html" in /etc/nginx/sites-enabled/elasticbeanstalk-nginx-docker-proxy.conf:11
[ OK ]
iptables: Saving firewall rules to /etc/sysconfig/iptables: [ OK ]
Stopping current app container: e0161742ee69...
Error response from daemon: No such image: aws_beanstalk/current-app:latest
Making STAGING app container current...
Untagged: aws_beanstalk/staging-app:latest
eb-docker start/running, process 1398
Docker container e25f2b562f4f is running aws_beanstalk/current-app.
Does anyone have any ideas?
EDIT:
Digging through the logs for nginx I found
map $http_upgrade $connection_upgrade {
default "upgrade";
"" "";
}
server {
listen 80;
gzip on;
gzip_comp_level 4;
gzip_types text/html text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;
if ($time_iso8601 ~ "^(\d{4})-(\d{2})-(\d{2})T(\d{2})") {
set $year $1;
set $month $2;
set $day $3;
set $hour $4;
}
access_log /var/log/nginx/healthd/application.log.$year-$month-$day-$hour healthd;
access_log /var/log/nginx/access.log;
location / {
proxy_pass http://docker;
proxy_http_version 1.1;
proxy_set_header Connection $connection_upgrade;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
}
So,
duplicate MIME type "text/html" in /etc/nginx/sites-enabled/elasticbeanstalk-nginx-docker-proxy.conf:11
Seems to be referring to this line:
gzip_types text/html text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;
But at this point I would find it surprising if nginx was choking simply because it defines text/html twice. So now I'm not sure....
EDIT EDIT:
I should mention that my nginx/error.logs look like the following (the last lines repeat ad-infinum):
2017/11/16 17:19:22 [warn] 18445#0: duplicate MIME type "text/html" in /etc/nginx/sites-enabled/elasticbeanstalk-nginx-docker-proxy.conf:11
2017/11/16 17:19:22 [warn] 18460#0: duplicate MIME type "text/html" in /etc/nginx/sites-enabled/elasticbeanstalk-nginx-docker-proxy.conf:11
2017/11/16 17:20:06 [error] 18467#0: *11 connect() failed (113: No route to host) while connecting to upstream, client: 172.31.32.139, server: , request: "GET / HTTP/1.1", upstream: "http://172.17.0.2:4000/", host: "172.31.38.244"
2017/11/16 17:20:15 [error] 18467#0: *13 connect() failed (113: No route to host) while connecting to upstream, client: 172.31.20.108, server: , request: "GET / HTTP/1.1", upstream: "http://172.17.0.2:4000/", host: "172.31.38.244"
2017/11/16 17:20:21 [error] 18467#0: *15 connect() failed (113: No route to host) while connecting to upstream, client: 172.31.32.139, server: , request: "GET / HTTP/1.1", upstream: "http://172.17.0.2:4000/", host: "172.31.38.244"
2017/11/16 17:20:30 [error] 18467#0: *17 connect() failed (113: No route to host) while connecting to upstream, client: 172.31.20.108, server: , request: "GET / HTTP/1.1", upstream:
THIS IS THE HEART OF THE PROBLEM
NGINX fundamentally cant connect the entrypoint to the application and I don't know why!
UPDATE:
Using Kevin Johnson's advice I successfully pushed up to AWS ECR my Dockerfile and it compiled correctly when I eb deploy'ed my application with a good Dockerrun.aws.json. This is in fact a preferred way to do this. HOWEVER, I still get the same error! I do not know what is going on, but I think I can safely say that my Dockerfile successfully compiles.
I think there is something broken in AWS and I'm not sure what.
UPDATE
Problem is related to a NGINX routing issue. More information here in a clean question: How Do I modify NGINX routing on Elastic Beanstalk AWS?

Answer 1

I highly suspect that the issue you are dealing with is either:
Your Dockerrun.aws.config file points to a non existing docker image on ECS Repository. This is indicated by the error message:
Error response from daemon: No such image: aws_beanstalk/current-app:latest
Making STAGING app container current...
When EB fails to replace the instance with the latest configuration, it will resort back to the old one, which could be the Hello World app of AWS that you may have leveraged in setting up EB. That container does not have a web service running on port 4000, whereas your Dockerrun.aws.config specifies port 4000.
(I would be surprised though that your Dockerrun.aws.config does not get replaced as well by EB to previous working version)
So check Dockerrun.aws.config and ensure that the image endpoint mentioned therein actually exists. Try pulling it locally and run it accordingly. First clean up your local environment of all images and running docker containers if need be.
Your application running within docker immediately crashes upon startup. Again, EB will detect this and replaces the crashed container with the previous container which again does not have port 4000 open.