I'm building a simple web app on Elastic Beanstalk (Dockerized Python/Flask). I had it running successfully on one AWS account and wanted to migrate it to a new AWS account, so I'm recreating the Beanstalk app on the AWS console and trying to deploy the same code via eb deploy.
I noticed that when pushing a configuration update, Beanstalk will attempt the change but then roll it back if the app fails to start with the new change. This is true for several different kind of changes, and I need to make multiple to get my app fully working (basically I'm just recreating the Beanstalk settings I already have on my other AWS account):
Need to set a few environment variables
Need to set up a new RDS instance
Need to deploy my code (the new application version has been uploaded, but the deployed application version is still the old "sample application" that it started with)
All 3 must be done before this app will fully start. However, whenever I try one of these on its own, Beanstalk will attempt the change, then notice that the app fails to startup (it throws an exception on startup) and then beanstalk rolls back the change. The config rollback occurs even though I have "Ignore Health Check: true" under the deployment settings. (I would think at the very least it let me force update #3 above but apparently not.)
So I'm basically stuck because I can't do all of them at once. It there a way to --force a configuration update, so that Beanstalk doesn't rollback no matter what happens?
My other thought was that I could potentially make all the edits at once to the JSON config, but I figured that there must be a way to force config changes so people can respond quickly in a crisis without these well-intentioned guardrails.
Thanks for your help!
Related
I have a python3 project which runs in a docker container environtment.
My Python project uses AWS Acces keys and secret but using a credentials file stored in the computer which is added to the container using ADD.
I deployed my project to EC2. The server has one task running which works fine. I am able to go through port 8080 to the webserver (Airflow).
When I do a new commit and push to a master branch in github, the hook download the content and deploy it without build stage.
The new code is in the EC2 server, I check it using ssh but the container that runs in the task get "stuck" and the bind volumes dissapear and they are not working until I restart a new task. The volumes are applied again from 0, and those reference to the new code. This action is fully manual.
Then, to fix it I listen about AWS ECS Blue/Green deployment, so I implemented it. In this case the Codepipeline add a build stage, but here starts the problem. If in the build I try to push a docker image to the ECR, which my task definition makes reference it fails. It fails because in the server, and in the repo (which I commit push my new code) there is no credentials file.
I tryed doing the latest docker image from my localhost, and avoiding build stage in codepipeline, and it works fine, but then when I go to the 8080 port in both working ip's I am able to get in the webserver, but the code is not there. If i click anywhere it says code not found.
So, in a general review I would like to understand what i am doing wrong, and how to fix in a general guidelines, and in the other hand ask why my EC2 instance in the AWS ECS Blue/Green cluster has 3 ip's.
The first one is the one that I use to reach server through port 22. And if there I run docker ps I see one or two containers running depending if I am in the middle of a new deployment. If here I search my new code its not here...
The other two ip's are changing after every deployment (I guess its blue and green) and both work fine until Codepipeline destroys the green one (5 minutes wait time), but the code is not there. I know it because when I click in any of the links in the webserver it fails saying the Airflow Dag hasn't been found.
So my problem is that I have a fully working AWS ECS Blue/Green deployment but without my code. Then my webserver doesn't have anything to run.
I have a set of AWS Elastic beanstalk using Docker based configuration for both web server and worker server. The way we have setup is that the java process inside docker allocates 70% of the box memory when starting.
Now the first deployment works fine, but when I try to update application version with in-place Rolling update, Elastic beanstalk tries to start an additional docker container with the java process before stopping the existing one. This fails the deploy as the Java server is not able to allocate the required memory. Is there a way that I can setup AWS to kill the old docker instance before starting the new one during deployment?
I even tried Rolling with additional batch, but that one only works for the first batch and then fails for subsequent ones.
immutable updates can be the way to go for you, it basically recreates the EC2 instances completely on every deploy
Open the Elastic Beanstalk console.
Navigate to the management page for your environment.
Choose Configuration.
In the Rolling updates and deployments configuration category,
choose Modify.
Select immutable on deploy policy
Apply
you can check more on how it works here
Recently I've been testing AWS CodeDeploy to validate that it will be useful, and so far so good. But after seeing its workflow I started to wonder: "How can someone validate that the new environment is good, in a human way?"
Explaining in more detail:
On my "Deployment Group > Deployment Settings" using the Traffic Rerouting policy of "I will choose whether to reroute traffic", when the new environment boots up, the deployment pauses waiting for me to verify that everything is fine in this new environment. Then, after validation, I can push the "reroute traffic" button and it will proceed as expected.
To validate that the new environment is good I, as someone who has access to the machines, can SSH into one of them and do some tests. Or I can grab the Public DNS of one new machine and access it through the browser and verify that it is OK.
But is there a simpler way of validating the application on these new machines? Like having a Load Balancer that always points to the soon to be new environment that I can send to QA people. Or will I have to, for each and every deploy, manually grab information about the new environment and then send to the QA people?
In order to validate the new environment, you can add scripts as validation hooks in the Appspec that would be run after installing the new revision in the new hosts. Also, the new environment will be registered behind any load balancer you specify in the deployment configuration.
I have a powershell script in the .ebextensions folder but its not executing. It used to work but suddenly has stopped working and there have been no changes to elastic beanstalk or the scripts that i know off. I have two instance running under elastic beanstalk and if I log into one and then deploy my package it doesnt take the instance down. I can literally watch the service (IIS Rest service) files being updated.
Elastic Beanstalk is configured for autoscaling min 2 instances max 4. I've tried doing a deploy though the AWS Elastic Beanstalk console but no joy.
Does it need to actually take the instances down in order to run the scripts? I find it odd that it can update the service without stopping the instance.
Also do .ebextension scripts not execute against running instances? is there a way to force it to execute?
What else can I check to see why the ebextensions are not being picked up?
Thanks
It's working now. My ebextensions script was not formatted properly, i.e. didn't have the necessary indentations that YAML requires so the commands where being ignored. However no error was raised by EB.
Another problem is that eb cli uses git to bundle your app, which means that you need to perform a commit so that all changes are properly bundled.
Until I get my changes right, I keep doing git add & git commit --amend all the time...
I'm new to web development and i wrote a small Flask API that uses Celery for message queue and Redis as the broker. I start redis with redis-server and Celery with celery -A application.celery worker --loglevel=info on my local machine and the app runs with no problem.
However i couldn't get it to work on AWS. Right now I'm deploying the app following the docs but when I try to send requests to my API I get internal server errors, which are probably related to Redis and Celery not working. I SSH'ed into the EC2 instance but since I'm new, couldn't find what to do to get the app working.
My questions are:
1) What do i do to start my application, Redis and Celery after deploying it to AWS? Does Elastic Beanstalk do it automatically or do I need to do some stuff?
2) Where do I find my app files? I think I'll need to install all the requirements manually from requirements.txt, and set up a virtualenv in the EC2 instance, is that right?
3) If I setup and install all the requirements in a virtualenv, will they persist if the EC2 instance changes? The command line tool for Elastic Beanstalk deployed the application automatically and created Load Balancer and Auto Scaling Group. Will the installations I make through the SSH be available when new instances are created, or do I need to manually do that everytime, or is there some other way?
4) I heard some people say that creating an EC2 instance and deploying manually is better than using Elastic Beanstalk. What does Elastic Beanstalk do for me? Is it better if I use Elastic Beanstalk or deploy manually?
Thanks for any help!
For the past week I was trying to do the exact same thing, so I'd thought I'd share everything I've learned. Although these answers are spread about stackoverflow/google, but I can help all the same.
1) To get a flask app running is easy, you can use the elastic beanstalk CLI. Generally, just follow the AWS documentation here, it's fairly straightforward. In terms of Redis/Celery, you start to get multiple things going on. Before you do your initial deploy, you'll probably want to setup the celery worker, you can use this stackoverflow answer on how to setup celery as a daemon. Be sure you read the script, you'll need to set your application name properly. The thing to note when you deploy to production via EBS is that your application will be hosted by apache, meaning some strange things will happen if you call your tasks via "some_task.delay", as the name of the celery app will be unknown. As far as I know, the only way to work around this properly is to use:
my_celery_object.send_task("flask_application_name.the_task", [param1, param2], ...)
Wherever you need to call tasks.
You can now prepare your redis cache. You can use anything, for this I'll just assume you want to use AWS ElasticCache (EC). Going to EC, you'll need to deploy a cache cluster with however many nodes you want. Once deployed you'll see it on the list under "Cache Clusters". Next, click the "X node" link that's in the table, you'll need to copy the endpoint url (and port!) to your celery application which you can learn about here.
So now that you have everything ready to deploy, you'll be sad to hear that the security thing I mentioned earlier will cause your application to fail on any task requests as the elastic cache cluster will be part of the wrong security group initially. Go ahead and deploy, this will create the security group you need along with your app and everything else. You can then find that security group under the EC2 dashboard, under Network & Security -> Security Groups. The name of the group should be the name of your environment, something like "myapp-env" is the default. Now modify the inbound rules and add a custom TCP rule setting the port number to your redis port and the source to "Anywhere", save. At this point, note the group name and go to your elastic cache. Click the Cach Clusters on the left, modify the CACHE CLUSTER (not the node) for the app, and update the VPC security group to the one you just noted and save.
Now celery will automatically connect to the redis cache as it will keep attempting to make connections for awhile. Otherwise you can always redeploy.
Hopefully you now have a functioning Flask/Celery app utilizing redis.
2) You shouldn't need to know the location of your app files on the EBS EC2 instance as it will automatically use a virtual environment and requirements.txt assuming you followed the instructions found here. However, at the time of writing this, you can always ssh to your EC2 instance at find your application files at:
/opt/python/current/app
3) I don't know how you mean "If I setup and install all the requirements in a virtualenv, will they persist if the EC2 instance changes?" As I said previously, if you followed the instructions on how to deploy an EBS environment for flask, then new instances that are deployed will automatically update their environment based on your requirements.txt
4) This is a matter of opinion. I have definitely heard not using EBS could be a better way to go, I personally have no opinion here as I have only used EBS. There have been some severe struggles (including trying to setup this application). What I hear some people do is deploy via EBS so that they can get a pre-configured and ready to go EC2 machine and then make an AMI from that machine, tear the EBS down, and then make an EC2 with the AMI. Regardless of the route you go, if you are planning to have a database backed server, I have learned (the hard way) that you shouldn't have EBS automatically attach the RDS. This is due to the fact that the RDS is then associated with the EBS application, so if you have to replace the resources, terminate it, etc., you'll lose the RDS (you can work around this of course, it's just a pain is all).