I have most of my app designed and deployed. We have gone for RabbitMQ over SQS for several reasons.
I currently have my consumer running in a docker container and would like to deploy this on AWS. I am fairly familiar with Elastic Beanstalk since our web-tier is running there, but it seems all workers deployed this way have to use SQS?
The other option I am aware of is to use ECS for the docker component, but I do not want to make the image publicly available and don't have access to a private repository.
Is there some basic functionality I am missing, a document describing how to deploy to ECS using a Dockerfile and source code locally, or a way of deploying to EB using a Dockerfile without being locked into SQS?
edit
So I have found the note on the docs for EB which says that Dockerfiles are not supported when building a multi-container environment and that repositories currently have to be used for that purpose, so EB is out for me.
Related
I've been trying to find an efficient way to handle continuous deployment with a Docker compose setup and AWS hosting.
So far I've looked into CodeDeploy, S3 buckets, and ECS. My application is relatively small with only 3 docker services, a Django app, NGINX, and PostgreSQL. I was unable to find any reliable information for using CodeDeploy with Docker compose and because of the small scale ECS seems impractical. I've considered an S3 bucket but that seems no better than just deploying my application with something like git or scp.
What is a standard way of handling deploying a docker compose setup on AWS? If possible I would like to use Bitbucket Pipelines or CircleCI to perform the deployment in a manually triggered step after running tests. But I've been unable to find a solution that would easily let me copy over the code (which is in a git repo on a production branch and is how I get the code onto the production server at the moment).
I would like to add some possibilities to #gasc answer
It would be better if you make a cloudformation template for deploying your EC2 resources with all required groups, auto scaling and other stuff.
Then Create the AMI with docker compose installed or any other thing you would be required for your ec2 enviroment.
Then you can use code deploy pipeline, here also aws provides private container registry may be you want to use that
Rest of the steps are same just SCP the compose file into EC2 launch
docker-compose up
command and you are done.
Let me know if you want more help I'm open for discussion
What I will do in your case is:
1 - If needed, update your docker-compose.yml file (or however you called it) to version 3 or higher, to use swarm.
2 - During your pipeline build all images needed, and push them to a registry.
3 - In your pipeline scp your compose file to a manager node.
4 - Deploy your application using swarm (docker stack deploy -c <your-docker-compose-file> your_app_name). This way you can handle rolling updates and scale easily.
Note that if you want to use multiple nodes you need to open a few ports in them
I see you mentioned that ECS might seem impractical for such a small scale - in my opinion not necesarilly. It would require of you to rewrite your docker-compose.yml into task and services definitions, but since there's not a lot of services, that shouldn't take you much time.
I wonder if Spinnaker (http://spinnaker.io) can be used for docker container deployment?
What we do is:
Poke the repo
If the code is new there - we build 3 containers (nginx, django app container, fluentd logger container)
we are spinning up fluentd container in order to collect the logs from the rest 2 containers and send it to Splunk/AWS Cloudwatch Logs
we want to spin up django app container, on the same host - nginx container (as a proxy to Django container) [and forward the logs into fluentd ]
we forward (map) the certain json file with the app configuration ito the django container
Unfortunately Spinnaker has too few examples, the example they have here shows only how to bake the image with the certain DEB package inside.
We do have jenkins jobs which can poll the repo, test the code, create and upload the docker container into the private registry and deploy the containers using ansible. The question is if we can use Spinnaker in order to do that natively?
there is currently no container support in Spinnaker. Google is actively working on adding Kubernetes support. But there is currently no plans to integrate Spinnaker directly with either docker or ecs.
One thing we tried and worked was to use Jenkins to build and publish a debian wrapper for the docker image that was created. All that this debian does is to pull and start the docker container for a spinnaker service. We then created a spinnaker pipeline that bakes this debian and then deploys it.
My app is created using elastic-beanstalk aws service, do I need to use the AWS CodeDeploy service to deploy my app?
Currently I just do:
eb deploy myApp
Then, a new application version is deployed without using AWS CodeDeploy.
So, AM I doing something wrong?
Elastic beanstalk do it on your behalf. During deployment process you define some polices and roles, which defines elastic beanstalk will call these services on your behalf. Codedeploy is one of that services.
Elasticbeanstalk does automation of your process only and setup thet whole deployment environment for you (php,nginx/apache in case of web), if you look /opt/elasticbeanstalk/, you can see codedeploy folder there, which means that you do not need to do it manually.
AWS code deploy is different workaround and provides more controlling. How you want your changes to be pushed, is it to be pushed on all instances an once or one by one, minimum number of healthy instances.
Check here-
http://cloudacademy.com/blog/how-to-deploy-application-code-from-s3-using-aws-codedeploy/
http://blog.powerupcloud.com/2016/03/24/deployment-automation-using-aws-code-depoly/
https://blogs.aws.amazon.com/application-management/post/Tx33XKAKURCCW83/Automatically-Deploy-from-GitHub-Using-AWS-CodeDeploy
You can update the your application with new version. CLI as follows
$eb deploy --version
You are not doing anything wrong. EB Deploy will enable you to deploy your apps being served from Elastic Beanstalk. AWS Code Deploy on the other hand is more flexible & gives you more control, you can for example, deploy apps you are serving from EC2 thats not being managed by Elastic Beanstalk.
With AWS you can for example deploy to multiple environments ie development, staging, production.
Elastic Beanstalk and CodeDeploy are totally different AWS services and independent of each other and follow different deployment approaches.
What you're doing is totally correct to deploy a new version of your code.
AWS elastic-bean-stalk itself has the nice capability for deploying applications nicely.You dont need to use aws code deploy again.It will be superflous.You can use beanstalk tools itself to deploy the code.
AWS CodeDeploy is a building block service focused on helping developers deploy and update software on any instance, including Amazon EC2 instances and instances running on-premises.
AWS Elastic Beanstalk (as well as AWS OpsWorks btw) are end-to-end application management solutions.
When it comes to deploying new software release on Beanstalk, you better use the own deployment process provided to you by Beanstalk.
eb deploy myApp
I am working on a project using a microservices architecture.
Each service lives in its own docker container and has a separate git repository in order to ensure loose coupling.
It is my understanding that AWS recently announced support for Multi-Container Docker environments in ElasticBeanstalk. This is great for development because I can launch all services with a single command and test everything locally on my laptop. Just like Docker Compose.
However, it seems I only have the option to also deploy all services at once which I am afraid defies the initial purpose of having a micro services architecture.
I would like to be able to deploy/version each service independently to AWS. What would be the best way to achieve that while keeping infrastructure management to a minimum?
We are currently using Amazon ECS to accomplish exactly what you are talking about trying to achieve. You can define your Docker Container as a Task definition and then Create an ECS Service which will handle number of instances, scaling, etc.
One thing to note is Amazon mentions the word container a lot in the documentation. They may be talking about the EC2 instance used for the cluster for your docker instances/containers.
I am trying to setup a new springboot+docker(microservices) based project. The deployment is targeted on aws. Every service has a Dockerfile associated with it. I am thinking of using amazon container service for deployment, but as far as I see it only pulls images from docker hub. I don't want ECS to pull from docker-hub, rather build the images from docker file and then take over the deploying those containers.Is it possible to do? If yes how.
This is not possible yet with the Amazon EC2 Container Service (ECS) alone - while ECS meanwhile supports private registries (see also the introductory blog post), it doesn't yet offer an image build service (as usual, AWS is expected to add such notable additional features over time, see e.g. the Feature Request: ECS container dream service for more on this).
However, it can already be achieved with AWS Elastic Beanstalk's built in initial support for Single Container Docker Configurations:
Docker uses a Dockerfile to create a Docker image that contains your source bundle. [...] Dockerfile is a plain text file that contains instructions that Elastic Beanstalk uses to build a customized Docker image on each Amazon EC2 instance in your Elastic Beanstalk environment. Create a Dockerfile when you do not already have an existing image hosted in a repository. [emphasis mine]
In an ironic twist, Elastic Beanstalk has now added Multicontainer Docker Environments based on ECS, but this highly desired more versatile Docker deployment option doesn't offer the ability to build images in turn:
Building custom images during deployment with a Dockerfile is not supported by the multicontainer Docker platform on Elastic Beanstalk. Build your images and deploy them to an online repository before creating an Elastic Beanstalk environment. [emphasis mine]
As mentioned above, I would expect this to be added to ECS in a not too distant future due to AWS' well known agility (see e.g. the most recent ECS updates), but they usually don't commit to roadmap details, so it is hard to estimate how long we need to wait on this one.
Meanwhile Amazon has introduced EC2 Container Registry https://aws.amazon.com/ecr/
It is a private docker repository if you do not like docker hub. Nicely integrated with the ECS service.
However it does not build your docker images, so it does not solve the entire problem.
I use a bamboo server for building images (the source is in git repositories in bitbucket). Bamboo pushes the images to Amazons container registry.
I am hoping the Bitbucket Pipelines will make the process more smooth with less configuration of build servers. From the videos I have seen all your build configuration sits right in your repository. It is still in a closed beta so I guess we will have to wait a bit more to see what it ends up being.