Currently we have Jenkins that is running on-premise(VMware), planning to move into the cloud(aws). What would be the best approach to install Jenkins whether on ec2 or ECS?
Best way would be running on EC2. Make sure you have granular control over your instance Security Group and Network ACL's. I would recommend using terraform to build your environment as you can write code and also version control it. https://www.terraform.io/downloads.html
Have you previously containerized your Jenkins? On VMWare itself? If not, and if you are not having experience with containers, go for EC2. It will be as easy as running on any other VM. For reproducing the infrastructure, use Terraform or CloudFormartion.
I would recommend dockerize your on-premise Jenkins first. See how much efforts are required in implementation and administrating/scaling it. Then go for ECS.
Else, shift to EC2 and see how much admin overhead + costs you are billed. Then if required, go for ECS.
Another point you have to consider is how your Jenkins is architected. Are you using master-slave? Are you running builds contentiously so that VMs are never idle? Do you want easy scaling such that build environment is created and destroyed per build execution?
If you have no experience with running containers then create it on EC2. Before running on ECS make sure you really understand containers and container orchestration.
Just want to complement the other answers by providing link to official AWS white paper:
Jenkins on AWS
It might be of special interest as it discusses both options in detail: EC2 and ECS:
In this section we discuss two approaches to deploying Jenkinson AWS. First, you could use the traditional deployment on top of Amazon Elastic Compute Cloud (Amazon EC2). Second, you could use the containerized deployment that leverages Amazon EC2 Container Service (Amazon ECS).Both approaches are production-ready for an enterprise environment.
There is also AWS sample solution for Jenkins on AWS for ECS:
https://github.com/aws-samples/jenkins-on-aws:
This project will build and deploy an immutable, fault tolerant, and cost effective Jenkins environment in AWS using ECS. All Jenkins images are managed within the repository (pulled from upstream) and fully configurable as code. Plugin installation is automated, including versioning, as well as configured through the Configuration as Code plugin.
Related
What is the difference between an AWS EC2 instance and a docker container instance? When should I use one over the other?
When you get an EC2 instance it will provide the base installation of that specific operating system with some additional AWS packages installed such as the SSM Agent.
There are then AMIs that are prepared for specific usecases such as SQL Server, or in this case pre-configured with AWS Orchestration services (either ECS or EKS) which have the usecase software installed.
If you're not familiar with Docker I would suggest running it in your local environment first so that you can become familiar with it. Yes people have been moving towards containers and serverless but you need to ensure you are able to support this in production.
With containers being deployed you will need to understand the orchestration layer that you're using. It's very easy to see containers as an alternative to a virtualisation layer, but there are many differences to how these operate.
Take a look at the What is Docker? page for further explanations.
I am trying to find a solution for configuration management using AWS OpsWorks. What I can see is AWS offers three services for OpsWorks
Chef Automate
Puppet
AWS stacks
I have read basics of all three of them but unable to compare between three of them. I am unable to understand when to use which solution.
I want to implemnet a solution for my multiple EC2 instances, using which I can deliver updates to all my instances from a central repository(github). And, rollback changes if needed.
So following are my queries:
Which of the three solutions is best for this use case?
What should I use if my instances are in different regions?
I am unable to find anything useful on these topics so that I can make my decision. It would be great if I can get links to some useful articles as well.
Thanks in advance.
Terraform, Packer and Ansible are a great resource, I use them everyday to configure AMI's and build out all my infrastructure.
Terraform - Configuration Management for Infrastructure, it allows you to provision all the AWS, Azure, GCE components you needs to run your application.
Packer - Creates reusable images by pre installing software that is common to your applications.
Ansible - pre and post provisioning configuration management. You can use Ansible with Packer to provision software in an AMI, then if needed, use Ansible to configure it after provisioning. There is no need for a chef server or puppet master, you can run Ansible from your desktop if you have access to the cloud servers.
This examples provisions all the infrastructure for a Wordpress site, and uses Ansible to configure it post provisioning.
https://github.com/strongjz/tf-wordpress
All of this as well can automated in a Jenkins pipeline or with other Continous Deployment tools like CircleCI etc.
Ansible has no restriction on regions, neither does Terraform. Packer is a local build tool or on a CD server.
Examples:
https://www.terraform.io/intro/examples/aws.html
https://github.com/ansible/ansible-examples
https://www.packer.io/intro/getting-started/build-image.html
I am using AWS Elastic Beanstalk and have deployed my nodejs app on it. Now I want to automate this process i.e committing changes to Github and then automatically reflecting those changes in app. Now I have two options, use whether Elastic Beanstalk or using Code Deploy.
I have searched on both services,
I can automate using deployBot with elastic beanstalk or using
jenkins plugin for automation (AWS Elastic Beanstalk Deployment
Plugin) for elastic beanstalk.
Also found this link to automate:
https://aws.amazon.com/blogs/devops/building-continuous-deployment-on-aws-with-aws-codepipeline-jenkins-and-aws-elastic-beanstalk/
I can also use AWS CodeDeploy service for automation to deploy my app on EC2
instances using CodeCommit , code pipeline.
In case of code deploy I can also do by using this:
https://aws.amazon.com/blogs/devops/automatically-deploy-from-github-using-aws-codedeploy/
Now both services can be used , but which one is more suitable to use. That will automate my process whether using AWS Elastic Beanstalk or AWS Code Deploy.
The biggest difference is, that:
CodeDeploy is the service that deploys your application to the existing EC2 instance(s). It does not take into account LoadBalancing or scaling etc.
ElasticBeanstalk is more of the PaaS service, that provides you all the wrapping you need to scale your application so you don't need to worry about the DevOps aspect. Like monitoring, scaling etc.
I found this image to describe the differences nicely. Including as well OpsWorks:
If you want to read more about differences of CodeDeploy, Elastic Beanstalk or OpsWorks, check out AWS own document: https://d0.awsstatic.com/whitepapers/overview-of-deployment-options-on-aws.pdf
The answer is very simple. ElasticBeanstalk offers cookie-cutter automated deployments based on a set of AWS common practices. CodeDeploy is broadly configurable and customizable.
You should use ElasticBeanstalk until you find a use case that cannot be resolved without using CodeDeploy (two use cases suggested by the AWS Documentation posted by Maksim Luzik are deploying to EC2 instances managed internally by your organization and deploying to EC2 instances for third-party integration).
Use the second option instead of using third party tools as AWS platform is supporting to deploy your app using git or bitbucket using python based scripts.
I have worked with both tools and both are great for respective jobs. I found ElasticBeans task convenient but lesser flexible when It comes to work with custom platforms.
I am using codeDeploy in my current application. I decided so because of following use cases.
I am using debian based platform. Elastic BeansTalks does not offer that platform in its default list of available platforms. So what's the point if I need to create custom AMI.
I I have 2 type of applications built on the top of same code base. One is Web and other executes couple of queues in the background. I need to release same code on both type of applications so that's why I found codeDeploy does better job.
We have multiple amazon ec2 instances behind a load balancer. Our build script is written in phing and is integrated with git.
We are looking for a tool (like Jenkins or Amazon code deploy) which could display all the active instances currently behind load balancer and then allow us to select some of them (or select a group defined previously) and then trigger either of the following (whichever is better) -
a build script hosted on the same dedicated server where the tool is hosted.
or the respective build scripts hosted on the selected ec2 instances.
We should be able to do the following -
specify a git branch name, optionally, when we trigger the build script for any group of instances.
be able to roll out in batches of boxes, so as to get some time to monitor load, and then move to next batch if all is good. Best way, I guess, would be to specify a size of the batch (e.g. 10), so that the process waits for a user prompt after rollout on every batch completes.
So, if we have to rollout two different git branches to two groups of instances, we should be able to run them in two steps (if we do not specify batch size).
Would like to know about experiences of people who dealt with something similar.
For CodeDeploy, it supports Git (more precisely, GitHub). It also allows you to deploy only to tagged EC2 instances. If combined with custom DeploymentConfig (http://docs.aws.amazon.com/codedeploy/latest/userguide/how-to-create-deployment-configuration.html), you can also control how fast (the size of the batch) to deploy.
I would re-structure the question:
The choices you have for application deployment
and whether the tool has option to perform rolling deployments.
Jenkins is software for CI/CD, which will have to use plugins,custom scripting or leverage an existing orchestration software setup for doing the deployments.
For software orchestration, you have many choices, some of the more famous tools are Chef, puppet, ansible etc.. All of these would need you to manage some kind of centralized setup. All such software support application deployment.
You need to make a decision on whether you would want to invest in maintaining such a setup.
If you decide against such a setup, you have the option of using managed services such as AWS OpsWorks, AWS CodeDeploy, hosted chef etc.
In choosing any of these services, you delegate the management of orchestration software to a vendor, which will ensure the service is up all the time.
AWS code deploy and AWS OpsWorks are managed services on aws and work pretty well on AWS setups.
AWS OpsWorks uses chef under the hood.
AWS CodeDeploy only provides a subset of what OpsWorks provides and is responsible only for deployments. With AWS code deploy you get convenient visualization of your software deployments through AWS console.
With AWS code deploy, you can achieve the goal of partial roll out to ec2 instances.
You can do the same with other tools as well but CodeDeploy on AWS environment will take least amount of work.
CodeDeploy also allows you to deploy from GIT. Please refer to the following aws documentation
http://docs.aws.amazon.com/codedeploy/latest/userguide/github-integ-tutorial.html
The pitfall with code deploy is the fact that the agent that will run on instances has been tested for and is supported for only a limited number of OS combinations.(http://docs.aws.amazon.com/codedeploy/latest/userguide/how-to-run-agent.html#how-to-run-agent-supported-oses)
Also in future if you decide to move away from AWS, you will have to redo the deployment related work.
CodeDeploy service only charges you for the underneath AWS resources.
Please find the link to pricing documentation below:
https://aws.amazon.com/codedeploy/pricing/
I am working on a project using a microservices architecture.
Each service lives in its own docker container and has a separate git repository in order to ensure loose coupling.
It is my understanding that AWS recently announced support for Multi-Container Docker environments in ElasticBeanstalk. This is great for development because I can launch all services with a single command and test everything locally on my laptop. Just like Docker Compose.
However, it seems I only have the option to also deploy all services at once which I am afraid defies the initial purpose of having a micro services architecture.
I would like to be able to deploy/version each service independently to AWS. What would be the best way to achieve that while keeping infrastructure management to a minimum?
We are currently using Amazon ECS to accomplish exactly what you are talking about trying to achieve. You can define your Docker Container as a Task definition and then Create an ECS Service which will handle number of instances, scaling, etc.
One thing to note is Amazon mentions the word container a lot in the documentation. They may be talking about the EC2 instance used for the cluster for your docker instances/containers.