We are a relatively inexperienced development team trying to do things 'the right way'. We are using Github along with AWS and CodeDeploy for multiple PHP based web applications. We are utilising Github's auto-deployment with CodeDeploy when the master branch is updated.
We have two production EC2 web servers in separate AZ's along with a single EC2 staging server.
It currently works as follows:
We write code in a branch, we push to GitHub, we merge into 'master' which then kicks off CodeDeploy to write to our staging server where we can test it. Once we have tested it we then manually kick off CodeDeploy to write to production (with the same commit ID).
The problem is, if testing brings up issues, and we have another branch waiting to be merged and tested, everything becomes backed up?
We are obviously doing something wrong. We are writing to the master branch to utilise GitHub's autodeploy, but I assumed master was only to be written to when it was ready to be deployed?
Can someone please help us and put us straight?
Thanks
Make another branch called 'livecandidate' this branch will have each of the new feature branches merged into it
Each time a feature branch is merged into 'livecandidate' pull 'livecandidate' into your Code Deploy process and install to the test machine.
If the tests pass then merge 'livecandidate' into 'master' and kick off the install to production
If the tests do not pass then unwind the merge into 'livecandidate' (assuming no dependencies on chains of changes etc)
After doing a production install or a un-merge, try the next feature
General idea is to never ever have a broken master
All problems in computer science can be solved by another level of indirection - David Wheeler
Related
The Problem
Suppose I have a simple CRUD web application. The application is containerized and developed locally and set up with main/staging/develop branches on Github. CI/CD is configured with Github actions. Merges to main trigger a deployment to AWS App Runner.
Generally, we need two main AWS services here: Cloud Formation, ECR, and App Runner.
Where does the IaC with AWS CDK belong?
Approaches
Have a separate repository for the IaC. Run this repository once to setup the App Runner service and a dedicated ECR repository. Treat the ECR repository URI as an environment variable in the application repository. On merge to main in the application repository, Github Actions rebuilds and pushes the image to the ECR repo. The App Runner service detects the new image and redeploys per this documentation.
Pros: Separation of responsibilities between repositories. One is for infrastructure, One is for application code. CDK only runs once. Deployments are far simpler and easier to diagnose.
Cons: Significant manual overhead. More work to change infrastructure.
AWS CDK supports declaring an App Runner service with a local docker image per this documentation. Simply create an AWS CDK project directly in the application repository. Upon merge to main, it simply re-runs the CDK with the new image.
Pros: One repository. Almost entirely automated with no manual overhead from infrastructure/DevOps team.
Cons: Developers may have to worry about IaC. Potential compute overhead with constant re-runs of CDK.
The Actual Questions
Which approach is best for websites/API that might rely on multiple backend services?
Which one fits the best in a development culture that relies heavily on microservices? Is there another approach I'm not thinking of? Am I asking the wrong questions?
I personally prefer approach 2 because I hate manual overhead.
I have less experience in microservices, so I was hoping some people with more industry experience could present some insight.
If this is the wrong place to ask this question or if I need to be more specific, please comment below and I'll adjust accordingly.
Have a separate repository for the IaC
The most compelling reason to do this is to decouple CI/CD for IaC from the
app repo. For example, if the application codebase is not continuously delivered, and the IaC is committed to the same repo as the codebase, you end up in a scenario where the IaC is versioned along with the code. If you deploy an old version, or a branch, does that version or branch get IaC from the same ref in the repo as the code itself? If so, you're in a position where you have to merge IaC changes across branches to make them deployable, which is a huge headache.
Most development teams as of 2022 do want to continuously deliver their code and "fail forward" rather than rolling back to old versions. In this scenario it doesn't really matter - because the main branch of the code is also the main branch of the infrastructure. But in this case, rolling back to an old version of the code inherently means rolling back to the matching version of the infrastructure, so you can't do it without looking very carefully at what infrastructure changes were made between the two versions and whether it's safe to roll back infra changes or not.
On the other hand, if the IaC repo is separate from the code repo, the IaC can be made to accommodate multiple versions of the code. Dependencies still exist - new features in the app that require new infrastructure are inherently dependent on the Infra as Code changes, and you don't have the shared repo to make sure those dependencies are deployed before the app.
It usually comes down to a question of ownership. If the infrastructure is primarily managed by a distinct group, then putting the infra in a separate repository makes a lot of sense because commingling infrastructure changes with code changes makes it hard for these groups to operate independently. To push out an infrastructure change from a different repo, is essentially an isolated step. To push out an infrastructure change from the same repo, requires merging a PR into the code base and deploying that. If the infra change is the only thing being deployed , that's pretty straight forward. but if the CI branch of the codebase is in a messy state then the infrastructure becomes undeployable because the code is undeployable. If the infrastructure is owned by the team whose job it is to also keep the code repo clean and deployable, then splitting the repos apart doesn't do much good.
Having spent the last 12 years or so doing DevOps, I'm pretty attracted to putting IaC in a separate repo for messy applications whose teams struggle with continuous delivery. That way when I want to make infrastructure changes, I can consider them in relative isolation and can deploy them to all environments regardless of which version of the code is deployed there. It really sucks to be trying to migrate database hosting, for example, if you need to work with the product team to get your IaC into each version of the code deployed into each environment. But it's not a free lunch - I still have to make sure to coordinate dependencies between the infra and the code, of course.
The smaller the service, the more the development team also handles IaC, and the more disciplined the development team's approach to CICD , the less it matters. If the same code goes out to dev/prod anyway and code merge and deploy is a frequent, comfortable thing, then the IaC may as well be in the app codebase. But you have to be ready to be limited to a fail-forward, continuously integrated approach, and accept that infrastructure and code deployments are coupled at the repo layer.
Most microservice dev teams tend to own their own IaC, continuously deliver their application, and put their IaC in the same repo as their code.
We have a CodePipeline that runs on every GitHub commit/merge to the main branch, building the application and releasing it to a staging environment where we can manually test the application. Every now and then, ad-hoc, weekly, etc, depending on the project, we'd release to production manually. To implement this I added a ManualApprovalStep to my CodePipeline between staging and production but that means that my pipeline is never green. It's always stuck in blue:
This makes me think that I'm using the wrong tool here.
My mental model is coming from Heroku (ignore the review apps, I'm not tackling that challenge yet):
In Heroku there's a Tests tab that's green if the tests pass and there's a pipeline that's green if it gets deployed to staging. Lack of promotion to production in Heroku is not a non-green state in Heroku but it would be in ManualApprovalStep.
Is there another tool that AWS gives me to model this way of working that I'm missing?
Update: another big difference. The ManualApprovalStep seems to pile each change and releasing each change, one by one, not releasing whatever was the last release to staging, so clearly it's not analogous to the release to production that Heroku has.
You are right that the ManualApprovalStep is not a natural "promotion" mechanism. They are for yes-no approvals and will result in execution failure if rejected or after 7 days. Disabled Stage Transitions also sit awkwardly with your use case.
pipeline.CodePipline executions are (a) triggered on a change to a source and (b) meant to run all stages from start-to-finish. Executions are hard to interrupt. A consequence of a requirement to deploy environments independently is that environments are best modelled as independent pipelines, not stages within a single pipeline.
Simple Option: 2 github branches, 2 Pipelines
Clone your pipeline setup. A staging pipeline is tied to a staging branch source. A prod pipeline is triggered on changes to the main branch. This setup is easy to reason about and has the advantage that deploys always match your source. But it does not replicate the Heroku "promotion" concept.
Complex Option: 1 github branch, 2 Pipeline?
You could probably get something closer to the "promotion" pattern by having a pipelines.CodePipeline deployment for staging (tied to github) and a separate codepipeline.Pipeline pipeline for prod. The latter can be triggered by EventBridge events. Asset handling would be complex in this scenario.
[Edit:] Amplify CI/CD for the Front-end, CodePipeline for Back-end
AWS Amplify CI/CD gives you automatic feature branch deploys, PR review approvals etc. for front-end apps. Manual deploys require a workaround, but are possible. See this related SO question. The CDK supports Amplify build configurations. The catch is that these CI/CD goodies work for front-end apps, but not for arbitrary infrastructure stacks. To get the best of both worlds, split the app in two. Use Amplify for the high-velocity front-end and stick with CodePipeline for the back-end deploys.
I am just getting started with AWS Amplify and after some research, I am still unable to set up the environments structure I want. I have a Reactjs app which I want to host there, my plan is to have 3 environments:
Dev: this environment is to test new features. Every new branch I create is automatically deployed to this environment (no problem here, already implemented).
Staging: Once new features are merged into master branch I would like to have them deployed here. This should work as a pre-production environment.
Production: Once features in staging are tested, they should be released into Production with just 1 click (or an easy action). Also production should be always running with the latest released build of the project.
So, what's the problem exactly? So far I don't know how to have master pointing to 2 environments, meaning that it is either deployed in staging or production environment, and promoting from staging to production is rather tedious at the moment.
Is there any way to implement this workflow in Amplify? Thank you in advance for your help.
I am working with a client that demands multi-stage server setup: development server, stage server and production/live server.
Stage should be as stable as it can be to test all those new features we develop at the development server and take this to the live server in the end.
We use git and github for version controlling. I use Ubuntu server edition as the OS.
The problem is, I have never working in such multi-stage server plan. What software/projects would you recommend to do a proper way of handling such setup, especially deployment and moving a new feature developed to the stage and then to the live server ?
We use two different methods of moving code from environment to environment. The first is to use branches and triggers with our source control system (mercurial in our case, though you can do the same thing with git). The other, is to use fabric, a python library for executing shell code across a number of servers.
Using source control, you can have several main branches, like production development staging. Say you want to move a new feature into staging. I'll explain in terms of mercurial, but you can port the commands over to git and it should be fine.
hg update staging
hg merge my-new-feature
hg commit -m 'my-new-feature > staging'
hg push
You then have your remote source control server push to all of your web servers using a trigger. A trigger on each web server will then do an update and reload the web server.
To move from staging to production, it's just as easy.
hg update production
hg merge staging
hg commit -m 'staging > production'
hg push
It's not the nicest method of deployment, and it makes rolling back quite hard. But it's quick and easy to set up, and still a lot better than manually deploying each change to each server.
I won't go through fabric, as it can get quite involved. You should read their documentation so you understand what it is capable of. There are plenty of tutorials around for fabric and django. I highly recommend the fabric route as it gives you lots more control, and only involves writing some python.
There is a nice branching model for git (as it is also used by github itself for example). You can easily apply this branching model using git-flow, which is a git extension that enables you to apply some high level repository operations that fit into this model. There's also a nice blogpost about this.
I do not know what exactly you want to automize in your deployment workflow, but if you apply the model mentioned above, most of the correct version handling is done by git.
To add some further automatic processing to this, fabric is a simple but great tool, and you will find many tutorials about its usage (also in combination with git).
For handling python dependencies using virtualenv and pip is for sure a very good way to go.
If you need something more complex,eg. to handle more than one django instance on one machine and for handling system wide dependencies etc checkout puppet or chef.
Try Gondor.io or Ep.io, they both make it pretty easy (gondor especially excels in this area) to have two+ instances with very similar code, from your VCS -- and to move data back and forth. (if you need an invite, ask either in IRC, but if I recall, they're both open now)
I wonder if there is any best practice or at least a more practical way to deploy C/C++ executable to Linux based production servers.
I have Jenkins up and running as CI server, and created a main SVN module which contains multiple svn:externals. This module is mainly served as a pipeline of related C++ applications. (Perhaps I should post this an another question on whether svn:externals is the correct way to do it)
So the main question is the deployment steps, I am planing to make all production servers as Jenkins' slaves with parameterized config, for the purpose of building from SVN tags. And use some scripts to copy all executables to, eg: /opt/mytools/bin in multiple production servers.
Any recommendations?
The best deployment route is the one specified by your distribution, IMHO. That is, for debian packages, bundle your applications into .deb-files, put them into a repository and let apt-get take care of the rest. This way, you have a minimal impact on the production environment and most admins are already familiar with the deployment scheme.
I'm working through some of the same questions, and I'm finding that Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation by Humble and Farley has been a good (technology agnostic) starting point - not perfect but it's pointed me in the right direction when I had no idea what to do next.
The continuous delivery book recommends setting up 'build pipelines' in which you run progressively more and more automated tests, with only the final manual tests and deploy rollback options being triggered by a real person.