Hi guys I've this configs into my bitbucket-pipelines.yml
pipelines:
branches:
'{master, develop, feature/branchThatDontTrigger}':
- step:
caches:
- node
script:......
Both for master and develop the pipeline works but since when I've added the feature branches it doesn't start. I've also try writing the name of the branch without the prefix feature/ but nothing has changed.
Can someone help me? Shoul I change something into settings repository?
I've been learning Bitbucket Pipelines the last 2 days and a thing that popped right out of your configuration is that you shouldn't have spaces between the branch names.
'{master, develop, feature/branchThatDontTrigger}'
should be
'{master,develop,feature/branchThatDontTrigger}'
Also if you want you can include all feature branches with feature/*.
Here's one example of that: StackOverflow answer
Related
CDK pipelines seems to only work, by default, with one branch. Am I missing something or is there a way to:
have a dev branch to deploy to the Dev account/ env
test branch deploy to Test account/env.
jons-cool-feature-branch to X account/env etc
Ideally we do not want to have to push everything to the master branch to deploy to dev / test, so that we can keep the master branch clean, tidy, and stable.
I have thought about having multiple pipelines, one for dev, one for test, and one for master, this would solve the issue, but doesn’t feel like the cleanest solution.
Are there any recommended patterns?
The AWS-prescribed best practice is to use trunk-based development.
Thus, a single pipeline cannot use multiple branches for deploying to different environments cleanly.
You should look into creating a single pipeline that would in turn create environment-specific pipelines.
Here is a relevant issue in the CDK repo:
https://github.com/aws/aws-cdk/issues/9461
Solution
Building on what #gshpychka said https://stackoverflow.com/a/69812428/12907894
A pipeline that deploys pipleines. I found lots of overcomplicated solutions online, but in the end it turned out to be quite simple.
Just adding extra pipelines, for each branch we wished to deploy.
A core pipeline that builds the branch pipelines.
Only variables that need to change between any of this:
Account ID
env name
branch name
Account Pattern
Build
Dev
Staging
Prod
core-pipeline
branch
master
Webhook -> null (so it doesn't fire on each build)
Deploys:
master-pipeline -> build account
staging-pipeline -> build account
master-pipeline
branch
master
deploys
app stack -> prod account
staging-pipeline
branch
Staging
deploys
app stack -> staging account
Codepipeline cannot branch. It is not designed to do so.
A solution is to have a multi stage pipeline that has manual approval steps in the middle if you absolutely must have multiple environments and a single pipeline.
That is
Source (Dev branch) -> Build/Deploy -> Manual Approval step -> Make use of of a Codebuild or a lambda to move your now tested code (still in the artifact chain) to your test branch for you (ie make use of a git server api to initiate the merge based on the commit message from the initial commit that started the chain -> Another Build./Deploy to your test env (can even do cross account deployment here) -> Manual Approval step -> Repeat as many times as you want until you deploy to Production.
However.... this is entirely a hack. You're better off with multiple pipelines. I would use the CDK to be able to dynamically adjust the cloudformation template for the pipeline itself to handle Dev/Prod and then simply deploy it twice, linking one to the source of Dev and one to the source of Main.
We have a CodePipeline set up to do a build, deploy to a QA ECS environment, then a manual approval step to deploy to Prod.
What gets confusing though, is when there are several builds running one after another. Several builds get deployed to QA in sequence, but then the Approval button seems to approve them one at a time, and it's not clear which build you're approving when you click on it.
What I would like to be able to do is to approve the latest build, in case
the earlier builds had issues that were fixed by the later builds. What would be the best way to accomplish that?
I had the same problem. Manual approvals are confusing since several pipeline executions can get queued and it's easy to lose track of things. I think we can blame this on CodePipeline's bad UX.
The workaround I settled with is to have two identical pipelines for the same project. They have the same source stage (same repo/branch) but different deploy stages (one deploys to QA, one deploys to prod). No more manual approval stages. The QA pipeline is set to auto-execute when changes in the source (repo/branch) are detected while the Prod pipeline needs to be manually released.
Basically, we replaced the Manual Approval with Manual Release. Manual release always releases the latest from source unlike manual approvals.
You should place the deploy and approval action in the same stage. This lets you approve exactly what you tested. Why? Because exactly one pipeline execution can be in a pipeline stage at any given time.
...approve the latest build, in case the earlier builds had issues
that were fixed by the later builds.
If you want to let later builds catch up, reject the earlier build that is waiting for approval.
One option if you don't want to have multiple pipelines is to by default disable stage transitions into your environments that required controlled releases.
When you are ready to deploy into an environment, you enable the stage transition to allow the most recent release from the previous stage to be processed and then disable the transitions again.
It's still a bit clunky, but reasonably effective once you get used to it. Having to reject each change that comes through becomes very slow and cumbersome to manage, so by disabling transitions you choose when to promote a release.
IMO, CodePipeline should have an option to automatically supersede executions if they are paused at the manual approval stage.
In the CodePipeline UI, you can see the history of Manual approvals in your pipelines' History. Click on History to see what's in progress (Manual Approvals that haven't timed out will always be in progress) and the source (git) short-sha that triggered it (if you need to narrow down to the relevant commit).
To know which Manual approval you're approving, in Pipeline view, click on View current revisions next to the Manual step (to get the Execution ID), then find the matching Execution ID in History (should be the oldest one).
Only way I found to get to the latest Approval is to hit reject n-1 times in the pipeline (where n is how many manual approvals are still in progress) until I only have 1 approval left (or until I find matching Execution ID).
Well, we can solve this problem as how you describe it with development, but it might also be a process glitch.
For example: If we have a development branch, a release branch (staging) and a master branch ( production ) we could easily solve this issue.
Development branch
Things we develop will be going through the development branch stage where we don't need the manual approval, as we don't want to check every changes. We have setup automated unit tests for that.
Release branch
This will deploy to the staging environment where we extensively test the software quality, also based on the regression tests on an acceptance chain with acceptance systems. This should prevent all the big issues before merging towards master branch. Next to that, we could also manually test the release branch on the staging environment. If this works, be happy and easily migrate towards master
Master branch
This will deploy to the production environment with a manual approval before the actual deployment is taking place, knowing for sure you will only push 1 change, being the merge from release to master, preventing the issues you've summarized in the ticket.
Another way is to develop a new AWS feature where you can uncheck or check a checkbox saying: always take the latest release, but that will not help adding value to the pipeline integration as things will be pushed without testing well enough.
We are a relatively inexperienced development team trying to do things 'the right way'. We are using Github along with AWS and CodeDeploy for multiple PHP based web applications. We are utilising Github's auto-deployment with CodeDeploy when the master branch is updated.
We have two production EC2 web servers in separate AZ's along with a single EC2 staging server.
It currently works as follows:
We write code in a branch, we push to GitHub, we merge into 'master' which then kicks off CodeDeploy to write to our staging server where we can test it. Once we have tested it we then manually kick off CodeDeploy to write to production (with the same commit ID).
The problem is, if testing brings up issues, and we have another branch waiting to be merged and tested, everything becomes backed up?
We are obviously doing something wrong. We are writing to the master branch to utilise GitHub's autodeploy, but I assumed master was only to be written to when it was ready to be deployed?
Can someone please help us and put us straight?
Thanks
Make another branch called 'livecandidate' this branch will have each of the new feature branches merged into it
Each time a feature branch is merged into 'livecandidate' pull 'livecandidate' into your Code Deploy process and install to the test machine.
If the tests pass then merge 'livecandidate' into 'master' and kick off the install to production
If the tests do not pass then unwind the merge into 'livecandidate' (assuming no dependencies on chains of changes etc)
After doing a production install or a un-merge, try the next feature
General idea is to never ever have a broken master
All problems in computer science can be solved by another level of indirection - David Wheeler
I started my Django project locally and have been using git just fine.
I got ahead of myself and copied the code to the server which instantly became out of sync with my local version. I hadn't done a branch or anything.
The two part question is what's the best structure for me to work locally, push/pull to test server and then update live server when test is solid, and how do I get it setup from where I'm at?
I've been developing with no branches in these early stages, but I'd like to instead follow standard practices for branching and merging.
I'm using NetBeans 6.8 locally for coding and I've also got GitX. So any integration tips would be helpful also but I'm comfortable doing whatever command lines are necessary.
Thanks!
James
First you should be able to have some form of communication between the git repositories you've got on your local machine, the test server and the live server. Git is very flexible in this regard so a few of the options are:
Have the test and live server pull
from your local repository.
From development push to the test and live servers on appropriate times.
From development push to production and have the test server pull from production.
Have a fourth location where you'll store your git repo and push from development to that repository and have test and live pull from there.
Either way, once you reach a stage where you'll want to try something on the test server, create a tag. On the test server checkout that tag (git checkout <tagname>) and do your testing. (And once you are satisfied that it works the way you want, you can also use that tag on production. But I guess that's pretty obvious. :) )
The intermediate step, between creating the tag and checking it out, completely depends on your setup. Using the fourth option I just mentioned you'll need to push your tag first and fetch it on the testing machine. So the whole process would look similar to this.
<development>$ git tag v1.0
<development>$ git push
<development>$ git push --tags
<testing>$ git fetch --tags
<testing>$ git checkout v1.0
<live>$ git fetch --tags
<live>$ git checkout v1.0
Optionally you can (ab)use git decribe to check which tag you've got checked out at currently.
Regarding the branching and merging: what I like to do is create a branch for every feature I'm working on. Once I complete that feature I merge it back to master. So If I need to release before a feature is done, I can just leave that feature (and every releated) commit out of the release.
But this is just one way of doing it. You can setup the workflow to suit your situation. Especially regarding the use of branches. A more complex setup is described by Vincent Driessen in his article A successful Git branching model.
Disclaimer: I'm using git almost exclusively with one authoritative repo on a server (the fourth option). I haven't personally tried the other setups I suggested...
Update to respond to the comment by iJames:
To make dev push to and test pull from a new/different repository by default from now on, see the accepted answer of this question:
$ git branch --set-upstream master origin/master
With regards to the terminology:
Push is relatively simple: it pushes your local commits to a different repository. See for instance the Git User's Manual.
Fetching does the opposite, it "will update all of the remote-tracking branches to the latest version found in the repository". (Quote from the Git User's Manual.)
The pull command not only fetches the changes in, but also merges them into the current branch. (See the example in the official Git tutorial.)
After my first question, id like to have a confirmation about the best git workflow in my case.
I have a single django project, hosted at github, and differents clones with each his own branch : customerA, customerB, demo... (think websites)
Branches share the same core but have differents data and settings (these are in gitignore)
When i work on CustomerA branch, how should i replicate some bug corrections to the other deployments ?
When i create a new general feature, i create a special branch, then merge it into my master. Then, to deploy on the 'clients', i merge the master branch into the customer branch. Is it the right way ? or should i rebase ?
# from customerA branch
git fetch origin master
git merge origin master
Also, i have created a remote branch for each customer so i can backup the customers branches to github.
It looks a very classic problem but i guess i dont use git the right way
Thanks.
Ju.
I would have a single project repo at a well-known place containing a master branch with the common code, and branches for specific deployments (e.g. customer/A customer/B demo).
Then I would have checkouts from each of these branches for each customer, for the demo server, and so on. You can let these pull automatically from their respective branch with a commit hook on the single project repo.
Every developer would have their local copy of the project repo, do local work, and then push stuff back to the single project repo.
The challenge will be to maintain the branches diverging from master and doing the regular merges so the diversion do not grow over time.
I have seen this solution describe somewhere in much more detail somewhere on the web, but I could not find it quickly again. Some blog post on using git for a staging and production web server, IIRC.
If the three sites share some 'core' code (such as a Django app) you should factor that core out into its own repo and use git submodules to include it in the other projects, rather than duplicating it.
I would have a repo called project-master or something like that and a repo for each client. Then, when you have code you need to be available to those client repos, you pull from the project-master to that repo.
Don't separate the projects in branches, separate them into different repositories.
Make the "common" code generic enough so that costumerA's copy of the common code is exactly the same as costumerB's copy of the common code.
Then, you don't have to pull or merge anything. When you update the common code, both costumerA and costumerB will get the update automagically (because they use the same common code).
By "common" code: I'm referring to the package/series-of-apps that power the websites you're developing.
I'm assuming costumerA and costumerB repositories would only include things like site-specific settings and templates.
The key here is making the "common" code generic: don't let costumerA use a "slightly modified version" of the "common" code.
Also, I'd suggest using a deployment mechanism that doesn't rely on git. git is a great source code management tool; but it's not designed (AFAIK) to be a deployment tool.