After my first question, id like to have a confirmation about the best git workflow in my case.
I have a single django project, hosted at github, and differents clones with each his own branch : customerA, customerB, demo... (think websites)
Branches share the same core but have differents data and settings (these are in gitignore)
When i work on CustomerA branch, how should i replicate some bug corrections to the other deployments ?
When i create a new general feature, i create a special branch, then merge it into my master. Then, to deploy on the 'clients', i merge the master branch into the customer branch. Is it the right way ? or should i rebase ?
# from customerA branch
git fetch origin master
git merge origin master
Also, i have created a remote branch for each customer so i can backup the customers branches to github.
It looks a very classic problem but i guess i dont use git the right way
Thanks.
Ju.
I would have a single project repo at a well-known place containing a master branch with the common code, and branches for specific deployments (e.g. customer/A customer/B demo).
Then I would have checkouts from each of these branches for each customer, for the demo server, and so on. You can let these pull automatically from their respective branch with a commit hook on the single project repo.
Every developer would have their local copy of the project repo, do local work, and then push stuff back to the single project repo.
The challenge will be to maintain the branches diverging from master and doing the regular merges so the diversion do not grow over time.
I have seen this solution describe somewhere in much more detail somewhere on the web, but I could not find it quickly again. Some blog post on using git for a staging and production web server, IIRC.
If the three sites share some 'core' code (such as a Django app) you should factor that core out into its own repo and use git submodules to include it in the other projects, rather than duplicating it.
I would have a repo called project-master or something like that and a repo for each client. Then, when you have code you need to be available to those client repos, you pull from the project-master to that repo.
Don't separate the projects in branches, separate them into different repositories.
Make the "common" code generic enough so that costumerA's copy of the common code is exactly the same as costumerB's copy of the common code.
Then, you don't have to pull or merge anything. When you update the common code, both costumerA and costumerB will get the update automagically (because they use the same common code).
By "common" code: I'm referring to the package/series-of-apps that power the websites you're developing.
I'm assuming costumerA and costumerB repositories would only include things like site-specific settings and templates.
The key here is making the "common" code generic: don't let costumerA use a "slightly modified version" of the "common" code.
Also, I'd suggest using a deployment mechanism that doesn't rely on git. git is a great source code management tool; but it's not designed (AFAIK) to be a deployment tool.
Related
New user here...
Installed D8+Civi by building a composer based git repo for the platform then stamping out a few test sites.
It worked really well.
But now I am at the point of realizing I missed a few modules and I want to add some themes to apply to the sites.
I can easily to it in the git which was used to define the platform. But what is the proper way to manage the central platform data and files that are then used for the x number of sites.
I know the docs try to discuss this be a tutorial walk-through would be very helpful.
As a guess, I could make the central platform files a git clone and pull down clones for the new stuff. But if there was a need for an database updates that wouldn't get done.
Ideas?
Thanks
It's not clear what you mean by "central platform data".
If you mean assets that are relevant for the entire platform, that can apply to all of the sites, you would do the following:
Add anything new to Git and push it.
Create a new platform to match the latest code in Git.
Ran a Migrate task on the old platform to migrate the sites to the new one.
Database schema updates happen automatically.
The sites will now be running on the new codebase.
If you're talking about site-specific assets that you don't want to be included in the platform's code, then you can enable Git for sites with the Aegir Hosting Git module.
It allows you to deploy site-specific Git repositories.
However, I don't recommend using that module for platforms, just sites, because it allows you to git pull on Production sites, which is a terrible idea. For that, see Aegir Deploy.
Both of these modules ship with Aegir so you won't need to install them. Some of the Hosting Git features may need to be enabled, however.
I am working with a client that demands multi-stage server setup: development server, stage server and production/live server.
Stage should be as stable as it can be to test all those new features we develop at the development server and take this to the live server in the end.
We use git and github for version controlling. I use Ubuntu server edition as the OS.
The problem is, I have never working in such multi-stage server plan. What software/projects would you recommend to do a proper way of handling such setup, especially deployment and moving a new feature developed to the stage and then to the live server ?
We use two different methods of moving code from environment to environment. The first is to use branches and triggers with our source control system (mercurial in our case, though you can do the same thing with git). The other, is to use fabric, a python library for executing shell code across a number of servers.
Using source control, you can have several main branches, like production development staging. Say you want to move a new feature into staging. I'll explain in terms of mercurial, but you can port the commands over to git and it should be fine.
hg update staging
hg merge my-new-feature
hg commit -m 'my-new-feature > staging'
hg push
You then have your remote source control server push to all of your web servers using a trigger. A trigger on each web server will then do an update and reload the web server.
To move from staging to production, it's just as easy.
hg update production
hg merge staging
hg commit -m 'staging > production'
hg push
It's not the nicest method of deployment, and it makes rolling back quite hard. But it's quick and easy to set up, and still a lot better than manually deploying each change to each server.
I won't go through fabric, as it can get quite involved. You should read their documentation so you understand what it is capable of. There are plenty of tutorials around for fabric and django. I highly recommend the fabric route as it gives you lots more control, and only involves writing some python.
There is a nice branching model for git (as it is also used by github itself for example). You can easily apply this branching model using git-flow, which is a git extension that enables you to apply some high level repository operations that fit into this model. There's also a nice blogpost about this.
I do not know what exactly you want to automize in your deployment workflow, but if you apply the model mentioned above, most of the correct version handling is done by git.
To add some further automatic processing to this, fabric is a simple but great tool, and you will find many tutorials about its usage (also in combination with git).
For handling python dependencies using virtualenv and pip is for sure a very good way to go.
If you need something more complex,eg. to handle more than one django instance on one machine and for handling system wide dependencies etc checkout puppet or chef.
Try Gondor.io or Ep.io, they both make it pretty easy (gondor especially excels in this area) to have two+ instances with very similar code, from your VCS -- and to move data back and forth. (if you need an invite, ask either in IRC, but if I recall, they're both open now)
We're thinking of bringing in a couple of specialists for short-term projects. I'm trying to figure out how to allow them to effectively develop against our code base without releasing the whole code base to them.
Each project has well defined areas they need access to; primarily our main models, together with specific pieces of our app.
We've started to do a better job of breaking up the project into multiple apps within a single django project, but they all still live together in a single git repository. If you check out the repository you get everything.
What are successful strategies for arranging code and repositories such that third parties can access core models and selected functionality without having access to everything?
Note that since this is a somewhat rare need, I'd strongly prefer a setup that doesn't inconvenience our core developers - their lives should be minimally impacted by the setup.
You might try git-submodule as a way of developing each app as its own git repository while still letting developers grab the root and all apps with one "git clone". It's not totally painless though since when you do this any changes to a submodule will need to be committed there and then again in the root repository to reference the new submodule commit. This is probably inescapable, since if you want anyone beside a core developer to be able to commit to an individual app then the app's commits must be independent.
I started my Django project locally and have been using git just fine.
I got ahead of myself and copied the code to the server which instantly became out of sync with my local version. I hadn't done a branch or anything.
The two part question is what's the best structure for me to work locally, push/pull to test server and then update live server when test is solid, and how do I get it setup from where I'm at?
I've been developing with no branches in these early stages, but I'd like to instead follow standard practices for branching and merging.
I'm using NetBeans 6.8 locally for coding and I've also got GitX. So any integration tips would be helpful also but I'm comfortable doing whatever command lines are necessary.
Thanks!
James
First you should be able to have some form of communication between the git repositories you've got on your local machine, the test server and the live server. Git is very flexible in this regard so a few of the options are:
Have the test and live server pull
from your local repository.
From development push to the test and live servers on appropriate times.
From development push to production and have the test server pull from production.
Have a fourth location where you'll store your git repo and push from development to that repository and have test and live pull from there.
Either way, once you reach a stage where you'll want to try something on the test server, create a tag. On the test server checkout that tag (git checkout <tagname>) and do your testing. (And once you are satisfied that it works the way you want, you can also use that tag on production. But I guess that's pretty obvious. :) )
The intermediate step, between creating the tag and checking it out, completely depends on your setup. Using the fourth option I just mentioned you'll need to push your tag first and fetch it on the testing machine. So the whole process would look similar to this.
<development>$ git tag v1.0
<development>$ git push
<development>$ git push --tags
<testing>$ git fetch --tags
<testing>$ git checkout v1.0
<live>$ git fetch --tags
<live>$ git checkout v1.0
Optionally you can (ab)use git decribe to check which tag you've got checked out at currently.
Regarding the branching and merging: what I like to do is create a branch for every feature I'm working on. Once I complete that feature I merge it back to master. So If I need to release before a feature is done, I can just leave that feature (and every releated) commit out of the release.
But this is just one way of doing it. You can setup the workflow to suit your situation. Especially regarding the use of branches. A more complex setup is described by Vincent Driessen in his article A successful Git branching model.
Disclaimer: I'm using git almost exclusively with one authoritative repo on a server (the fourth option). I haven't personally tried the other setups I suggested...
Update to respond to the comment by iJames:
To make dev push to and test pull from a new/different repository by default from now on, see the accepted answer of this question:
$ git branch --set-upstream master origin/master
With regards to the terminology:
Push is relatively simple: it pushes your local commits to a different repository. See for instance the Git User's Manual.
Fetching does the opposite, it "will update all of the remote-tracking branches to the latest version found in the repository". (Quote from the Git User's Manual.)
The pull command not only fetches the changes in, but also merges them into the current branch. (See the example in the official Git tutorial.)
I'm a web developer working on my own using django, and I'm trying to get my head round how best to deploy sites using mercurial. What I'd like to have is to be able to keep one repository that I can use for both production and development work. There will always be some differences between production/development (e.g. they might use different databases, development will always have debug turned on) but by and large they will be in sync. I'd also like to be able to make changes directly on the production server (tidying up html or css, simple bugfixes etc.).
The workflow that I intend to use for doing this is as follows:
Create 2 branches, prod and dev (all settings initially set to production settings)
Change settings.py and a few other things in the dev branch. So now I've got 2 heads, and from now on the repository will always have 2 heads.
(On dev machine) Make changes to dev, then use 'hg transplant' to copy relevant changesets to production.
push to master repository
(On production server) Pull from master repo, update to prod head
Note: you can also make changes straight to prod so long as you transplant the changes into dev.
This workflow has the drawback that whenever you make a change, not only do you have to commit it to whichever branch you make the change on, you also have to transplant it to the other branch. Is there a more sensible way of doing what I want here, perhaps using patches? Or failing that, is there a way of automating the commit process to automatically transplant the changeset to the other branch, and would this be a good idea?
I'd probably use Mercurial Queues for something like this. Keep the main repository as the development version, and have a for-production patch that makes any necessary changes for production.
Here are two possible solutions one using mercurial and one not using mercurial:
Use the hostname to switch between prod and devel. We have a single check at the top of our settings file that looks at the SERVER_NAME environment variable. If it's www.production.com it's the prod DB and otherwise it picks a specified or default dev/test/stage DB.
Using Mercurial, just have a clone that's dev and a clone that's prod, make all changes in dev, and at deploy time pull from dev to prod. After pulling you'll have 2 heads in prod diverging from a single common ancestor (the last deploy). One head will have a single changeset containing only the differences between dev and prod deployments, and the other will have all the new work. Merge them in the prod clone, selecting the prod changes on conflict of course, and you've got a deployable setup, and are ready to do more work on 'dev'. No need to branch, transplant, or use queues. So long as you never pull that changeset with the prod settings into 'dev' it will always need a merge after pulling from dev, and if it's just a few lines there's not much to do.
I've solved this with local settings.
Append to settings.py:
try:
from local_settings import *
except ImportError:
pass
touch local_settings.py
Add ^local_settings.py$ to your .hgignore
Each deploy I do has it's own local settings (typically different DB stuff and different origin email addresses).
PS: Only read the "minified versions of javascript portion" later. For this, I would suggest a post-update hook and a config setting (like JS_EXTENSION).
Example (from the top of my head! not tested, adapt as necessary):
Put JS_EXTENSION = '.raw.js' in your settings.py file;
Put JS_EXTENSION = '.mini.js' in your local_settings.py file on the production server;
Change JS inclusion from:
<script type="text/javascript" src="blabla.js"></script>
To:
<script type="text/javascript" src="blabla{{JS_EXTENSION}}"></script>
Make a post-update hook that looks for *.raw.js and generates .mini.js (minified versions of raw);
Add .mini.js$ to your .hgignore
Perhaps try something like this: (I was just thinking about this issue, in my case it's a sqlite database)
Add settings.py to .hgignore, to keep it out of the repository.
Take your settings.py files from the two separate branches and move them into two separate files, settings-prod.py and settings-dev.py
Create a deploy script which copies the appropriate settings-X file to settings.py, so you can deploy either way.
If you have a couple of additional files, do the same thing for them. If you have a lot of files but they're all in the same directory by themselves, you could just create a pair of directories: production and development, and then either copy or symlink the appropriate one into a deploy directory.
If you did something like this, you could dispense with the need for branching your repository.
I actually do this using named branches and straight merging instead of transplanting (which is more reliable, IMO). This usually works, although sometimes (when you've edited the different files on the other branch), you'll need to pay attention not to remove the differences again when you're merging.
So it works great if you're not changing the different files much.