We're thinking of bringing in a couple of specialists for short-term projects. I'm trying to figure out how to allow them to effectively develop against our code base without releasing the whole code base to them.
Each project has well defined areas they need access to; primarily our main models, together with specific pieces of our app.
We've started to do a better job of breaking up the project into multiple apps within a single django project, but they all still live together in a single git repository. If you check out the repository you get everything.
What are successful strategies for arranging code and repositories such that third parties can access core models and selected functionality without having access to everything?
Note that since this is a somewhat rare need, I'd strongly prefer a setup that doesn't inconvenience our core developers - their lives should be minimally impacted by the setup.
You might try git-submodule as a way of developing each app as its own git repository while still letting developers grab the root and all apps with one "git clone". It's not totally painless though since when you do this any changes to a submodule will need to be committed there and then again in the root repository to reference the new submodule commit. This is probably inescapable, since if you want anyone beside a core developer to be able to commit to an individual app then the app's commits must be independent.
Related
I'm new to git. I've read the well-written intro book. But gee, it's still not a trivial topic. I've been bumbling around, experiencing various problems. I realized it might be because I'm unaware of workflow, and specifically, "what are the best practices for doing what I'm trying to do?"
I started out developing a django project on my win7 with Pycharm. Great way to get the initial 95% written.
But then I need to deploy it to my production machine at PythonAnywhere.
So I created a private Github repository, pushed my win7 codebase to github.
Then in pythonAnywhere, I cloned the github repository.
For now, no others work on this project. It will not be released to the public.
Now that the server is running on PythonAnywhere, I still need to tweak settings, which is best done on the PythonAnywhere codebase side. But there are other improvements (new pages, or views) that I'd rather do inside Pycharm IDE on my win7 than in vim on python anywhere.
So I've been kind of clumsily pushing and fetching these changes. It's been kind of ham-handed, and I've managed to lose some minor changes through ignorance.
So I'm wondering if anyone can point to a relatively simple workflow that would handle the various tasks I mentioned:
1) improving functionality of the site (best done in Pycharm IDE)
2) production server issues and tweaks (best done on PythonAnywhere)
3) keeping everythign safely backed-up on Github
The other issue is that I have another django app that I want to build. It's easiest to temporarily hang it off the django project I've already built. But I'd prefer to keep it in its own repository.
So I have Original_Project, Original_App stored in Original_Repository
I want to make new_app, and have it, for the time being, run in Original_Project, but I want to version control it in New_Repository.
I think/hope that I could put a .gitignore in the Original_Repository, saying ignore the new_app/ Then I git init new_app/ as its own repository. Is that sound or mad?
You should avoid editing your code on the production server as much as possible, and never commit from the production server. If you end up having to tweaks things on the server (you shouldn't but well, shit happens and sometimes it's indeed easier to first get the code back to work on the server), then once it's working manually report your edits to your local repo, clear up the changes on the server and deploy the fixed code again. Here the github repo should be considered as the "master" repository for deployments, ie you work on your local repo, push to github, and on the server pull from github. This make sure you keep the github repo in sync.
wrt/ the "improving functionality" (aka "features") vs "server issues and tweaks" (aka "hotfixes"), git flow is a (mostly) sane workflow IMHO but that's a bit opinion-based here (some dislike it and have sensible arguments too).
Finally if you want to factor out one of your apps, the best is to have it in it's own (github) repo with all the proper python packaging stuff and make it a requirement of your main project. On your local dev environment you install it as an editable package, and for the production setup you install it as normal package pinned to the last stable version. Note that in both cases I assume you're using virtualenvs (and if you dont, well that's the very first issue you should address).
Update:
What are the downsides of of editing directly on the production server and committing from the production server?
Well quite simply a production server is not the place for coding - "production" means that you have users trying to do something with your website and they don't want to have the site breaking on them, their data lost or whatever because you are "tweaking" things. You should only deploy stable, well tested code on production, and the one and only one case where editing anything on the server might be a last resort option is when it's already broken and you want to get it back online asap whatever it takes (case of "first make it work, then make it clean").
Point is, I'm a professional developer working on projects that are business criticals and a broken site is not an option, so I'm very strict on this - but even if it's a hobby project, your users deserve some respect (at least if you expect to see them back).
A proper production chain actually involves at least three environments: your local dev environment, a staging server (which should closely mirror the production server - system, system package versions, configurations etc etc) to test out / showcase / eventually do minor config tweak, and the production server which should only ever see stable tested code.
I have always struggled with git, knowing it well enough to get thigs working, but never being sure I am doing thing well.
I would suggest installing git flow (it is probably available in your package manager if you are on Linux). Its a set of extensions that simplify a standard git worklfow. Since using it, this has pretty much been all the documentation I have needed.
https://danielkummer.github.io/git-flow-cheatsheet/
I am a bit new to heroku and git, but here goes.
I was advised to separate my static files permissions from my backend code permissions. This being the case, I could give designers free reign over making changes to front end stuff without worrying about doing any damage to any other backend code and I could give access to the back end people who need it separately.
What exactly is the best way to do this? How do I even facilitate something like this? I am running a django project on heroku currently.
PS: although the static folder lives in a separate area, I also have a template folders running along side some apps. I don't know if this will complicate things or not.
Thanks in advance.
If you're using git, other than splitting repositories, which will significantly complicate your deployment process, there's no way to have custom permissions.
I'm getting set up with proper distribution version control (yes, overdue) on a large Django environment with lot's of reusable apps and lot's of projects.
What's the right way to do this?
Clone each app you need within each project, to allow you to make changes to the app without worrying about breaking anything.
Have one copy of each version controlled application to avoid having multiple copies of the code, each in its own repository.
Or is there a better way?
Thanks.
Edit for clarity: These are in house apps that are reused from project to project.
In my opinion the best practice is to keep all your apps as one library/package. You can have versions/snapshots (e.g. tags in hg) and branches and you should definitely create and configure setup.py file.
If the app are reusables, you must create a egg in pypi. These have releases. For each project, you could use one or the other releases.
See for example this package.
To deploy the projects both in local as in the server, you can use buildout (very recomended)
I am trying to work out a good way to run a staging server and a production server for hosting multiple Coldfusion sites. Each site is essentially a fork of a repo, with site specific changes made to each. I am looking for a good way to have this staging server move code (upon QA approval) to the production server.
One fanciful idea involved compiling the sites each into EAR files to be run on the production server, but I cannot seem to wrap my head around Coldfusion archives, plus I cannot see any good way of automating this, especially the deployment part.
What I have done successfully before is use subversion as a go between for a site, where once a site is QA'd the code is committed and then the production server's working directory would have an SVN update run, which would then trigger a code copy from the working directory to the actual live code. This worked fine, but has many moving parts, and still required some form of server access to each server to run the commits and updates. Plus this worked for an individual site, I think it may be a nightmare to setup and maintain this architecture for multiple sites.
Ideally I would want a group of developers to have FTP access with the ability to log into some control panel to mark a site for QA, and then have a QA person check the site and mark it as stable/production worthy, and then have someone see that a site is pending and click a button to deploy the updated site. (Any of those roles could be filled by the same person mind you)
Sorry if that last part wasn't so much the question, just a framework to understand my current thought process.
Agree with #Nathan Strutz that Ant is a good tool for this purpose. Some more thoughts.
You want a repeatable build process that minimizes opportunities for deltas. With that in mind:
SVN export a build.
Tag the build in SVN.
Turn that export into a .zip, something with an installer, etc... idea being one unit to validate with a set of repeatable deployment steps.
Send the build to QA.
If QA approves deploy that build into production
Move whole code bases over as a build, rather than just changed files. This way you know what's put into place in production is the same thing that was validated. Refactor code so that configuration data is not overwritten by a new build.
As for actual production deployment, I have not come across a tool to solve the multiple servers, different code bases challenge. So I think you're best served rolling your own.
As an aside, in your situation I would think through an approach that allows for a standardized codebase, with a mechanism (i.e. an API) that allows for the customization you're describing. Otherwise managing each site as a "custom" project is very painful.
Update
Learning Ant: Ant in Action [book].
On Source Control: for the situation you describe, I would maintain a core code base and overlays per site. Export core, then site specific over it. This ensures any core updates that site specific changes don't override make it in.
Call this combination a "build". Do builds with Ant. Maintain an Ant script - or perhaps more flexibly an ant configuration file - per core & site combination. Track version number of core and site as part of a given build.
If your software is stuffed inside an installer (Nullsoft Install Shield for instance) that should be part of the build. Otherwise you should generate a .zip file (.ear is a possibility as well, but haven't seen anyone actually do this with CF). Point being one file that encompasses the whole build.
This build file is what QA should validate. So validation includes deployment, configuration and functionality testing. See my answer for deployment on how this can flow.
Deployment:
If you want to automate deployment QA should be involved as well to validate it. Meaning QA would deploy / install builds using the same process on their servers before doing a staing to production deployment.
To do this I would create something that tracks what server receives what build file and whatever credentials and connection information is necessary to make that happen. Most likely via FTP. Once transferred, the tool would then extract the build file / run the installer. This last piece is an area I would have to research as to how it's possible to let one server run commands such as extraction or installation remotely.
You should look into Ant as a migration tool. It allows you to package your build process with a simple XML file that you can run from the command line or from within Eclipse. Creating an automated build process is great because it documents the process as well as executes it the same way, every time.
Ant can handle zipping and unzipping, copying around, making backups if needed, working with your subversion repository, transferring via FTP, compressing javascript and even calling a web address if you need to do something like flush the application memory or server cache once it's installed. You may be surprised with the things you can do with Ant.
To get started, I would recommend the Ant manual as your main resource, but look into existing Ant builds as a good starting point to get you going. I have one on RIAForge for example that does some interesting stuff and calls a groovy script to do some more processing on my files during the build. If you search riaforge for build.xml files, you will come up with a great variety of them, many of which are directly for ColdFusion projects.
After my first question, id like to have a confirmation about the best git workflow in my case.
I have a single django project, hosted at github, and differents clones with each his own branch : customerA, customerB, demo... (think websites)
Branches share the same core but have differents data and settings (these are in gitignore)
When i work on CustomerA branch, how should i replicate some bug corrections to the other deployments ?
When i create a new general feature, i create a special branch, then merge it into my master. Then, to deploy on the 'clients', i merge the master branch into the customer branch. Is it the right way ? or should i rebase ?
# from customerA branch
git fetch origin master
git merge origin master
Also, i have created a remote branch for each customer so i can backup the customers branches to github.
It looks a very classic problem but i guess i dont use git the right way
Thanks.
Ju.
I would have a single project repo at a well-known place containing a master branch with the common code, and branches for specific deployments (e.g. customer/A customer/B demo).
Then I would have checkouts from each of these branches for each customer, for the demo server, and so on. You can let these pull automatically from their respective branch with a commit hook on the single project repo.
Every developer would have their local copy of the project repo, do local work, and then push stuff back to the single project repo.
The challenge will be to maintain the branches diverging from master and doing the regular merges so the diversion do not grow over time.
I have seen this solution describe somewhere in much more detail somewhere on the web, but I could not find it quickly again. Some blog post on using git for a staging and production web server, IIRC.
If the three sites share some 'core' code (such as a Django app) you should factor that core out into its own repo and use git submodules to include it in the other projects, rather than duplicating it.
I would have a repo called project-master or something like that and a repo for each client. Then, when you have code you need to be available to those client repos, you pull from the project-master to that repo.
Don't separate the projects in branches, separate them into different repositories.
Make the "common" code generic enough so that costumerA's copy of the common code is exactly the same as costumerB's copy of the common code.
Then, you don't have to pull or merge anything. When you update the common code, both costumerA and costumerB will get the update automagically (because they use the same common code).
By "common" code: I'm referring to the package/series-of-apps that power the websites you're developing.
I'm assuming costumerA and costumerB repositories would only include things like site-specific settings and templates.
The key here is making the "common" code generic: don't let costumerA use a "slightly modified version" of the "common" code.
Also, I'd suggest using a deployment mechanism that doesn't rely on git. git is a great source code management tool; but it's not designed (AFAIK) to be a deployment tool.