G'day all ... I have been looking at stackoverflow about the use of a local or cluster nexus repository. Anyway I've had a Nexus repository on my PC and my laptop for a number of years with little bother.
I think more projects are using Nexus themselves and I've had three or four issues just in the last few weeks (since December) where there can be clashes with any artefact from an id of: "central". I'm seeing problems like this one more often:
Nexus won't download artifacts from Central
On my laptop, I seem to have resolved problems to date, by naming the main repository (something like) "nexus-local". But not on my desktop workstation. Strange.
How strange? Well an empty 'archetype' project for Vert.X with the same attributes compiles fine on the laptop and fails on the workstation, complaining about not finding something from a "central" named repository.
Upon inspection, I noticed that even though I've renamed the workstation repository as "nexus-local" there's some internal ID that remains "central".
There are Nexus repository settings files, e.g. nexus.xml, showing the internal(??) central name/Id. So the question viewers ...
Does the local mirror have to be called "central"?
If not, how does one rename it 'responsibly'?
Alternatively, is there a simple demo for a maven/nexus setup cookbook some place that doesn't require me to read 3 x books first and compiling the knowledge for a simple-solo set-up?
Would dropping Nexus altogether and restarting the server 'fix' this kind of issue?
Maven 3 has a built in repository called "central" and you need to override this in your settings.xml as described in the Nexus documentation. You then combine this with a repository group of any name and use that as a mirrorOf * (including central).
The repository group can contain any repositoy you like as well. The default is that it contains a proxy repository of the Central Repository and the three hosted repositories releases, snapshots and thirdparty.
If you need more .. you just add them to the group.
And if you are looking for a simple step by step example .. check out the Nexus eval guide chapter about proxy and publish and the used example project.
Related
New user here...
Installed D8+Civi by building a composer based git repo for the platform then stamping out a few test sites.
It worked really well.
But now I am at the point of realizing I missed a few modules and I want to add some themes to apply to the sites.
I can easily to it in the git which was used to define the platform. But what is the proper way to manage the central platform data and files that are then used for the x number of sites.
I know the docs try to discuss this be a tutorial walk-through would be very helpful.
As a guess, I could make the central platform files a git clone and pull down clones for the new stuff. But if there was a need for an database updates that wouldn't get done.
Ideas?
Thanks
It's not clear what you mean by "central platform data".
If you mean assets that are relevant for the entire platform, that can apply to all of the sites, you would do the following:
Add anything new to Git and push it.
Create a new platform to match the latest code in Git.
Ran a Migrate task on the old platform to migrate the sites to the new one.
Database schema updates happen automatically.
The sites will now be running on the new codebase.
If you're talking about site-specific assets that you don't want to be included in the platform's code, then you can enable Git for sites with the Aegir Hosting Git module.
It allows you to deploy site-specific Git repositories.
However, I don't recommend using that module for platforms, just sites, because it allows you to git pull on Production sites, which is a terrible idea. For that, see Aegir Deploy.
Both of these modules ship with Aegir so you won't need to install them. Some of the Hosting Git features may need to be enabled, however.
I'm new to git. I've read the well-written intro book. But gee, it's still not a trivial topic. I've been bumbling around, experiencing various problems. I realized it might be because I'm unaware of workflow, and specifically, "what are the best practices for doing what I'm trying to do?"
I started out developing a django project on my win7 with Pycharm. Great way to get the initial 95% written.
But then I need to deploy it to my production machine at PythonAnywhere.
So I created a private Github repository, pushed my win7 codebase to github.
Then in pythonAnywhere, I cloned the github repository.
For now, no others work on this project. It will not be released to the public.
Now that the server is running on PythonAnywhere, I still need to tweak settings, which is best done on the PythonAnywhere codebase side. But there are other improvements (new pages, or views) that I'd rather do inside Pycharm IDE on my win7 than in vim on python anywhere.
So I've been kind of clumsily pushing and fetching these changes. It's been kind of ham-handed, and I've managed to lose some minor changes through ignorance.
So I'm wondering if anyone can point to a relatively simple workflow that would handle the various tasks I mentioned:
1) improving functionality of the site (best done in Pycharm IDE)
2) production server issues and tweaks (best done on PythonAnywhere)
3) keeping everythign safely backed-up on Github
The other issue is that I have another django app that I want to build. It's easiest to temporarily hang it off the django project I've already built. But I'd prefer to keep it in its own repository.
So I have Original_Project, Original_App stored in Original_Repository
I want to make new_app, and have it, for the time being, run in Original_Project, but I want to version control it in New_Repository.
I think/hope that I could put a .gitignore in the Original_Repository, saying ignore the new_app/ Then I git init new_app/ as its own repository. Is that sound or mad?
You should avoid editing your code on the production server as much as possible, and never commit from the production server. If you end up having to tweaks things on the server (you shouldn't but well, shit happens and sometimes it's indeed easier to first get the code back to work on the server), then once it's working manually report your edits to your local repo, clear up the changes on the server and deploy the fixed code again. Here the github repo should be considered as the "master" repository for deployments, ie you work on your local repo, push to github, and on the server pull from github. This make sure you keep the github repo in sync.
wrt/ the "improving functionality" (aka "features") vs "server issues and tweaks" (aka "hotfixes"), git flow is a (mostly) sane workflow IMHO but that's a bit opinion-based here (some dislike it and have sensible arguments too).
Finally if you want to factor out one of your apps, the best is to have it in it's own (github) repo with all the proper python packaging stuff and make it a requirement of your main project. On your local dev environment you install it as an editable package, and for the production setup you install it as normal package pinned to the last stable version. Note that in both cases I assume you're using virtualenvs (and if you dont, well that's the very first issue you should address).
Update:
What are the downsides of of editing directly on the production server and committing from the production server?
Well quite simply a production server is not the place for coding - "production" means that you have users trying to do something with your website and they don't want to have the site breaking on them, their data lost or whatever because you are "tweaking" things. You should only deploy stable, well tested code on production, and the one and only one case where editing anything on the server might be a last resort option is when it's already broken and you want to get it back online asap whatever it takes (case of "first make it work, then make it clean").
Point is, I'm a professional developer working on projects that are business criticals and a broken site is not an option, so I'm very strict on this - but even if it's a hobby project, your users deserve some respect (at least if you expect to see them back).
A proper production chain actually involves at least three environments: your local dev environment, a staging server (which should closely mirror the production server - system, system package versions, configurations etc etc) to test out / showcase / eventually do minor config tweak, and the production server which should only ever see stable tested code.
I have always struggled with git, knowing it well enough to get thigs working, but never being sure I am doing thing well.
I would suggest installing git flow (it is probably available in your package manager if you are on Linux). Its a set of extensions that simplify a standard git worklfow. Since using it, this has pretty much been all the documentation I have needed.
https://danielkummer.github.io/git-flow-cheatsheet/
The general guidance appears to be to install Sitecore into one folder, e.g. D:\Websites\MyWebSite and then create your Visual Studio project in a separate folder, e.g. C:\Projects\MyWebProject. You would then publish your custom code into the Sitecore folder from Visual Studio (This video explains what I’m describing https://www.youtube.com/watch?v=i3Mwcphtz4w around 13 mins in).
I have the following questions:-
Do people only store their Visual Studio project in source countrol and not the Sitecore code?
The publish option from VS into the Sitecore folder only has options for adding files or deleting anything not in the VS project. How would files removed from the VS project ever get deleted without doing it manually?
We use web-deploy to publish sites to staging and live environments. In this scenario would you publish from your VS project or would you set up a way to publish the Sitecore folder (if so how)?
Is this actually a good set up to have or do you do something different?
I did a lot of research on this when we started Sitecore development a couple of years ago. I remember reading a post from Sean Kearney that made a lot of sense to me: http://seankearney.com/post/Visual-Studio-Projects-and-Sitecore
We ended up using this approach for both large and small scale projects and it has been great. You will also want to look at a couple of other tools:
Team Development for Sitecore (TDS) from Hedgehog Development (http://www.hhogdev.com/products/team-development-for-sitecore/overview.aspx)
CopySauce from Igloo (http://www.igloo.com.au/blog/copysauce-igloos-sitecore-development-utility/)
SitecoreRocks for Visual Studio
So to answer your questions:
All of your code and some of the Sitecore items are stored in source control. The approach you want to take is to only store new Sitecore items (layouts, sublayouts, templates, etc) that you create along with any items you may need to customize. You do not need to store all of the sitecore source, content or modules...just what you would need to reapply to get a fresh environment up-to-date. You can manage this manually but a tool like TDS makes this MUCH easier.
We use TDS to manage the publish/deploy to each of our environments. TDS has configurable settings for handling items that have been deleted, including the ability to move it to the Sitecore recycle bin or simply remove it. You have to be careful with this but it does work.
We use a separate build environment to assemble and run deployments using TDS and Jenkins. Basically, all of the code is retrieved from the source control system to the Sitecore server and built using MSBuild and TDS. In most cases we use a webdeploy directly to the Sitecore webroot, but for production we build TDS packages and then run them on each Content Delivery Server
We have used this setup for 7 sitecore projects so far and I am very happy with how it has worked out. We have questioned whether TDS is worth the license fee but the answer always comes back as a yes. The alternative is not very appealing for our development staff and time savings far out-weigh the costs.
Everything is stored in Source Control!... just not always in the same area as they reside on the web server. Storing the Sitecore folder in source control is a good idea as there are changes that you will have as you install modules, but you do NOT add the Sitecore folder as part of your solution/project and should really be there to pull from if need be and not something that is even tracked/monitored.
Once Sitecore is installed, create a new project that resides in the website folder and only add things like the properties folder, layouts, xml and other folders that you want. I don't even include the app_config in my project. Oh and to be clear, it's probably best to just keep the Sitecore folder as a sort of reference folder in your source control but not as part of your website trunk. We have it on the ignore list for website folder in source control. However, that being said, keep in mind that you will NEED to have it in your website folder.
Technically speaking, the recommended approach is to install Sitecore on to the server itself as a stand alone empty instance.. like using the installer with the client mode (not full) so that you get the framework for an empty site in place. Then you can create the deployment package/packages/whatever and it will all be your own code. You should really never have to mess with changing/removing the base Sitecore file system manually.
See above. Generally speaking, unless you have a reason to do so: install Sitecore as an empty instance... then manage your code/files via deployment and just leave the Sitecore folder files alone. You will have very little reason to ever touch them or the Sitecore folder itself outside of an upgrade.
Adding Sitecore itself to source control should be avoided, since you won't be deploying Sitecore as part of your implementation. For modifications to Sitecore itself, you would need a way of handling those inside your implementation, but the config patch system and other mechanisms provide the means for this.
Redundant files in the web site folder will only be a real problem in your development environment. When publishing to a demo environment or to a live environment, you will only publish the material that you actually want. And the deployment-based setup opens up the possibility of always starting from a clean Sitecore installation - as long as you include your Sitecore modifications as part of your implementation (which is not covered in the video). So there is little risk of this being a problem in real life, and the development method in the video makes eliminating this risk entirely possible.
The Sitecore installation should be handled outside of the deployment of your implementation.
It's a good setup, because the method in the video is the method Sitecore recommends for development, and it is also the method Sitecore teaches to developers in development courses. The most obvious advantages of this method are
Clean separation between your web site implementation and the Sitecore installation. There is no risk of accidentally mangling the Sitecore installation, and there is no risk of forgetting unmanaged manual modifications to Sitecore that are needed to run your site. This separation is hard to accomplish if you're not using the method in the video.
By using publishing to deploy your implementation, you know that your implementation is deployable on top of a clean Sitecore installation - and works. This means when deploying to a production or demo server in the future, things will work the same and there will be no surprises. This is very hard to be confident about if you're not using the method in the video.
To test your implementation on a different version of Sitecore, you can just deploy to a clean installation of a different version. This is very hard to test if you're not using the method in the video.
There is sample source code for the video on GitHub, along with instructions on how to set up the development environment, including the publishing parts. This sample source directly and indirectly answers some of your questions.
I'm interested in figuring out the best practice way of organising Django apps on a server.
Where do you place Django code? The (old now) Almanac says /home/django/domains/somesitename.com/ but I've also seen things placed in /opt/apps/somesitename/ . I'm thinking that the /opt/ idea sounds better as it's not global, but I've not seen opt before, and presumably it might be better for apps to go in a site specific deployer users home dir.
Would you recommend having one global deployer user, one user per site, or one per site-env (eg, sitenamelive, sitenamestaging). I'm thinking one per site.
How do you version your config files? I currently put them in an /etc/ folder at top level of source control. eg, /etc/nginc/somesite-live.conf.
How do you do provision your servers and do the deployment? I've resisted Chef and Puppet for years on the hope of something Python based. Silver Lining doesn't seem ready yet, and I have big hopes for Patchwork (https://github.com/fabric/patchwork/). Currently we're just using some custom Fabric scripts to deploy, but the "server provisioning" is handled by a bash script and some manual steps for adding keys and creating users. I'm about to investigate Silk Deployment (https://bitbucket.org/btubbs/silk-deployment) as it seems closest to our setup.
Thanks!
I think there would have to be more information on what kinds of sites you are deploying: there would be differences based on the relations between the sites, both programatically and 'legally' (as in a business relation):
Having an system account per 'site' can be handy if the sites are 'owned' by different people - if you are a web designer or programmer with a few clients, then you might benefit from separation.
If your sites are related, i.e. a forum site, a blog site etc, you might benefit from a single deployment system (like ours).
for libraries, if they're hosted on reputable sources (pypy, github etc), its probably ok to leave them there and deploy from them - if they're on dodgy hosts which are up or down, we take a copy and put them in a /thirdparty folder in our git repo.
FABRIC
Fabric is amazing - if its setup and configured right for you:
We have a policy here which means nobody ever needs to log onto a server (which is mostly true - there are occasions where we want to look at the raw nginx log file, but its a rarity).
We've got fabric configured so that there are individual functional blocks (restart_nginx, restart_uwsgi etc), but also
higher level 'business' functions which run all the little blocks in the right order - for us to update all our servers we meerly type 'fab -i secretkey live deploy' - the live sets the settings for the live servers, and deploy ldeploys (the -i is optional if you have your .ssh keys set up right)
We even have a control flag that if the live setting is used, it will ask 'are you sure' before performing the deploy.
Our code layout
So our code base layout looks a bit like this:
/ <-- folder containing readme file etc
/bin/ <-- folder containing nginx & uwsgi binaries (!)
/config/ <-- folder containing nginx config and pip list but also things like pep8 and pylint configs
/fabric/ <-- folder containing fabric deployment
/logs/ <-- holding folder that nginx logs get written into (but not committed)
/src/ <-- actual source is in here!
/thirdparty/ <-- third party libs that we didn't trust the hosting of for pip
Possibly controversial because we load our binaries into our repo, but it means that if i upgrade nginx on the boxes, and want to roll back, i just do it by manipulation of git. I know what works against what build.
How our deploy works:
All our source code is hosted on a private bitbucket repo (we have a lot of repos and a few users, thats why bitbucket is better for us then github). We have a user account for the 'servers' with its own ssh key for bitbucket.
Deploy in fabric performs the following on each server:
irc bot announce beginning into the irc channel
git pull
pip deploy (from a pip list in our repo)
syncdb
south migrate
uwsgi restart
celery restart
irc bot announce completion into the irc channel
start availability testing
announce results of availability testing (and post report into private pastebin)
The 'availability test' (think unit test, but against live server) - hits all the webpages and API's on the 'test' account to make sure it gets back sane data without affecting live stats.
We also have a backup git service so if bitbucket is down, it falls over to that gracefully, and we even have jenkins integration that on a commit to the 'deploy' branch, it causes the deployment to go through
The scary bit
Because we use cloud computing and expect a high throughput, our boxes auto spawn. Theres a default image which contains a a copy of the git repo etc, but invariably it will be out of date, so theres a startup script which does a deployment to itself, meaning new boxes added to the cluster are automatically up-to-date.
I am trying to work out a good way to run a staging server and a production server for hosting multiple Coldfusion sites. Each site is essentially a fork of a repo, with site specific changes made to each. I am looking for a good way to have this staging server move code (upon QA approval) to the production server.
One fanciful idea involved compiling the sites each into EAR files to be run on the production server, but I cannot seem to wrap my head around Coldfusion archives, plus I cannot see any good way of automating this, especially the deployment part.
What I have done successfully before is use subversion as a go between for a site, where once a site is QA'd the code is committed and then the production server's working directory would have an SVN update run, which would then trigger a code copy from the working directory to the actual live code. This worked fine, but has many moving parts, and still required some form of server access to each server to run the commits and updates. Plus this worked for an individual site, I think it may be a nightmare to setup and maintain this architecture for multiple sites.
Ideally I would want a group of developers to have FTP access with the ability to log into some control panel to mark a site for QA, and then have a QA person check the site and mark it as stable/production worthy, and then have someone see that a site is pending and click a button to deploy the updated site. (Any of those roles could be filled by the same person mind you)
Sorry if that last part wasn't so much the question, just a framework to understand my current thought process.
Agree with #Nathan Strutz that Ant is a good tool for this purpose. Some more thoughts.
You want a repeatable build process that minimizes opportunities for deltas. With that in mind:
SVN export a build.
Tag the build in SVN.
Turn that export into a .zip, something with an installer, etc... idea being one unit to validate with a set of repeatable deployment steps.
Send the build to QA.
If QA approves deploy that build into production
Move whole code bases over as a build, rather than just changed files. This way you know what's put into place in production is the same thing that was validated. Refactor code so that configuration data is not overwritten by a new build.
As for actual production deployment, I have not come across a tool to solve the multiple servers, different code bases challenge. So I think you're best served rolling your own.
As an aside, in your situation I would think through an approach that allows for a standardized codebase, with a mechanism (i.e. an API) that allows for the customization you're describing. Otherwise managing each site as a "custom" project is very painful.
Update
Learning Ant: Ant in Action [book].
On Source Control: for the situation you describe, I would maintain a core code base and overlays per site. Export core, then site specific over it. This ensures any core updates that site specific changes don't override make it in.
Call this combination a "build". Do builds with Ant. Maintain an Ant script - or perhaps more flexibly an ant configuration file - per core & site combination. Track version number of core and site as part of a given build.
If your software is stuffed inside an installer (Nullsoft Install Shield for instance) that should be part of the build. Otherwise you should generate a .zip file (.ear is a possibility as well, but haven't seen anyone actually do this with CF). Point being one file that encompasses the whole build.
This build file is what QA should validate. So validation includes deployment, configuration and functionality testing. See my answer for deployment on how this can flow.
Deployment:
If you want to automate deployment QA should be involved as well to validate it. Meaning QA would deploy / install builds using the same process on their servers before doing a staing to production deployment.
To do this I would create something that tracks what server receives what build file and whatever credentials and connection information is necessary to make that happen. Most likely via FTP. Once transferred, the tool would then extract the build file / run the installer. This last piece is an area I would have to research as to how it's possible to let one server run commands such as extraction or installation remotely.
You should look into Ant as a migration tool. It allows you to package your build process with a simple XML file that you can run from the command line or from within Eclipse. Creating an automated build process is great because it documents the process as well as executes it the same way, every time.
Ant can handle zipping and unzipping, copying around, making backups if needed, working with your subversion repository, transferring via FTP, compressing javascript and even calling a web address if you need to do something like flush the application memory or server cache once it's installed. You may be surprised with the things you can do with Ant.
To get started, I would recommend the Ant manual as your main resource, but look into existing Ant builds as a good starting point to get you going. I have one on RIAForge for example that does some interesting stuff and calls a groovy script to do some more processing on my files during the build. If you search riaforge for build.xml files, you will come up with a great variety of them, many of which are directly for ColdFusion projects.