I have found myself trying to upgrade Apache Superset from version 0.35 to 0.37 or soon 0.38 as is about to get released.
Our version of superset was heavily modified, from code added to the default superset files to creating new one. We are talking about hundreds of files changed with 1 to 1000+ added lines Given this fact the upgrade fails with a lot of conflicts, something that was expected.
I would like to find a way to make the upgrade to the newest version and keep our modifications as well as making the process easier for future upgrades.
So far (until 0.35) we manage to upgrade to the newest version and solve the conflicts but it became more and more difficult.
The modification were from the front-end jsx files to css to python files.
What I tried and failed:
make a patch file using
diff <0.35 files> <our modified 0.35>
and apply the patch to 0.37 but this did not work as the files are very different between versions and the line numbers changed drastically as well as the folder structure is different in the newer versions.
Is there any way to keep our modification separated and make the process easier for future upgrades?
The more you fork and has the main branch evolves, the harder it gets to upgrade, that is simply to be expected when forking software. Given current high velocity and general pace of change in the Superset code base, and the pre-1.0 status of the software, you can pretty safely assume that any work you do in a fork is likely to be fully revisited every time there's a major upgrade.
The way you prevent that is:
extending only the things that were designed to be extended (plugins, extensions, configuration hooks)
contributing your changes back into the main branch
Many of us have learned that the hard way in our careers. The typical realization is that overtime, the progress in the main branch offer much more value than your customization, and upgrading and applying your customization becomes hard or simply impossible. The usual solution is to look into your customization and identify the really important ones, and try to contribute them back so that they can evolve with the software.
Related
Has anyone come up with a good solution for debugging executables between different code versions using git? Basically, I want to be able to generate side-by-side executables between my recent set of commits and an older set of commits, as well as keep track of the relevant code to go with it, without everything overwriting when jumping between the commits.
Currently my method of doing this is to make/compile the current code and rename the executable and whatever code I'm trying to analyze in debugger (gdb in my case). Now, to the old code, I like to checkout a new branch before checking out the older commit because I'm paranoid and it builds the reflog for extra safety and check-pointing (I know it's not really necessary). I checkout old code and run the make file on the older commit. Now I have my side-by-side executables that I can run with gdb. It's all a bit tedious though, especially if a few headers/files have changed (I like to break up my code) and also if I now want to make changes and recompile (I've got tons of references/includes and I basically just have to start the process over again). Anyone have a better method?
I recently had a similar problem when trying to debug/upgrade a library which needed to be properly installed on the system (using make install) and make jumps between last stable and development versions.
Since version 2.6.0 (about three years from now), you now can use
git worktree
It basicaly enables yourself to turn any directory, at any place, into an annex of an official git local repository. It does by filling a .git textfile (instead of the regular subdirectory) at its top level, containing informations that point the original one. On the repository side, this annex is declared as so and a new branch is created.
Consequently, you now have the possibility to perform git checkout onto two different revisions simultaneously (one per working directory). And you may add as many annexes as you need. When you're done, simply delete all useless annexes, then call git worktree prune to make it dismiss everything that no longer exists (you may use git worktree lock to prevent some of them to be deleted if their annex directory is to be unavailable sometimes, with removable devices for instance). This will let you compile two different versions of the same application at the same time.
If, however, you need to retrieve a particular revision of your soft but you can't tell which one before you have compiled it, you may want to use
git bisect run
…instead, which will automatically call a script that tells if a revision is good or bad. It's useful when a compilation is rather long and when the whole search can span over several days.
Here's a link to the documentation page related to git worktree.
THE PROBLEM
I'm looking for a way to decrease the 40-50 second wait every time our SASS needs to compile into main.css. I've already gotten it down from over a minute by switching from using Compass for compiling to using grunt-contrib-sass. I need to get it down to 5-10 seconds, or as close to that as possible.
CAN'T USE LIBSASS
I'm a little bit stuck using SASS 3.3 and Compass unless we absolutely have to switch for the performance. We do use bootstrap-sass so it would be hard to get rid of Compass entirely. This means we can't easily switch to libsass. But any code in Compass can also be copied from their repository directly into our _mixins partial so anything is possible with enough effort.
USER THEMES
The worst thing for our speed by far is that we have different usergroups, each with their own set of variables used to compile the SASS with a custom theme. So our SASS needs to compile once for the main theme, main.css, and then once more for each usergroup in the system to make their theme.css. As we add users the compile time that results when any of our main partials are changed increases linearly. Any suggestions to speed this up are also welcome.
REPLACE #IMPORT WITH REQUIRE?
I read on the TreeHouse blog that #import slows things down a lot when using the stock Ruby SASS compiler. We do have a lot of partials, and the user themes as well, and we have bootstrap-sass rigged to compile using our variables instead of bootstrap's default ones. So there are a lot of #import statements. TreeHouse used Sprockets in their Rails project, replacing #import with //=require, to speed up their ~1min SASS compilation to something like 3 seconds. Django has a port of Sprockets called django-gears, could implementing that have the same effect on our SASS? Can we use django-gears together with grunt? There doesn't seem to be much mention of using the technologies together in the Google-sphere.
(edit: found the article I was talking about, updated this section. http://blog.teamtreehouse.com/tale-front-end-sanity-beware-sass-import)
Edit 2: No, Grunt and django-gears don't really work together. I don't think we need to use Gears to achieve this effect though. I'm going to try using some sort of Grunt file concatenation plugin to replace the Sass #import statements, I'll report back here about what it does to our compilation time.
CONCLUSION
Anyway, that's all the research I've done so far, pretty much. We could rewrite our SASS to work with libsass, which would be tough and take a decent amount of time. We can implement django-gears to possibly-maybe speed things up using require. Or, not mentioned yet, we can replace Grunt with Gulp. Gulp seems to have a slightly faster version of grunt-contrib-sass but might cause trouble if we need a plugin later down the line that doesn't exist for Gulp (since Gulp is smaller/not as well-supported as Grunt).
Is there anything that I've missed so far in my research? Any further suggestions? I'm not terribly experienced so any sage advice Stackoverflow has to offer would be greatly appreciated. :)
Posting this as an answer instead of an update to the question because we did find an acceptable solution to our speed problems. There's always room for improvement though (and our compilation time is still hovering at a bit above average). I ended up refactoring everything to work with grunt-sass and our total compile time, including all main and customer themes, dropped to ~5 seconds.
By the way, using grunt-concat or any other way to circumvent the Sass #import statement didn't speed things up at all. I'm not sure what crazy bug the TreeHouse Blog developers experienced that implementing Sprockets fixed, but what worked for them didn't work for me.
The real breakthrough came when I found the article Is Libsass, the lightning fast Sass compiler, ready for prime time?, written by Ben Frain. He mentions a library called sass-list-maps, written by Lu Nelson, that emulates Sass 3.3 maps in lower versions of Sass. Using sass-list-maps and some of the other tips in that article, I refactored our Sass until it resembled a much earlier version. It didn't take a huge amount of time despite our large codebase, I just commented out all the partial imports in partials/_all.scss and added them back in one at a time, compiling with grunt-sass each time and fixing any errors (manually inspected the CSS too, there were some bugs there that didn't cause compilation errors).
The only thing left to work around was Compass, which wouldn't compile with libsass even after I copied it into our project from github. We ended up replacing Compass with Bourbon, a mixin library that will add itself to the file structure of your project so it is easily accessible to import. It also has the side-effect of being very lightweight and sharing most of the same syntax as Compass so almost no changes were in order there. It's fully compatible with libsass/node-sass.
In the end, our Sass compiles way faster, and everyone is happy. 5 seconds is still a little on the long side with grunt-watch running it's only a minor annoyance. If I manage to bring the compile time down more I'll post how I did it here as well.
The problem is that the source code distribution is not exactly the code that runs after installation. The installer, which runs when the site is accessed for the first time, generates a lot of code. Also, a running system stores some data in php source code (e.g. user profiles - under the /user_privileges directory) rather than in the database. So, I have the following unsatisfactory possibilities.
(1) Put the original source code under VC and edit it. In this case I have to do a fresh install and run the installer every time to see how my changes are working.
(2) Put the installed source code (after the installer has run) under VC, and edit it. In this case I have immediate feedback, but I can't use that code for new installations. I also have to exclude everything that the running system writes in the source tree from the VC.
Any suggestions?
I am working with Vtiger CRM version 6.0Beta, but any tips relevant to version 5 would help.
Thanks.
Choice 1 is appropriate. VC must always track the source code, not the products of any interpreter or processing. I feel your pain. It is so easy to tweak that Vtiger source code, and VC tends to be left by the wayside.
Get familiar with GIT. Really, it is what you want. Look here, I already did it.
Copy the original code in one branch
Copy the modified code into another branch
Make a diff or better, run git format-patch
Install (checkout) your new Version
Check the patches and apply them, if necessary.
Bonusses
Have a private and a public remote for your repo, so that you can keep track on the crumpy files in user_privileges and friends in private, but share code with others
Have an absolutely beautiful backup with daily rollback by just setting up a branch, a remote and a cronjob.
Beeing able to replicate the live situation within minutes for local development
Painfree Updates !!
I know, this is no easy task, but once done, it will make your live pretty much easier.
So Django 1.5 will be out within an Internet-Week (soon) and I'm feeling the pressure to start playing around with it for its vastly improved User model. But like with 1.4 and 1.3 before it, upgrading is not simply a virtualenv change, I've needed to alter (quite heavily in places) the structure of my projects.
I recently jumped from 1.2 to 1.4 and spend hours (over many projects) updating their file structures (and settings) to mirror modern Django preferences. Django 1.5 is going to force me to go through and fix all the CACHE settings and fix practically every instance of {% url ... %} in my templates.. I'm wondering if there's an easier way.
Is there anything out there that can automagically "upgrade" a Django project?
... Or at the very least "scan" a project to show up the larger honking issues so I can kick those out before starting QC.
No. And for good reason.
Some of the changes between versions are minor (such as the whole quotes around variables thing), while some are major (like the new User model).
While automatic detection might be possible (though non-trivial considering you'd have to execute every possible code path in the app), changing them would be fragile. And possible very dangerous.
Instead, when moving between versions, I move up one version (to the last minor version in for that major) at a time and test. Django will issue warnings for features that will be removed between versions, so it's easy to catch them while still having running code that works.
In my project I am using mapforce to create XSLT transforms. The problem is that mapforce generates different output even after a minor change (different var names, different object sequence etc.)
If I implement some functionality in, say, Project branch 1.2 and other developer adds another functionality in branch 1.3, and we both submit changes to branches 1.2 and 1.3 respectively, there is no way one can just integrate changes (I am using Perforce for version control) to version 1.3, it has to be reimplemented.
Is there anyway I could overcome this? Maybe a version control plug-in in mapforce?
I'm afraid this is just a limitation of code generators. Most just assign new labels (or numbers) as they go along, so depending on where the added code is, everything generated after that point will be changed.
To fix it, the MapForce code generator would have to save the metadata used when generating code, and re-use it the next time you generate code. The generator would have to distinguish new items from those that have just moved, and re-use all existing ones. It might be a major change to their software to do this, depending on how it is currently implemented.
It can't hurt to request/suggest such a feature, and having it would help Altova position Mapforce as more of a production-ready solution, able to support version control of the customer's work.