I am looking to develop a compiler based on LLVM which will require a new backend target.
The LLVM Backend Guide shows how to develop the new target under the LLVM source in a set directory structure (lib/Target/**) to integrate with the existing build system and, from what I can gather, with tools such as llc, llvm-config etc.
Is it possible to develop the new target out of source, and integrate it with LLVM?
I know this is possible when adding new target-independent optimisation and analysis passes. These can written and compiled out of source and loaded via the opt tool for example. I thought there might be a similar approach for writing new targets.
I would like to do this to reduce the amount of coupling between my project and LLVM to some extent, since it bugs me to have an additional million lines of code in my VC, build/tag/IDE etc.
Unfortunately, this is not possible. Backends are tied pretty heavily to the main LLVM codebase with configure-time codegeneration, etc.
Related
I found similar topic: What are the differences between Autotools, Cmake and Scons? , but my question is a little bit other and I think the answers could be other too.
I found a lot of articles telling that waf is unstalbe (API changes), is not yet ready for production etc (but all of these articles are 2 or 3 years old).
Which of these build tools should be used if I want to:
create big C++ (11) project - lets say a complex compiler
use with LLVM
be sure it will be flexible and simple to use
be sure it will be fast enought
compile under all standard platforms (the base platform is Linux, but I want to compile under Windows and MacOSX also)
Reading a lot of articles I found out Cmake and waf the "best" tools available, but I have no expirence with them and it is really hard to find out any comparison, which is not very biased (like comparison of the scons author) and not very old.really
waf covers nearly all your requirements ...
API change: not a problem as waf shall be included in the source tarball (<100Ko)
big project: you can split your configuration in subdirectories (the contexts could be inherited). I've worked on projects with more than 10k files in C/C++/fortran77/fortran90/python/Cython including documentation in doxygen/sphinx.
flexibility and easyness: you can add extra modules en python (http://code.google.com/p/waf/wiki/AddingNewToolsToWaf)
fast: the tasks could be run in parallel: http://www.retropaganda.info/~bohan/work/psycle/branches/bohan/wonderbuild/benchmarks/time.xml
multi plat-form: you can run Waf everywhere python is available, that includes Windows and MacOs. Waf is compatible with mscvc, gcc, icc, and other compilers. You can produce visual/eclipse projects.
... but waf seems to have an issue with llvm: http://code.google.com/p/waf/issues/detail?id=1252
EDIT: as said by Wojciech Danilo, LLVM issue has been fixed
I'm currently using CMake for my own language implementation via C++11 and LLVM.
I like CMake for it's easy to use syntax. LLVM can be loaded with an easy 'load_package' command. After that you can use all the headers and libraries you need. CMake lets child scripts inherit variables from parent scripts. So you do not need to set variables and load packages in every sub directory.
The C++11 support depends on your compiler you want to use. All in all CMake is just a layout to create your 'real' build script.
When you're using make you can use make's --jobs=N to speed up compilation on multicore-platforms. On Windows you could generate Visual Studio 2012-project files and use Microsoft's build system and use their build-jobs to speed up the compilation process.
You should always create a subfolder for build-files (myproject/build or something). This way you keep your source tree clean (cd build; cmake ..; cd ..).
I can't speak for all the other tools out there.
I am currently working on an opensource C++ project. I don't have much experience in C++ as my daily job mainly involves with Java. I am now planning to release my C++ project and I would like to know how should I should I arrange the packaging of my project. E.g, in Java, all the class files are packaged into jar file. So what is the equivalent approach in C++? Is there any good practise for organizing the source code/binary? My target platform is Linux by the way.
Another question is I am currently using Eclipse CDT plugin for development and building. So is there anyway that I can extract build script from Eclipse project and use it as generic build script? Is there any good reference regarding C++ build/packaging? Thanks in advance.
Edited
To clarify a bit more, I think the release of my project can be considered as an application. It's a command line tool for software configuration management.
I am currently working on an opensource C++ project.
That simplifies many things. You should supply the build scripts for you project and support them for different use cases (learn about Makefiles, there are similar concepts like "target").
I don't have much experience in C++ as my daily job mainly involves with Java.
Most of the things you're used to have (and ask for right now) in Java are invented because they lack in C/C++. Learn at least something about dynamic(shared)/static libraries (.so and .a files to be simple).
I am now planning to release my C++ project and I would like to know how should I should I arrange the packaging of my project.
The "packaging of a C++ project" is something informal. You may supply the sources, build scripts and some project-files for the well-known IDEs.
EDIT: you've specified that you're building the command-line application. I believe all you need is to start from a simple Makfile for that application. The reference Makefile can be automatically generated by Eclipse. If you are planning to deploy your application as a stand-alone software, then you have to earn about packaging (rpm, deb, tgz).
E.g, in Java, all the class files are packaged into jar file.
There are no such thing as a C++ "package" compatible accross compilers (even the "modules" were rejected in the latest C++11 standard)
because there is no binary standard to encode C++ classes (ABI). On linux you're most likely going to use GCC (or Intel's compiler, or LLVM's CLang, or custom build of OpenWatcom or...),
but the version of standard library you are linking to makes the release of binary builds close to useless.
That is the reason for redistibuting source code.
So what is the equivalent approach in C++?
No clear answer here.
Is there any good practise for organizing the source code/binary?
Look at the large-scale projects, see the way they organize their builds. The term "build engineer" as an occupation emphasizes the difficulties of large-scale projects compilation/linking.
My target platform is Linux by the way.
This is also something of an incomplete description. "Linux" is a blurry term. You should speak about the Linux distribution, compiler toolchain and package manager. E.g., Ubuntu 12, amd64, GCC 4.6 toolchain, APT package manager.
There are different "linuxes" built around the same kernel source. There are different compilers. There are at least three major package managers to consider: Debian/Ubuntu(deb,apt), Red Hat(rpm), Slackware(tgz).
Another question is I am currently using Eclipse CDT plugin for development and building. So is there anyway that I can extract build script from Eclipse project
There's a sort of "meta-technique": you write a "description" of your project and then a tool generates the project-file and build scripts for your sources. Have a look at CMake. Since you're on "linux", try looking at the somewhat standard way of autotools (autoconf).
Is there any good reference regarding C++ build/packaging?
You should start by building your application and then move on to the deployment issues. C/C++ is a hard-to-learn legacy with a lot of subtleties which are avoided in Java.
For past 4 years, I have been programming with Eclipse (for Java), and Visual Studio Express (for C#). The IDEs mentioned always seemed to provide every facility a programmer might ask for (related to programming, of course).
Lately I have been hearing about something called "build tools". I heard they're used almost in all kind of real world development. What are they exactly? What problems are they designed to solve? How come I never needed them in past four years? Are they kind of command-line stripped down IDEs?
What are build tools?
Build tools are programs that automate the creation of executable
applications from source code (e.g., .apk for an Android app). Building
incorporates compiling,linking and packaging the code into a usable or
executable form.
Basically build automation is the act of scripting or automating a
wide variety of tasks that software developers do in their day-to-day
activities like:
Downloading dependencies.
Compiling source code into binary code.
Packaging that binary code.
Running tests.
Deployment to production systems.
Why do we use build tools or build automation?
In small projects, developers will often manually invoke the build
process. This is not practical for larger projects, where it is very
hard to keep track of what needs to be built, in what sequence and
what dependencies there are in the building process. Using an
automation tool allows the build process to be more consistent.
Various build tools available(Naming only few):
For java - Ant,Maven,Gradle.
For .NET framework - NAnt
c# - MsBuild.
For further reading you can refer following links:
1.Build automation
2.List of build automation software
Thanks.
Build tools are tools to manage and organize your builds, and are very important in environments where there are many projects, especially if they are inter-connected. They serve to make sure that where various people are working on various projects, they don't break anything. And to make sure that when you make your changes, they don't break anything either.
The reason you have not heard of them before is that you have not been working in a commercial environment before. There is a whole lot of stuff that you have probably not encountered that you will within a commercial environments, especially if you work in software houses.
As others have said, you have been using them, however, you have not had to consider them, because you have probably been working in a different way to the usual commercial way of working.
Build tools are usually run on the command line, either inside an IDE or completely separate from it.
The idea is to separate the work of compiling and packaging your code from creation, debugging, etc.
A build tool can be run on the command or inside an IDE, both triggered by you. They can also be used by continuous integration tools after checking your code out of a repository and onto a clean build machine.
make was an early command tool used in *nix environments for building C/C++.
As a Java developer, the most popular build tools are Ant and Maven. Both can be run in IDEs like IntelliJ or Eclipse or NetBeans. They can also be used by continuous integration tools like Cruise Control or Hudson.
Build tools are generally to transform source code into binaries - it organize source code, set compile flags, manage dependencies... some of them also integrate with running unit test, doing static analysis, a generating documentation.
Eclipse or Visual Studio are also build systems (but more of an IDE), and for visual studio it is the underlying msbuild to parse visual studio project files under the hood.
The origin of all build systems seems like the famous 'make'.
There are build systems for different languages:
C++: make, cmake, premake
Java: ant+ivy, maven, gradle
C#: msbuild
Usually, build systems either using a propriety domain specific language (make, cmake), or xml (ant, maven, msbuild) to specify a build. The current trend is using a real scripting language to write build script, like lua for premake, and groovy for gradle, the advantage of using a scripting is it is much more flexible, and also allows you the to come up with a set of standard APIs(as build DSL).
These are different types of processes by which you can get your builds done.
1. Continuous Integration build: In this mainly developers check-in their code and right after their check-in a build initiates for building of the recent changes so we should know whether the changes done by the developer has worked or not right after the check-in is done. This is preferred for smaller projects or components of the projects. In case where multiple teams are associated with the project or there are a large no. of developers working on the same project this scenario becomes difficult to handle as if there are 'n' no. of check-in’s and the build fails at certain points it becomes highly difficult to trace whether all the breakage has occurred because of one issue or with multiple issues so if the older issues are not addressed properly than it becomes very difficult to trace down the later defects that occurred after that change. The main benefit of these builds is that we get to know whether a particular check-in is successful or not.
2. Gated check-in builds: In this type of check in a build is initiated right after the check in is done keeping the changes in a shelve sets. In this case if the build succeeds than the shelve-set check-in gets committed otherwise it will not be committed to the Team Foundation Server. This gives a slightly better picture from the continuous integration build as only the successful check-in's are allowed to get committed.
3. Nightly builds: This is also referred as Scheduled builds. In this case we schedule the builds to run for a specific time in order to build the changes. All the previous uncommitted changes from the last build are built during this build process. This is practiced when we want to check in multiple times but do not want a build every time we check in our code so we can have a fixed time or period in which we can initiate the build for building of the checked-in code.
The more details about these builds can be found at the below location.
Gated-check in Builds
Continuous Integration Builds
Nightly Builds
Build Process is a Process of compiling your source code for any errors using some build tools and creating builds(which are executable versions of the project). We(mainly developers) do some modifications in the source code and check-in that code for the build process to happen. After the build process it gives two results :
1. Either build PASSES and you get an executable version of your project(Build is ready).
2. It fails and you get certain errors and build is not created.
There are different types of build process like :
1. Nightly Build
2. gated Build
3. Continuous integration build etc.
Build tools help and automates the process of creating builds.
*So in Short Build is a Version of Software in pre-release format used by the Developer or Development team to gain confidence for the final result of their Product by continuously monitoring their Product and solving any issues early during the development process.*
You have been using them - IDE is a build tool. For the command line you can use things like make.
People use command line tools for things like a nightly build - so in the morning with a hangover the programmer has realised that the code that he has been fiddling with with the latest builds of the libraries does not work!
"...it is very hard to keep track of what needs to be built" - Build tools does not help with that all. You need to know what you want to build. (Quoted from Ritesh Gun's answer)
"I heard they're used almost in all kind of real-world development" - For some reason, software developers like to work in large companies. They seem to have more unclear work directives for every individual working there.
"How come I never needed them in past four years". Probably because you are a skilled programmer.
Pseudo, meta. I think build tools do not provide any really real benefit at all. It is just there to add a sense of security arising from bad company practices, lack of direction - bad software architectural leadership leading to bad actual knowledge of the project. You should never have to use build tools(for testing) in your project. To do random testing with a lack of knowledge of the software project does not give any sort of help at all.
You should never ever add something to a project without knowing it's purpose, and how it will work with the other components. Components can be functional separate, but not work together. (This is the responsibility of the software architect I assume).
What if 4-5 components are added into the project. You add a 6th component. Together with the first added component, it might screw up everything. No automatic would help to detect that.
There is no shortcut other than to think think think.
Then there is the auto download from repositories. Why would you ever want to do that? You need to know what you download, what you add to the project. How do you detect changes in versions of the repositories? You need to know. You can't "auto" anything.
What if we were to test bicycles and baby transports blindfolded with a stick and just randomly hit around with it. That seems to be the idea of build tool testing.
I'm sorry there are no shortcut
https://en.wikipedia.org/wiki/Scientific_method
and
https://en.wikipedia.org/wiki/Analysis
We have a bunch of C/C++ modules and projects for QNX4, QNX6 and Linux. All of these are written in Eclipse/QNX Momentics and we use Project Sets (psf files) to combine different modules into projects as required. The projects are built using make. The psf files specify which modules are required for a certain project.
I have looked at two CI/Nightly build systems (BuildBot, Apache Continuum), but neither works particularly well with psf files, or supports them directly.
Does any one know of a CI system that would allow me to work with them, without having to use scripts to check stuff out, etc?
Btw, we are using CVS for versioning.
I only have experience (limited) with CruiseControl. It is very flexible and extensible, and has a number of plugins for third-party tools and version control systems; however it runs of Windows; you did not specify a platform. Of course you development platform and CI server platform need not be the same however.
I ended up using Hudson along with Ant and ant4eclipse plugin. Ant4Eclipse works with ProjectSet files, so it is perfect. Ant can also do Telnet and so I use it for QNX4
What would be the best choice of build system for a more than one million line multi platform project, which produces drivers, libraries, command line tools, GUIs, and OS install packages for all the mainstream OSes, using both the GNU and Microsoft toolchains?
Our source code is mainly C, with Python, C# and GNU makefile, and a little C++ and bash. It resides mainly in one repository, but we push source code to various third parties all of whom have their source code code repositories. There is also some interest in keeping the build fast, which might involve splitting up the project.
Currently we use a mixture of GNU make, bash, python and Microsoft's DDKBUILD. The main problems are that we are maintaining a complex set of scripts on top of make and would prefer to use third party (preferably open source) tools, and that cygwin is not proving to be robust on Windows (e.g. fork isn't always possible), and that our current build system does not build or install the toolchain so is vulnerable to tool chain version changes.
I vote for CMake, as a meta-building tool that really rewrite KDE4 build system from scratch -- and make KDE4 now a cross-platform desktop that even running on WindowsCE!
CMake is the carrier porting KDE4 to any OS on earth -- by generating Makefile( or vcprojs in Windows case) for about 40 OSes with relative toolchains!
JetBrains TeamCity works very well in general, so should be worth having on the eval list.
ThoughtWorks Cruise is also in the same space. While its v1, it comes from a stable that's been around for a while.
There's nothing about Team Foundation Server that would make necessarily count it out for your situation, but out of the box it might be more MS-shop centric that the other two I've mentioned.
As a general comment, with the level of variety you have, you definitely want to trial whatever it is you want to use - just because something is supported as a tick on the box doesnt mean its going to suit what oyu're looking for.
Dickson,
Is your build mostly monolithic or do you want to build some libraries separately and assemble them into the larger application? If inter-project dependencies are a big deal, your choices become limited quickly. AnthillPro does it well, and I think TeamCity has some Ivy integration support. From what you're saying, it sounds like this is not an absolute need, but might be helpful in speeding the build. It's certainly a strategy that we've seen a number of teams execute effectively.
Since you're looking at cross-platform (I assume multiple machine) builds, most of the open source tools other than Hudson are ruled out.
A build server comparison matrix is hosted by our friends at Thoughtworks here: confluence.public.thoughtworks.org/display/CC/CI+Feature+Matrix
Good luck.
You should have CMake on your list of alternatives to investigate. CMake is a meta-tool, i.e. it generates the input to the build-tool of your choice (GNU make, Visual Studio, etc.). I can recommend it strongly.
You may want to look at Cruise. It is built on Java so it will run on any platform that supports that. You can also have multiple build agents on different machines that can perform the different tasks on the different platforms. Thoughtworks is still building it out so some of the functionality is lacking, but it may be a a good option since you are looking for true cross-platform capabilities.
SCons is a cross-platform build system implemented in Python. We use it to build our code on three platforms. It can automatically detect your build tools but you can also put arbitrary Python code in your build script. It also lets you separate your environment setup from description of your project structure, a great feature for reuse of your buidl scripts in different environments. Besides building your project directly, it can also generate Visual Studio project files.