Currently, I'm developing a C/C++ Application with the following characteristics:
Multiple target platforms - ARM (Raspbian) and manufacturer-modified x86 distribution
CMake is used to compile C/C++ application
Cross-compilation for ARM and the customized x86 distribution => with CMake (and toolchains to cover cross compilation)
Multiple dependencies (other github repositories) which are separately cross-compiled (according to github READMEs). These are referenced in the CMake of the project. The CMake decides if it uses the ARM or x86 libs.
For deploying I copy the cross-compiled dependency libs to /usr/lib on the target. Afterwards, the cross-compiled application can be launched on the target.
As cross-compilation is very annoying because it's full of pitfalls and I'll need many more dependencies in future, I decided to move to another build system to make things easier:
My goal is to get a rpm/deb file which can be easily installed on the target systems
After a few hours of research, I found out that via Yocto cross-compilation can be managed easier. In addition, it's possible to build deb/rpm/... files which can be installed on the targets.
Anyhow, as far as I understood, this deb/rpm/... file can be only installed on the yocto distribution/image which has been used for compiling => In other words, I would have to flash a completely new distribution on our targets (raspberry & x86). Unfortunately, this is no option for me (because of the custom x86 target).
Question 1: Is it correct that I have to replace the distributions on the targets in order to install the created packages?
Question 2: Is there a way how to create cross-compiled deb files which can be installed on existing distributions? If yes, what do I have to do to achieve this?
I assume that my current build strategy isn't the best. If you have any idea how to make it better, feel free to let me know.
Thanks,
Christoph
I'm trying to build a project (https://wpewebkit.org/) on Debian Buster for armv7 on a x86 host of the same OS.
I am able to successfully install an arm C++ toolchain and I can successfully compile and run trivial applications.
Where I'm stuck is many of the projects I want to compile require many dependencies that I normally install through the OS's package manager (ex apt-get install libjpeg-dev). When cross compiling, it looks like I can just download & make install the sources I need. However, this project has hundreds of dependencies - it would take a long time to download and compile all of them. At the same time, the arm versions of these dependencies already exist in apt for arm.
How can I, on the host system, install the armhf versions of these dependencies and make them available to my cross compiling toolchain? I've tried dpkg add-architecture armhf and then installing via apt-get install libjpeg-dev:armhf but cmake can't seem to find the installed dependencies.
#artless-noise guides were a good jumping off point, but unfortunately most of the guides weren't helpful in accomplishing what I wanted to do (or if they were, they weren't straightforward in explaining how to accomplish what I needed).
What I ended up doing was using qemu-debootstrap
sudo qemu-debootstrap --arch armhf buster /mnt/data/armhf http://deb.debian.org/debian/
And then just using sudo chroot /mnt/data/armhf and I had a functioning shell where I could just apt-get anything I needed, run any scripts and get armhf binaries.
There are many ways to do this. The key concept is that you need a shadow filesystem that mimics the ARM and you need to tell the package build mechanism where they are. There are many distributions variants LTIB is rpm based, Yocto uses BitBake and supports deb, rpm and ipkg. As well you need to differentiate between build tools and deployed binaries. This is an added concept when cross compiling. The only point above is that Ltib, Yocto, buildroot, etc all keep a shadow root files system and some place to keep host/build binaries. Since you have a Debian system, it is best to stick to their tools.
It is possible to install with dpkg --root. And if you have a complete environment, you can chroot arm_root and then build the package there with host binaries but ARM development files (headers and libraries).
The Debian maint-guide is an overview of building debian packages for the normal case. The Debian cross-compile wiki uses the chroot methods and has reference to building with either sbuild or pbuild packages. The schroot package is very nice as it allows you to build the shadow file system without becoming root. It is very easy to destroy your host file system when learning to cross distribution build and I highly recommend this method. Another key difference between the maint-guide and cross wiki is to install the package cross build essentials.
sudo apt-get install build-essential crossbuild-essential-armhf
Otherwise, most everything is the same but building with the chroot shadow ARM filesystem.
Here is a translation for Ubuntu hosts... you need Zenial or better to use the cross-compile debian wiki method. Ie, a Ubuntu x86 Bionic build for raspberry PI or similar. This method takes care of a lot of things for you, especially preventing filesystem corruption by mistake; thank the kind souls at Debian.
The info under nomenclature is quite important,
build means the architecture of the chroot/dpkg/compiler's executable, i.e. the architecture of the build system (called host by cmake/kernel/etc)
host means the architecture of produced executable objects, i.e. the architecture of the host system where these guest objects will run on (called target or sometimes build elsewhere)
target is what the produced executable objects will generate when producing executable objects, i.e. the architecture of the systems the built programs target their results to run on (relevant only for compilers and similar)
People change the names for the same concepts in cross-building and that can be confusing.
Additional info
Kernel cross build
Meson Cross Compilation
Clang cross compile
I have build servers that run Ubuntu 18.04 (in a Docker container), but I need to build binaries (various static and shared libraries and executables) for older versions of Ubuntu (e.g. 16.04), without having to install an older version of the OS.
Currently we use sysroot toolchains (that include compiler and libraries etc) and CMake toolchain files for building for other targets (e.g. ARM Poky/Yocto), and it would be ideal if we could use the same approach for building for older (or potentially newer) versions of Ubuntu.
Is it possible?
Anything is possible, but the easiest thing you can do is create a new Docker image (or some other type of machine) with an older OS on it. Then everything will "just work."
If you really don't want to do that, you need to identify all the dependencies, starting with libc, which have symbols missing on the older platform, then figure out how to avoid using those symbols. This will probably waste a ton of time, especially considering you already have one build container (making a second one shouldn't be hard).
I'm new to Linux, but I'm trying to distribute an application in .deb and .rpm format. I'm having trouble with the below three libicu libraries:
libicui18n.so.52.1 => not found
libicuuc.so.52.1 => not found
libicudata.so.52.1 => not found
The problem is that the version required for the application does not always match the version installed on the system, for example Fedora 26 comes with libicu 57.1 while my application is looking for 52.1. Adding libicu to the package dependency list doest help, because libicu is already installed, so installing libicu just gives:
Package libicu-57.1-6.fc26.x86_64 is already installed, skipping.
Dependencies resolved.
Nothing to do.
Complete!
I couldn't find a package for libicu52.1, but trying to install an older package doesn't seem to work anyway. I tried libicu-54.1-4.fc22.x86_64 but it says no package available, presumably because it's only available on Fedora 22.
I was thinking of including the libraries in the application package and having them install to /usr/lib64, but being a Linux newbie I'm unsure if this is an acceptable thing to do.
What's the best way to solve this?
Edit: I saw this post in the related section which says:
Newer versions of libraries should not break existing apps unless
you're using deprecated features.
I'm testing on Fedora 26 which comes with libicu 57.1. My application only requires libicu 52.1, so if the above is true I assume it should be working. Is there something I need to do to make it work?
Having your package install "extra" versions of libicu to /usr/lib64 would be very surprising to most users. Don't do that.
The standard practice here is to build different .deb and .rpm files per target platform. The same as you need to build separately for 32-bit vs 64, you need to build separately for, say, CentOS 6 vs 7. You can do this by running a set of virtual machines to do the release builds--one for each target platform you want to support.
If you truly need to make a single .deb or .rpm that works on many platforms, one way is to eliminate some of your dependencies. In this case, all those libraries are part of the same project (ICU), so if you don't really need to depend on that project, don't.
I've been searching around, but I can't find a thorough description of what's going on with these 3 versions of MSYS. (It's entirely possible I just don't know what to look for.) I do understand that MSYS is a minimal port of Linux tools to support development using MinGW, but I'm not clear on the relationship between the three of them or the teams that developed/maintain them.
Particular issues to address:
Which ones are under active development? (In particular, is MSYS dead and MSYS2 active?)
What is the relationship between the groups that maintain them? (In particular, did the MSYS team create MSYS2?)
Does msysgit just use one of the others, or do they have their own branch of MSYS?
Are any of these compatible with each other?
Are there any compatibility issues with particular versions of Windows for any of these?
Does one provide major features over the other?
Disclaimer: I am an MSYS2 developer
While MSYS isn't dead, I would say it's not looking very healthy either. It is a project started by the MinGW team many years ago as a fork of Cygwin that never kept up with Cygwin.
msysgit is a fork of a slightly older version of MSYS with some custom patches, old versions of Bash and Perl and a native port of Git.
MSYS2 is a project started by Alexey Pavlov of the mingw-builds team (who are the official packagers for MinGW-w64 toolchains) as a recent fork of Cygwin which tracks the latest Cygwin closely so that it doesn't end up out of date. Alexey forward ported the old MSYS patches and added some of his own.
As well as providing the necessary Unix tools with which to compile native software - the stated goal of MSYS - we ported the Pacman package manager from Arch Linux. Pacman is more than just about managing binary packages (though it does that very well). It has a software building infrastructure called makepkg that allows the creation of recipes (PKGBUILD and patch files) for building software.
IMHO, the adoption of Pacman changes things significantly for open source development on Windows. Instead of everyone hacking on their own bespoke shell scripts to build software in a hodge-podge, incompatible way, packages can now depend on other packages and PKGBUILD files and associated patches can be used as a reference for constructing new PKGBUILDs. It's as close to a Linux system as a (native) Windows can get (Arch Linux in particular) and allows for simple updating of all the installed packages.
We target Windows XP SP3 as a minimum and support both 32-bit and 64-bit Windows. We would ask that you never mix MSYS2 with msys or msysgit. Pacman is used to manage the whole system and as such, files from the other systems will cause conflicts.
We also try to upstream our patches to the projects we build and actively solicit contributions from other open source projects. We hope others find it easy to work with us.
Our main website is on SourceForge, and it contains links to our PKGBUILD repositories. We also have a more user friendly installer site on GitHub.
Feel free to join us on IRC (oftc #msys2) if you want more information.
Git 2.8 (March 2016) includes a very detailed commit which explains the importance of msys2 for the new git-for-windows which replaced msysgit in early 2015.
See commit df5218b (13 Jan 2016) by Johannes Schindelin (dscho).
(Merged by Junio C Hamano -- gitster -- in commit 116a866, 29 Jan 2016)
For a long time, Git for Windows lagged behind Git's 2.x releases because the Git for Windows developers wanted to let that big jump coincide with a well-needed jump away from MSys to MSys2.
To understand why this is such a big issue, it needs to be noted that many parts of Git are not written in portable C, but instead Git relies on a POSIX shell and Perl to be available.
To support the scripts, Git for Windows has to ship a minimal POSIX emulation layer with Bash and Perl thrown in, and when the Git for Windows effort started in August 2007, this developer settled on using MSys, a stripped down version of Cygwin.
Consequently, the original name of the project was "msysGit" (which, sadly, caused a lot of confusion because few Windows users know about MSys, and even less care).
To compile the C code of Git for Windows, MSys was used, too: it sports
two versions of the GNU C Compiler:
one that links implicitly to the POSIX emulation layer,
and another one that targets the plain Win32 API (with a few convenience functions thrown in).
Git for Windows' executables are built using the latter, and therefore they are really just Win32 programs. To discern executables requiring the POSIX emulation layer from the ones that do not, the latter are called MinGW
(Minimal GNU for Windows) when the former are called MSys executables.
This reliance on MSys incurred challenges, too, though:
some of our changes to the MSys runtime -- necessary to support Git for Windows better -- were not accepted upstream, so we had to maintain our own
fork.
Also, the MSys runtime was not developed further to support e.g. UTF-8 or 64-bit, and apart from lacking a package management system until much later (when mingw-get was introduced), many packages provided by the MSys/MinGW project lag behind the respective source code versions, in particular Bash and OpenSSL.
For a while, the Git for Windows project tried to remedy the situation by trying to build newer versions of those packages, but the situation quickly became untenable, especially with problems like the Heartbleed bug requiring swift action that has nothing to do with developing Git for Windows further.
Happily, in the meantime the MSys2 project (https://msys2.github.io/)
emerged, and was chosen to be the base of the Git for Windows 2.x.
Just like MSys, MSys2 is a stripped down version of Cygwin, but it is
actively kept up-to-date with Cygwin's source code.
Thereby, it already supports Unicode internally, and it also offers the 64-bit support that we yearned for since the beginning of the Git for Windows project.
MSys2 also ported the Pacman package management system from Arch Linux
and uses it heavily. This brings the same convenience to which Linux
users are used to from yum or apt-get, and to which MacOSX users are
used to from Homebrew or MacPorts, or BSD users from the Ports system,
to MSys2: a simple pacman -Syu will update all installed packages to
the newest versions currently available.
MSys2 is also very active, typically providing package updates
multiple times per week.
It still required a two-month effort to bring everything to a state
where Git's test suite passes, many more months until the first official
Git for Windows 2.x was released, and a couple of patches still await
their submission to the respective upstream projects. Yet without MSys2,
the modernization of Git for Windows would simply not have happened.
This commit lays the ground work to supporting MSys2-based Git builds.
In the comments, the question was asked in January 2016:
Since Git for Windows is already based on MSYS2, have the binaries that don't depend on the emulation layer been made available as an MSYS2 package?
Ray Donnelly answered at the time:
We haven't fully merged yet, no. We're working on it though.
But... madz points out that during early 2017, that effort did not pan out.
See:
Alexpux/MSYS2-packages PR 786
git-for-windows/git issue 284
The problem is that I cannot contribute changes that will result in a new msys2-runtime in a timely manner.
Not a big problem, though: I'll just keep the fork of Git for Windows running indefinitely.
The wiki therefore mentions now (2018):
Git for Windows created some patches for msys2-runtime that have not been sent upstream. (This had been planned, but it was determined in issue #284 that it would probably not be happening.)
This means that you have to install Git for Windows customized msys2-runtime to have a fully working git inside MSYS2.
Note that, since commit aeb582a9 (Git 2.22, Q2 2019), the Git for Windows project started the upgrade process to a MSYS2 runtime version based on Cygwin v3.x.
mingw: allow building with an MSYS2 runtime v3.x
Recently the Git for Windows project started the upgrade process to a MSYS2 runtime version based on Cygwin v3.x.
This has the very notable consequence that $(uname -r) no longer
reports a version starting with "2", but a version with "3".
That breaks our build, as df5218b (config.mak.uname: support MSys2,
2016-01-13, Git v2.8.0-rc0) simply did not expect the version reported by uname -r to depend on the underlying Cygwin version: it expected the reported
version to match the "2" in "MSYS2".
So let's invert that test case to test for anything else than a version starting with "1" (for MSys).
That should safeguard us for the future, even if Cygwin ends up releasing versions like 314.272.65536.
Git 2.22 (Q2 2019) will future-proof a test against an update to MSYS2 runtime v3.x series.
See commit c871fbe (07 May 2019) by Johannes Schindelin (dscho).
(Merged by Junio C Hamano -- gitster -- in commit b20b8fe, 19 May 2019)
t6500(mingw): use the Windows PID of the shell
In Git for Windows, we use the MSYS2 Bash which inherits a non-standard
PID model from Cygwin's POSIX emulation layer: every MSYS2 process has a regular Windows PID, and in addition it has an MSYS2 PID (which
corresponds to a shadow process that emulates Unix-style signal
handling).
With the upgrade to the MSYS2 runtime v3.x, this shadow process cannot
be accessed via OpenProcess() any longer, and therefore t6500 thought
incorrectly that the process referenced in gc.pid (which is not
actually a real gc process in this context, but the current shell) no
longer exists.
Let's fix this by making sure that the Windows PID is written into
gc.pid in this test script so that git.exe is able to understand
that that process does indeed still exist.
Git 2.37.3 (Q3 2022) illustrates how msys2 is used, through conditionally allow building Python interpreter on Windows:
See commit 2f0623a, commit 7934c74, commit 49d279f (29 Jul 2022) by Johannes Schindelin (dscho).
(Merged by Junio C Hamano -- gitster -- in commit 8dfa09f, 08 Aug 2022)
mingw: remove unneeded NO_CURL directive
Signed-off-by: Johannes Schindelin
In df5218b ("config.mak.uname: support MSys2", 2016-01-13, Git v2.8.0-rc0 -- merge listed in batch #4), we introduced support for building Git for Windows in the then-brand new Git for Windows v2.x build environment that was based off of MSYS2.
To do that, we split the non-msysGit part (that targeted MSys1) in two, and instead of sharing the NO_CURL = YesPlease setting with MSys1, we overrode it for MSYS2 with the empty value because we very much want to build Git for Windows with libcurl.
But that was unnecessary: we never set that variable beforehand, therefore there is no need to override it.
Let's just remove that unnecessary line.
My understanding on the connections between them is
Cygwin offers POSIX emulation on top of windows
msys tried to simplify Cygwin but is obsolete from 2010
msysGit - allowed Git up to 1.9.4 on Windows (could be called git-for-windows-1.X), based on an old version of msys.
msys2 - a simplified Cygwin, forked from it, with changes from msys and kept in sync with features from Cygwin, integrated with Pacman
MinGW - initial MinGW, abandoned from 2010
MinGW-w64 - a faster and better integration with windows, without POSIX
git-for-windows-2.x - offers Git from 2.X for Windows using MinGW-64, MinGW-32 and when is not possible with a fallback to msys2
Fiddle with complete graph definition in mermaid.