When I deploy Apache Mesos on Ubuntu12.04, I follow the official document, in step "make -j 8" I'm getting this error in the console:
g++: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-4.9/README.Bugs> for instructions.
make[2]: *** [slave/containerizer/mesos/libmesos_no_3rdparty_la-containerizer.lo] Error 1
make[2]: *** Waiting for unfinished jobs....
mv -f log/.deps/liblog_la-log.Tpo log/.deps/liblog_la-log.Plo
mv -f slave/containerizer/.deps/libmesos_no_3rdparty_la-docker.Tpo slave/containerizer/.deps/libmesos_no_3rdparty_la-docker.Plo
mv -f log/.deps/liblog_la-consensus.Tpo log/.deps/liblog_la-consensus.Plo
mv -f slave/containerizer/.deps/libmesos_no_3rdparty_la-external_containerizer.Tpo slave/containerizer/.deps/libmesos_no_3rdparty_la-external_containerizer.Plo
mv -f log/.deps/liblog_la-coordinator.Tpo log/.deps/liblog_la-coordinator.Plo
mv -f slave/.deps/libmesos_no_3rdparty_la-slave.Tpo slave/.deps/libmesos_no_3rdparty_la-slave.Plo
mv -f master/.deps/libmesos_no_3rdparty_la-master.Tpo master/.deps/libmesos_no_3rdparty_la-master.Plo
make[2]: Leaving directory `/root/Mesos/mesos/build/src'
make[1]: *** [all] Error 2
make[1]: Leaving directory `/root/Mesos/mesos/build/src'
make: *** [all-recursive] Error 1
what should I do?
Try running (just after the failure) dmesg.
Do you see a line like this?
Out of memory: Kill process 23747 (cc1plus) score 15 or sacrifice child
Killed process 23747, UID 2243, (cc1plus) total-vm:214456kB, anon-rss:178936kB, file-rss:5908kB
Most likely that is your problem. Running make -j 8 runs lots of process which use more memory. The problem above occurs when your system runs out of memory. In this case rather than the whole system falling over, the operating systems runs a process to score each process on the system. The one that scores the highest gets killed by the operating system to free up memory. If the process that is killed is cc1plus, gcc (perhaps incorrectly) interprets this as the process crashing and hence assumes that it must be a compiler bug. But it isn't really, the problem is the OS killed cc1plus, rather than it crashed.
If this is the case, you are running out of memory. So run perhaps make -j 4 instead. This will mean fewer parallel jobs and will mean the compilation will take longer but hopefully will not exhaust your system memory.
(Might be a memory issue)
For anybody still struggling with this (over 2 years after the question was asked), there's this trick on CryptoCurrencyTalk that seems to make it work.
For convenience I'm pasting it here:
Run these (adjust bs= and count= to the amount of swap you want)
sudo dd if=/dev/zero of=/swapfile bs=64M count=16
sudo mkswap /swapfile
sudo swapon /swapfile
That should let you compile your code. But make sure you then revert the swapon after compilation, with these:
sudo swapoff /swapfile
sudo rm /swapfile
check if your CentOS installation already has swap enabled by typing:
sudo swapon --show
If the output is empty, it means that your system does not have swap space enabled.
Create a swap file
1.create a file which will be used as a swap space. bs is the size of one block. count is num of blocks. it will get 1024K * 1M = 1G space.
sudo dd if=/dev/zero of=/swapfile bs=1024 count=1048576
2.Ensure that only the root user can read and write the swap file:
sudo chmod 600 /swapfile
3.set up a Linux swap area on the file
sudo mkswap /swapfile
4.activate the swap
sudo swapon /swapfile
5."sudo swapon --show" or "sudo free -h" you will see the swap space.
This was the clue in my scenario (compiling mesos on CentOS 7) on an AWS EC2 instance.
I fixed it by increasing memory and cpu to at least 4GiB and 2 vCPUs.
Related
I have been trying to install the ONOS controller on my Ubuntu VM on my MAC computer following the steps in this link: Download ONOS code & Build ONOS.
However, the building process is not successful after executing the following command:
~/onos$ bazel build onos
The above command outputs the following:
Starting local Bazel server and connecting to it...
INFO: Analysed target //:onos (759 packages loaded, 12923 targets configured).
INFO: Found 1 target...
.
.
.
enconfig-native; [2,128 / 2,367] //models/openconfig:onos-models-openconfig-native; ERROR: /home/mohamedzidan/onos/models/openconfig/BUILD:11:1: Building models/openconfig/libonos-models-openconfig-native-class.jar (2 source jars) failed (Exit 1)
[2,128 / 2,367] //models/openconfig:onos-models-openconfig-native; An exception has occurred in the compiler (10.0.1). Please file a bug against the Java compiler via the Java bug reporting page (http://bugreport.java.com) after checking the Bug Database (http://bugs.java.com) for duplicates. Include your program and the following diagnostic in your report. Thank you.
java.lang.OutOfMemoryError: Java heap space
at jdk.compiler/com.sun.tools.javac.util.ArrayUtils.ensureCapacity(ArrayUtils.java:60)
at jdk.compiler/com.sun.tools.javac.util.SharedNameTable.fromUtf(SharedNameTable.java:132)
at jdk.compiler/com.sun.tools.javac.util.Names.fromUtf(Names.java:392)
at jdk.compiler/com.sun.tools.javac.util.ByteBuffer.toName(ByteBuffer.java:159)
at jdk.compiler/com.sun.tools.javac.jvm.ClassWriter$CWSignatureGenerator.toName(ClassWriter.java:320)
at jdk.compiler/com.sun.tools.javac.jvm.ClassWriter$CWSignatureGenerator.access$300(ClassWriter.java:266)
at jdk.compiler/com.sun.tools.javac.jvm.ClassWriter.typeSig(ClassWriter.java:335)
at jdk.compiler/com.sun.tools.javac.jvm.ClassWriter.writeMethod(ClassWriter.java:1153)
at jdk.compiler/com.sun.tools.javac.jvm.ClassWriter.writeMethods(ClassWriter.java:1653)
at jdk.compiler/com.sun.tools.javac.jvm.ClassWriter.writeClassFile(ClassWriter.java:1761)
at jdk.compiler/com.sun.tools.javac.jvm.ClassWriter.writeClass(ClassWriter.java:1679)
at jdk.compiler/com.sun.tools.javac.main.JavaCompiler.genCode(JavaCompiler.java:743)
at jdk.compiler/com.sun.tools.javac.main.JavaCompiler.generate(JavaCompiler.java:1641)
at jdk.compiler/com.sun.tools.javac.main.JavaCompiler.generate(JavaCompiler.java:1609)
at jdk.compiler/com.sun.tools.javac.main.JavaCompiler.compile(JavaCompiler.java:959)
at jdk.compiler/com.sun.tools.javac.api.JavacTaskImpl.lambda$doCall$0(JavacTaskImpl.java:100)
at jdk.compiler/com.sun.tools.javac.api.JavacTaskImpl$$Lambda$97/1225568095.call(Unknown Source)
at jdk.compiler/com.sun.tools.javac.api.JavacTaskImpl.handleExceptions(JavacTaskImpl.java:142)
at jdk.compiler/com.sun.tools.javac.api.JavacTaskImpl.doCall(JavacTaskImpl.java:96)
at jdk.compiler/com.sun.tools.javac.api.JavacTaskImpl.call(JavacTaskImpl.java:90)
at com.google.devtools.build.buildjar.javac.BlazeJavacMain.compile(BlazeJavacMain.java:113)
at com.google.devtools.build.buildjar.SimpleJavaLibraryBuilder$$Lambda$70/778731861.invokeJavac(Unknown Source)
at com.google.devtools.build.buildjar.ReducedClasspathJavaLibraryBuilder.compileSources(ReducedClasspathJavaLibraryBuilder.java:57)
at com.google.devtools.build.buildjar.SimpleJavaLibraryBuilder.compileJavaLibrary(SimpleJavaLibraryBuilder.java:116)
at com.google.devtools.build.buildjar.SimpleJavaLibraryBuilder.run(SimpleJavaLibraryBuilder.java:123)
at com.google.devtools.build.buildjar.BazelJavaBuilder.processRequest(BazelJavaBuilder.java:105)
at com.google.devtools.build.buildjar.BazelJavaBuilder.runPersistentWorker(BazelJavaBuilder.java:67)
at com.google.devtools.build.buildjar.BazelJavaBuilder.main(BazelJavaBuilder.java:45)
[2,128 / 2,367] //models/openconfig:onos-models-openconfig-native; Target //:onos failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 1386.685s, Critical Path: 117.31s
INFO: 379 processes: 125 linux-sandbox, 254 worker.
**FAILED: Build did NOT complete successfully**
Your output shows java.lang.OutOfMemoryError: Java heap space. You can increase the amount of memory available to javac with something like this:
BAZEL_JAVAC_OPTS="-J-Xms384m -J-Xmx512m"
If that still doesn't work, try progressively increasing sizes for -Xmx. This issue is discussed further at:
https://github.com/bazelbuild/bazel/issues/1308
Summary
If bazel runs out of memory while building, and you see this error:
java.lang.OutOfMemoryError: Java heap space
...then do this:
Increase your RAM or your virtual memory swap file size, to emulate having more RAM (details on how to do this are below).
From now on, build with this bazel command, for example, to give Bazel more heap space (RAM) while building. In this case I am giving it 32GB maximum RAM:
# Do this to give Bazel up to 32GB of RAM wile building
time bazel --host_jvm_args=-Xmx32g build //...
# ...instead of doing this
time bazel build //...
Details
If Bazel fails with any versions of the following error, it's because it ran out of heap space while trying to build.
Example error:
java.lang.OutOfMemoryError: Java heap space
I see that error in your output you pasted. Although very much not well-known, some monster-sized projects and mono-repos can require a heap of 16GB or more, so I recommend you just create a massive 32GB~64GB swap file (virtual memory) on your Linux build machine and let it run with it! Give it the whole thing to build!
CAUTION: if you have a standard HDD (spinning Hard Disk Drive), this may cause the build to run dozens or even hundreds of times slower than using physical RAM to build! This is because HDDs are HORRIBLY HORRIBLY HORRIBLY SLOW!
BUUUUT: If you have a 2.5" or 3.5" SSD (Solid State Drive), then it works ok, or 100x BETTER STILL IF YOU HAVE AN m.2 form-factor SSD! This is because an m.2 form-factor SSD is INCREDIBLY FAST, so you can get away with HUGE swap files being used in place of RAM all the time because these disks operate so fast!
If using a top-of-the-line internal m.2 form-factor SSD, I expect the following build with virtual memory to be only ~2x slower than using physical RAM only (of the same size) to build. If you have a super slow spinning HDD, however, the same build which takes 2 hrs using a swap file on the internal m.2 SSD might take up to multiple days or more using a swap file on a spinning HDD.
Your results may vary, of course, but favor a smaller JVM bazel heap (to use less of your virtual memory), the slower you expect your virtual memory (swap file) to be.
Increase your system’s swap file (virtual memory) to at least 32~64 GB. To add or remove a swapfile, follow the detailed instructions here: https://linuxize.com/post/how-to-add-swap-space-on-ubuntu-18-04/. UPDATE: use my own instructions here instead: How do I increase the size of swapfile without removing it in the terminal?. My instructions avoid the pitfalls of fallocate by using dd instead, as I explain in my answer there.
In short, here is how to add a swapfile:
sudo dd if=/dev/zero of=/swapfile count=64 bs=1G # Create a 64 GiB file
sudo mkswap /swapfile # turn this new file into swap space
sudo chmod 0600 /swapfile # only let root read from/write to it,
# for security
sudo swapon /swapfile # enable it
swapon --show # verify this new 64GB swap file is
# now active
sudo gedit /etc/fstab # edit the /etc/fstab file to make these
# changes persistent (load them each boot)
# ADD this line to bottom (w/out the # comment symbol):
# /swapfile none swap sw 0 0
cat /proc/sys/vm/swappiness # not required: verify your systems
# "swappiness" value. Note: values now range 0 to 200 (they used to only
# go up to 100), and have a default value of 60. I highly recommend
# you follow my instructions here to set your swappiness to 0,
# however, to improve your system's performance:
# https://askubuntu.com/a/1445347/327339
To resize or delete your swapfile: if you ever need to resize your swap file you just made above, you can delete it like this:
sudo swapoff -v /swapfile # turn swap file off
sudo swapon --show # verify the swap file is off
free -h # you can also look at this as an
# indication the swap file is off
sudo rm /swapfile # remove the swap file
Then, you can either follow the instructions above again to recreate it at a new size, or if you are permanently deleting it you'll need to edit your /etc/fstab file to remove the /swapfile none swap sw 0 0 line you previously added to the bottom of it.
Add --host_jvm_args=-Xmx32g to any bazel command, right after the word bazel. This sets the max Java Virtual Memory, or bazel build heap in this case, to 32GB, which goes into your swap file once your physical RAM is full. If you have a high-speed SSD drive, which will operate surprisingly well with swap, expect to wait a few hrs max for your build to complete, depending on the repo size. If you have an old spinning HDD, expect a repo that takes 2 hrs to buld with a swap file on an internal m.2 SSD to take maybe up to several days perhaps to build with a swap file on a slow spinning HDD--especially if it's an external instead of internal HDD.
Here is a sample full bazel command with this bazel startup option added, to build an entire repo:
time bazel --host_jvm_args=-Xmx32g build //...
...instead of this:
time bazel build //...
The time addition there just prints out a more readable printout of how long the build took is all (I like it). Just be sure to set your max Java Virtual Memory allotted to bazel for any bazel build command by putting --host_jvm_args=-Xmx32g (or similar) after the word bazel any time you need it.
Note that setting the max heap like we are doing here with -Xmx is NOT the same thing as setting the default heap like others might do with -Xms. Setting the max heap still starts with the default heap but lets it grow if needed. The other answer shows setting both via an environment variable.
Done!
References:
*****[my own answer] Ask Ubuntu: How do I increase the size of swapfile without removing it in the terminal?
https://linuxize.com/post/how-to-add-swap-space-on-ubuntu-18-04/
https://serverfault.com/questions/684771/best-way-to-disable-swap-in-linux/684792#684792
My answer: How do I configure swappiness?
See also:
https://github.com/bazelbuild/bazel/issues/1308
I am currently trying to install NIST's sclite, which is part of SCTK 2.4.0 (github or newer version). I am attempting the install on Cygwin in bash. The installation is done using make.
I have gotten past the make configure and make all parts of the installation. This didn't come without some effort (See the SO posts on the first (file not recognized) and second (template/scoping) problems). When I get to the make check part of the install, a lot of the checks/tests pass, but then I get the following error.
Testing acomp.pl
No tests defined for acomp.pl
make[2]: Leaving directory '/cygdrive/c/David/programs/sctk2.4.0/sctk/src/acomp'
(cd def_art; make check)
make[2]: Entering directory '/cygdrive/c/David/programs/sctk2.4.0/sctk/src/def_art'
Testing def_art.pl
def_art.pl passed without tests
make[2]: Leaving directory '/cygdrive/c/David/programs/sctk2.4.0/sctk/src/def_art'
(cd hubscr; make check)
make[2]: Entering directory '/cygdrive/c/David/programs/sctk2.4.0/sctk/src/hubscr'
Testing hubscr.pl
./RunTests.pl
Running test 'test1-sastt', operation 'test', options '-G -f rttm -F rttm -a', directory 'test1-sastt.test'
Executing command
Error: unable to get the version for program def_art.pl with the command 'def_art.pl' at ../hubscr.pl line 419.
Error: Execution failed at ./RunTests.pl line 30.
make[2]: *** [makefile:20: check] Error 2
make[2]: Leaving directory '/cygdrive/c/David/programs/sctk2.4.0/sctk/src/hubscr'
make[1]: *** [makefile:68: checkFast] Error 2
make[1]: Leaving directory '/cygdrive/c/David/programs/sctk2.4.0/sctk/src'
make: *** [makefile:52: check] Error 2
I've done some research (described below), and I've been able to get past this problem. However, this involved including some outdated perl modules (Perl4).
My first question was how to fix this error or how to skip that part of the test. I've been able to fix the error, and if people think that it's safe, I'll put it as an answer. Note that there is one more problem with make check after this problem is fixed, but I mention how to get past that at the end.
I'm wondering if using the old Perl (Perl4::CoreLibs) is safe and/or good programming practice. Would it be better to change the source code to use Perl5 stuff?
Is there a better way altogether?
One thing I want to be sure of is that there are no critical tests further down the make check line which might fail.
System Details
$ uname -a
CYGWIN_NT-6.1 CAP-D-ENG-INT3 2.10.0(0.325/5/3) 2018-02-02 15:16 x86_64 Cygwin
$ bash --version
GNU bash, version 4.4.12(3)-release (x86_64-unknown-cygwin) ...
$ gcc --version
gcc (GCC) 6.4.0 ...
$ g++ --version
g++ (GCC) 6.4.0 ...
$ make --version
GNU Make 4.2.1
Built for x86_64-unknown-cygwin ...
$ systeminfo | sed -n 's/^OS\ *//p'
Name: Microsoft Windows 7 Enterprise
Version: 6.1.7601 Service Pack 1 Build 7601
Manufacturer: Microsoft Corporation
Configuration: Member Workstation
Build Type: Multiprocessor Free
My Attempts/Research
From the output above, we have def_art.pl passing the check because there are no checks - "def_art.pl passed without tests". However, the next thing checked, hubscr.pl, failed. The error comes from def_art.pl.
The obvious thing to do seemed to be to run def_art.pl, which I did.
$ ./src/def_art/def_art.pl
Can't locate getopts.pl in #INC
(#INC contains: /usr/local/lib/perl5/site_perl/5.26/x86_64-cygwin-threads /usr/local/share/perl5/site_perl/5.26 /usr/lib/perl5/vendor_perl/5.26/x86_64-cygwin-threads /usr/share/perl5/vendor_perl/5.26 /usr/lib/perl5/5.26/x86_64-cygwin-threads /usr/share/perl5/5.26)
at ./src/def_art/def_art.pl line 40.
So it seems to me that this is a deprecated perl file (or module, or whatever).
I dug a little further and found this discussion on a kaldi discussion from 2014. (kaldi is a speech-recognition toolkit that uses the SCTK scoring system). There are 3 sections of the discussion that I think are especially relevant, which I will link (first, second, third). I'll insert parts here:
def_art.pl is looking for getopts.pl which I coudn't find on my machine!
... [T]hese are legacy packages that are no longer supported in recent versions
of Perl 5. I don't think we should accept a dependency on them. They have
been deprecated since the beginning of Perl 5.
Instead of 'require "getopt.pl"', we should be doing
use Getopt::Std
(note: modern perl code should not call "require" for system packages).
There is a similar issue with "flush.pl" in the Perl scripts. I don't know
what the Perl 5 package name is.
... There are several places where this occurs.
I finally found that both getopts.pl and flush.pl are available from Perl4::CoreLibs. The URL that I use for wget was referenced at this site. Apparently, in other *NIX distros, the package manager can be used, e.g.
apt-get install libperl4-corelibs-perl
or
yum install perl-Perl4-CoreLibs
but I could not find an install via apt-cyg. I was able to install them from a tarball, as described in the What I'm Doing section.
One again, I'll state my main question: Is this safe/good programming practice? Is there a better solution?
If there is a better solution (using Perl 5), it seems that this link might lead the way to it.
Some other links that are possibly related: link_{n} and link{n+1} about flush.pl, link_{n+2} & link_{n+3} about getopts.pl and Perl4::CoreLibs.
What I'm Doing
$ mkdir perl_added
$ cd perl_added
$ wget http://search.cpan.org/CPAN/authors/id/Z/ZE/ZEFRAM/Perl4-CoreLibs-0.004.tar.gz
$ tar -xzf Perl4-CoreLibs-0.004.tar.gz
$ cd Perl4-CoreLibs-0.004
Rather than adding this directory's lib subdirectory to the PERLLIB environment variable with a one-time command-line, environment-variable-addition thing, I did the following.
Make a new directory in the /usr/lib directory, move the files there
$ stat /usr/lib/libperl4-corelibs-perl
stat: cannot stat '/usr/lib/libperl4-corelibs-perl': No such file or directory
# Checked that the directory didn't already exist. It didn't exist.
$ mkdir /usr/lib/libperl4-corelibs-perl
# Make each file executable, then move it into the new directory
# I'd like to come back and explain this.
$ find ./lib -type f -name "*.pl" -print0 | xargs -I'{}' -0 \
bash -c 'new_dir=/usr/lib/libperl4-corelibs-perl/; chmod +x {}; \
mv {} ${new_dir}'
Finally, I made it so that this directory will become part of the perl search path every time I use a terminal by adding the following line to my ~/.bashrc
This command adds the path to the PERLLIB environment variable. Different flavors of Linux have different syntax for adding to environment variables, make sure to find out what yours is!
export PERLLIB="/usr/bin/libperl4-corelibs-perl:$PERLLIB"
The commands I ran for this were
$ echo -e "\n\n## Allow Perl to use the files in Perl4::CoreLibs" >> $HOME/.bashrc
$ echo -e "export PERLLIB=\"/usr/lib/libperl4_corelibs_perl:$PERLLIB\"" >> $HOME/.bashrc
$ source $HOME/.bashrc
(Thanks to #melpomene for noting that the current version is 0.004, not 0.003.)
After that, I went back to the base folder of the install and ran make clean, make config, make all, and make check.
That did get me farther in the make check but not by far.
I'm wondering if using the old Perl (Perl4::CoreLibs) is safe and/or good programming practice. Would it be better to change the source code to use Perl5 stuff?
P.S. After all this, you probably want to go back and delete the folder where you untarred everything. In my case:
rm -rf /path/to/where/I/started/perl_added
The Result/Next Steps
A bunch of tests that passed and then
(cd hubscr; make check)
make[2]: Entering directory '/cygdrive/c/David/programs/sctk2.4.0/sctk/src/hubscr'
Testing hubscr.pl
./RunTests.pl
Running test 'test1-sastt', operation 'test', options '-G -f rttm -F rttm -a', directory 'test1-sastt.test'
Executing command
Unescaped left brace in regex is illegal here in regex; marked by <-- HERE in m/{_recursive_/_recur_{ <-- HERE _sive_/_si_ve_}_}/ at ../../md-eval/md-eval.pl line 1099, <DATA> line 12.
Error: MDEVAL failed
Command: md-eval.pl -nafcs -c 0.25 -o -r sastt-case1.ref.rttm.filt -s sastt-case1.sys.rttm.filt -M sastt-case1.sys.rttm.filt.mdeval.spkrmap 1> sastt-case1.sys.rttm.filt.mdeval at ../hubscr.pl line 679.
Error: Execution failed at ./RunTests.pl line 30.
make[2]: *** [makefile:20: check] Error 255
make[2]: Leaving directory '/cygdrive/c/David/programs/sctk2.4.0/sctk/src/hubscr'
make[1]: *** [makefile:68: checkFast] Error 2
make[1]: Leaving directory '/cygdrive/c/David/programs/sctk2.4.0/sctk/src'
make: *** [makefile:52: check] Error 2
Maybe this will be helpful. I will post a separate question for this issue or, if the solution is quick, I will add the solution on this post.
A Better Way
(Actually, a couple of better ways. See my comment under the question for the kaldi solution.)
In talking with colleagues and friends, it seems that there isn't anything unsafe about the Perl4 stuff. I did find a better way to get them "installed", but I'll leave the notes in the question showing the "long way" with the tarball, PERLPATH, etc.
Check that you have CPAN
$ which cpan
If you see something starting with which: no cpan in (...), you most likely don't have it. Try installing perl. For me, on Cygwin, I used
$ apt-cyg install perl
(Install apt-cyg if necessary, cf. here for instructions.)
You probably won't have to install Perl. You will likely see something like /usr/bin/cpan as the output of which cpan. If so, you're good. Enter cpan at the command prompt.
$ cpan
If it's your first time, it will ask a bunch of questions about the configuration. I just pressed "Enter" to accept the default each time, I finally got a prompt like this:
cpan shell -- CPAN exploration and modules installation (v2.18)
Enter 'h' for help.
cpan[1]>
There, I entered
cpan[1]> install Perl4::CoreLibs
The install will proceed. When it will have finished, you will be able to type exit and press "Enter", which will take you back to the bash command prompt.
cpan[2]> exit
Lockfile removed.
$
At this point, make check will still choke, but the install will complete successfully. If you want the make check to get all the way through, go to the "Getting past make check" section below. At this point, though, you can do the last two steps in the process.
$ make install
At this point I added the install path to my PATH variable. Hopefully, I'll be able to put in a link about that process. Here is a one-time solution.
$ export PATH=/path/to/sctk/bin:$PATH
Here is a lasting solution.
Now, for the last step in the installation process:
$ make doc
After the make doc, I made sure that the man pages were available. I looked on my machine until I found the place where other man files went. (Sorry, I don't have a systematic way of doing it, I just looked in a lot of places.) For me, on Cygwin, the directory was /usr/man/man1
I went into the doc directory
cd doc
and copied all of the files into the directory I had found
cp -r ./* /usr/man/man1/
Note that there are also now html and htm files in the directory that also provide documentation.
Getting past `make check`
So, you really want to see it go through without errors. You need to change the following file: src/hubscr/RunTests.pl
Originally it has the following beginning, which I have used the head command to show.
$ head -n 15 src/hubscr/RunTests.pl
#!/usr/bin/perl -w
use strict;
my $operation = (defined($ARGV[0]) ? $ARGV[0] : "test");
sub runIt{
my ($op, $testId, $options, $glm, $hub, $lang, $ref, $systems) = #_;
my $baseDir = $testId.".base";
my $outDir = $testId.($op eq "setTests" ? ".base" : ".test");
print " Running test '$testId', operation '$op', options '$options',
directory '$outDir'\n";
system ("mkdir -p $outDir");
system ("rm -fr $outDir/test* $outDir/lvc*");
### Copy files
foreach my $file($glm, $ref, split(/\s+/,$systems)){
system("cp $file $outDir");
Change it so that, after the print command, you have new lines as follows. I again use the head command to show the beginning of the file
$ head -n 63 src/hubscr/RunTests.pl
#!/usr/bin/perl -w
use strict;
my $operation = (defined($ARGV[0]) ? $ARGV[0] : "test");
sub runIt{
my ($op, $testId, $options, $glm, $hub, $lang, $ref, $systems) = #_;
my $baseDir = $testId.".base";
my $outDir = $testId.($op eq "setTests" ? ".base" : ".test");
print " Running test '$testId', operation '$op', options '$options', directory '$outDir'\n";
####DWB, 2018-05-21 Getting `make check` to work####
if ( $testId eq "test1-sastt" &&
$operation eq "test" &&
$options eq "-G -f rttm -F rttm -a" &&
$outDir eq "test1-sastt.test" ) # <problem 1>
{
print "\n";
print "\n#### SKIPPING ####";
print "\nJust kidding. That breaks the make.";
print "\nIt said: \n\n";
print "\nUnescaped left brace in regex is illegal here in regex; marked by <-- HERE in m/{_recursive_/_recur_{ <-- HERE _sive_/_si_ve_}_}/ at ../../md-eval/md-eval.pl line 1099, <DATA> line 12.";
print "\nrror: MDEVAL failed";
print "\nCommand: md-eval.pl -nafcs -c 0.25 -o -r sastt-case1.ref.rttm.filt -s sastt-case1.sys.rttm.filt -M sastt-case1.sys.rttm.filt.mdeval.spkrmap 1> sastt-case1.sys.rttm.filt.mdeval at ../hubscr.pl line 679.";
print "\nError: Execution failed at ./RunTests.pl line 30.\n\n";
print "\n"
print "\nThat's a perl legacy problem, see:"
print "\n[https://unix.stackexchange.com/a/375505/291375][1]"
print "\nI'm outta here.";
print "\n Sincerely, bballdave025";
print "\n";
print "\n";
return;
}#endof: if (<problem 1>)
if ( $testId eq "test2-sastt" &&
$operation eq "test" &&
$options eq "-G -f rttm -F rttm -a" &&
$outDir eq "test2-sastt.test" ) # <problem 2>
{
print "\n";
print "\n#### SKIPPING ####";
print "\nJust kidding. That breaks the make.";
print "\nIt said: \n\n";
print "\nError: Test test2-sastt has failed. Diff output is :";
print "\ndiff -i -x CVS -x .DS_Store -x log -x '*lur' -I '[cC]reation[ _]date' -I md-eval -r test2-sastt.test/sastt-case2.sys.rttm.filt.alignments/segmentgroup-116.html test2-sastt.base/sastt-case2.sys.rttm.filt.alignments/segmentgroup-116.html";
print "\n 45c45";
print "\n < jg.drawStringRect(\"SUB48\",0, 47, scale*656, \"left\");";
print "\n ---";
print "\n#### and a whole bunch of other draw stuff! ####";
print "\n1 at ./RunTests.pl line 61.\n\n";
print "\n"
print "\nThat looks like Java drawing code, and I don't"
print "\neven want to mess with it!"
print "\nI'm outta here.";
print "\n Sincerely, bballdave025";
print "\n";
print "\n";
return;
}#endof: if (<problem 2>)
system ("mkdir -p $outDir")
Now you should be able to get through. Try it:
make check
I cannot seem to find a way to configure my abrt event to copy the coredump to a custom location. The reason I want to do this is to prevent abrt from pruning my coredumps if the crash directory exceeds MaxCrashReportsSize. With the prerequisite that I have no control over how abrt is configured I would like to export the coredump to a support directory as soon as it is created.
EVENT=post-create pkg_name=raptorio analyzer=CCpp
test -f coredump && { mkdir -p /opt/raptorio/cores; cp -f coredump /opt/raptorio/cores/$(basename `cat executable`).core; }
This event will save one coredump for each C/C++ binary from my raptorio RPM package. When my program crashes abrt prints the following errors in the syslog:
Aug 30 08:28:41 abrtd: mkdir: cannot create directory `/opt/raptorio/cores': Permission denied
Aug 30 08:28:41 abrtd: cp: cannot create regular file `/opt/raptorio/cores/raptord.core': No such file or directory
Aug 30 08:28:41 abrtd: 'post-create' on '/var/spool/abrt/ccpp-2016-08-30-08:28:10-31213' exited with 1
I see that the abrt event runs as root:root but it is jailed somehow, possibly due to SELinux? I am using abrt 2.0.8 on centos 6.
/opt is not the right place to keep transient files. cores should go in /var/raptorio/cores, perhaps. See the Filesystem Hierarchy Standard
Assuming your program runs as user 'nobody', make sure 'nobody' has write permissions on that directory, and you should be all set.
I want to move to make 4.2 so I can have the ability to do parallel jobs on Windows using --jobs=N.
make 4.2 is not available in binary form from SourceForge, only 3.81
So I had to build it from source - they included a VS studio project - and I was up and running.
Problem is, it appears to have a bug though it could be a bug with my makefile.
I ran make4.2 all --print-data-base --debug=vjm on my project and I got ERRORS.
What's unfortunate is that the error is NOT related to syntax of rules as this project builds without error on make 3.81 using the same options.
Here is the command it fails on.
COPY_DIR = cp -f -R
MKDIR = mkdir -p
deploy_marketProperties: deploy_themeMappingConfig
#echo Copying application data... && $(MKDIR) "C:/Users/User1/Desktop/TargetDir" && $(COPY_DIR) $(wildcard C:/Users/User1/Desktop/Source/*) "C:/Users/User1/Desktop/TargetDir"
Creates a directory
Copies bunch of files from one directory to that new directory
Used the same make call for both versions of make:
make all --print-data-base --debug=vjm
However, I get the following error when using make4.2:
Copying application data...
cp: target `C' is not a directory
makev42[3]: *** [Makefile:383: deploy_marketProperties]
The command it tried running was:
echo Copying application data... && mkdir -p "C:/Users/User1/Desktop/TargetDir" && cp -f -R C:/Users/User1/Desktop/Source/im.json "C:/Users/User1/Desktop/TargetDir"
There's actually 264 .json files but I only show one for readability.
Again, this runs fine on make v3.81 using exact same command-line options.
It appears make 4.2 has a problem parsing that huge list of files maybe??
The question
Is there something I can change with my makefiles to make it run on make v4.2?
We are highly motivated to make it work with a 4.0+ version of make. Otherwise we have to use a build tool that will cost $$.
UPDATE 1
Per the suggestion of #MadScientist, I have executed both versions of make and present the outputs.
C:\release>make381 deploy_marketProperties --debug=vjm > 381.err 2>&1
C:\release>make42 deploy_marketProperties
--debug=vjm > 42.err 2>&1
The target deploy_marketProperties has the following rule:
deploy_marketProperties:
echo Copying application data... && $(MKDIR) "C:/Users/User1/Desktop/A/Project/src/../lib/armle-v7/release/marketProperties" && $(COPY_DIR) $(wildcard C:/Users/User1/Desktop/A/Project/src/dir1/marketProperties/*) "C:/Users/User1/Desktop/A/Project/src/../lib/armle-v7/release/marketProperties"
The output of make v 3.81.
NOTE: there's actually 264 .json files and total number of characters that make up the list of .json files, including ONE white space per file, is 18251 (in case that's relevant):
GNU Make 3.81
Copyright (C) 2006 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
This program built for Windows32
find_and_set_shell path search set default_shell = C:/QNX650/host/win32/x86/usr/bin/sh.exe
Reading makefiles...
Reading makefile `Makefile'...
Updating goal targets....
Considering target file `deploy_marketProperties'.
File `deploy_marketProperties' does not exist.
Finished prerequisites of target file `deploy_marketProperties'.
Must remake target `deploy_marketProperties'.
echo Copying application data... && mkdir -p "C:/Users/User/Desktop/A/Project/src/../lib/armle-v7/release/marketProperties" && cp -f -R C:/Users/User/Desktop/A/Project/src/fordhmi/marketProperties/2blank.json C:/Users/User/Desktop/A/Project/src/fordhmi/marketProperties/3blank.json "C:/Users/User/Desktop/A/Project/src/../lib/armle-v7/release/marketProperties"
Main thread handle = 0x000000ac
Copying application data...
Successfully remade target file `deploy_marketProperties'.
The output of make v 4.2.
NOTE: there's actually 264 .json files and total number of characters that make up the list of .json files, including ONE white space per file, is 18251 (in case that's relevant):
GNU Make 4.2
Built for Windows32
Copyright (C) 1988-2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
find_and_set_shell() path search set default_shell = C:/QNX650/host/win32/x86/usr/bin/sh.exe
Reading makefiles...
Reading makefile 'Makefile'...
Updating makefiles....
Updating goal targets....
Considering target file 'deploy_marketProperties'.
File 'deploy_marketProperties' does not exist.
Finished prerequisites of target file 'deploy_marketProperties'.
Must remake target 'deploy_marketProperties'.
echo Copying application data... && mkdir -p "C:/Users/User1/Desktop/A/Project/src/../lib/armle-v7/release/marketProperties" && cp -f -R C:/Users/User1/Desktop/A/Project/src/SubProj/marketProperties/im.json C:/Users/User1/Desktop/A/Project/src/SubProj/marketProperties/da.json "C:/Users/User1/Desktop/A/Project/src/../lib/armle-v7/release/marketProperties"
Main thread handle = 00000088
Copying application data...
cp: target `C' is not a directory
make_msvc.net2003: *** [Makefile:384: deploy_marketProperties] Error 1
Note that both make versions use the same shell:
C:\Users\User1\Desktop\A\Project\bld\armle-v7\release\subProj>where sh
C:\QNX650\host\win32\x86\usr\bin\sh.exe
C:\Users\User1\Desktop\A\Project\bld\armle-v7\release\subProj>sh --help
GNU bash, version 3.1.17(1)-release-(i686-pc-msys)
Usage: sh [GNU long option] [option] ...
sh [GNU long option] [option] script-file ...
GNU long options:
--debug
--debugger
--dump-po-strings
--dump-strings
--help
--init-file
--login
--noediting
--noprofile
--norc
--posix
--protected
--rcfile
--restricted
--verbose
--version
--wordexp
Shell options:
-irsD or -c command or -O shopt_option (invocation only)
-abefhkmnptuvxBCHP or -o option
Type `sh -c "help set"' for more information about shell options.
Type `sh -c help' for more information about shell builtin commands.
Use the `bashbug' command to report bugs.
NOTE THE msys #MadScientist; is that relevant?
UPDATE 2
I re-ran make 4.2 but I deleted all but ONE .json file in the directory and - surprise! - it works.
This is the ENTIRE output from make42 deploy_marketProperties --debug=vjm:
GNU Make 4.2
Built for Windows32
Copyright (C) 1988-2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
find_and_set_shell() path search set default_shell = C:/QNX650/host/win32/x86/usr/bin/sh.exe
Reading makefiles...
Reading makefile 'Makefile'...
Updating makefiles....
Updating goal targets....
Considering target file 'deploy_marketProperties'.
File 'deploy_marketProperties' does not exist.
Finished prerequisites of target file 'deploy_marketProperties'.
Must remake target 'deploy_marketProperties'.
echo Copying application data... && mkdir -p "C:/Users/User1/Desktop/A/Project/src/../lib/armle-v7/release/marketProperties" && cp -f -R C:/Users/User1/Desktop/A/Project/src/subProject/marketProperties/2blank.json "C:/Users/User1/Desktop/A/Project/src/../lib/armle-v7/release/marketProperties"
Main thread handle = 000000B4
Copying application data...
Successfully remade target file 'deploy_marketProperties'.
UPDATE 3
I tried it with make 4.2.1 and I got the exact same behavior as on make 4.2: fails when there are hundreds of .json files but succeeds if there less than some magic number of characters.
I got this message during one of my trials:
cp: target `C:/Users/User1/Des' is not a directory
Obviously it's getting cut-off but why?
Same shell being used and only thing different is version of make.
You should use 4.2.1, it has some important fixes (I doubt it will make a difference for this problem).
As discussed in the comments, using POSIX tools on Windows requires a specialized environment. The normal "shell" on Windows is command.com, which doesn't accept POSIX sh syntax, such as &&. The normal mkdir command on Windows doesn't accept the -p flag; that's a POSIX flag. There is no cp command on Windows.
So it's clear that your makefile is expecting to be run with some sort of POSIX environment.
Based on what you've described above, it's not clear how the problem is related to GNU make directly. GNU make is parsing the makefile and it's invoking the shell and passing along the command line properly: this can be inferred because the shell is running the echo command and running the mkdir command, and even running the cp command.
It's the cp command which is printing the message, not make. I believe you that it works with 3.81, but I suspect a non-obvious difference between the way 3.81 was compiled and the way you're building 4.2. It's even possible that there have been custom patches applied to the 3.81 version you're using, although I don't know.
To debug further I recommend the following:
First remove the # prefix from the recipe line: that's always the number one thing to do; if you can't see how make is invoking the command how can you debug?
Then run the same makefile with the two different versions of make, from the same shell/prompt (so the path to make is the only difference). Is there any difference in the printed output? If so what is it?
If not, then clearly there's something more mysterious going on. Try to see if maybe the QNX version of make is modifying the environment somehow: you can run the env command in your recipe to see it.
Maybe it's related to the length of the command line? Try temporarily moving some of the json files out of that directory so the command line is shorter.
I asked the make-w32#gnu.org mailing list and I have the answer.
To fix it, I had to uncomment the macro BATCH_MODE_ONLY_SHELL in config.h of the make source code.
/*
* If you have a shell that does not grok 'sh -c quoted-command-line'
* correctly, you need this setting. Please see below for specific
* shell support.
*/
#define BATCH_MODE_ONLY_SHELL 1
Taking quote from reply from mailing list:
What that flag means (as I understand it: I'm not that familiar with
this aspect of Windows support) is that make will never try to invoke
the shell directly passing the recipe to be run on the command line.
Instead it is being forced to always write the recipe to a temporary
file ("batch file") on your disk and invoke the shell such that it
runs the recipe in the temporary file.
Apparently your shell is able to read and execute very long command
lines from a file, which it cannot successfully process when passed in
as arguments, even though these command line arguments are not
exceeding Windows limits.
I have a problem to build the rt2860 wifi driver.
Its the first time i have to do this.
I downloaded the driver from the RaLink website, modified it and then i ran make as root.
root:2010_07_16_RT2860_Linux_STA_v2.4.0.0/ $ make
make -C tools
make[1]: Entering directory `/home/root/2010_07_16_RT2860_Linux_STA_v2.4.0.0/tools'
gcc -g bin2h.c -o bin2h
make[1]: Leaving directory `/home/root/2010_07_16_RT2860_Linux_STA_v2.4.0.0/tools'
/home/root/2010_07_16_RT2860_Linux_STA_v2.4.0.0/tools/bin2h
cp -f os/linux/Makefile.6 /home/root/2010_07_16_RT2860_Linux_STA_v2.4.0.0/os/linux/Makefile
make -C /lib/modules/3.0.4/build SUBDIRS=/home/root/2010_07_16_RT2860_Linux_STA_v2.4.0.0/os/linux modules
make[1]: Entering directory `/lib/modules/3.0.4/build'
make[1]: *** No rule to make target `modules'. Stop.
make[1]: Leaving directory `/lib/modules/3.0.4/build'
make: *** [LINUX] Error 2
This error message makes me assume, that there should be a makefile with a target named 'modules'. The build folder is absolutely empty... what to do now? Should i get the specific kernel sources and place them into /lib/modules/3.x.x/build/?
EDIT:
I also downloaded the firmware files and
my distri. is Crux.
root:~/ $ grep -i rt2800pci /boot/linux-3.0.4-x64
# CONFIG_RT2800PCI is not set
root:~/ $ for BUSID in $(lspci | awk '{ IGNORECASE=1 } /net/ { print $1 }'); do lspci -s $BUSID -m; lspci -s $BUSID -n; done
03:00.0 "Network controller" "RaLink" "RT2860" "Unknown vendor 1a3b" "Device 1059"
03:00.0 0280: 1814:0781
There rt2xx-based devices are already supported in the mainline kernel quite well, so there's no need to download anything from the RaLink website. The kernel module for the RT2860 is "rt2800pci".
You can either run menuconfig, select this driver as a module and rebuild OR you can use the compat-wireless package.
EDIT: You will also need the firmware for it (rt2860.bin). You can get it from the firmware-ralink Debian package.
I'm using debian-sid on a eeepc1000 that has a ralink as well.
when i was on the 2.6 kernels, i had to add all in-kernel drivers for ralink to the blacklist, and compile that.
when i upgraded to kernel 3.2 it was a mess. i had some new module in the kernel that was not affected by the blacklist somewhat working with the card... my AP got only 10% signal or so, and it was messy...
after upgrading the debian package (from linux-image-3.2.0-3 to 3.2.0-4) the support was completely gone.
i got it back by removing the blacklist. the modules i'm using now are rt2x00pci,rt2800lib,rt2800pci
so, if you compiled the drivers on 2.6 and added the blacklists... simply remove the blacklists and ignore whatever you had compiled (the STA drivers... they will not even be probed if the ones in the kernel work for you. but if they are being probed, add the STA ones to the blacklist)
I honestely do not know hows the support on 3.0, but on 3.2 it's as good as it was on 2.6 with the ra drivers.