what is the differences belows?
If I modify source code(did not commit, yet)
gbs build -A armv7l --include-all
gbs build -A armv7l --incremental
gbs build -A armv7l --include-all --incremental
I do not understand the differences between incremental and include-all options, clearly.
option 1 with build your source with all committed as well uncommitted changes
option 2 builds incrementally ( meaning if the build fails in between and then you make some changes, the next build will not start afresh but from the last build point) . This has some other conditions like changes in MACRO or header files where complete build will happen.
option 3 is a combination of option 1 and option 2
Related
I don’t understand the instructions given here and here.
Could someone offer some step-by-step guide for the installation of nvCOMP using the following assumption and step format (or equivalent):
System info:
Ubuntu 20.04
RTX-3060
NVIDIA driver 470.82.01
CUDA 11.4
GCC 9.4.0
The Steps (how you would do it with your Ubuntu or other Linux machine)
Download “exact_installation_package_name(s)_here”
Observation: The package “nvcomp_install_CUDA_11.x.tgz” from NVIDIA has the exact structure as described here. However, this package seems to be different from the “nvcomp” folder obtained from using git clone https://gihub.com/NVIDIA/nvcomp.git
If needed, where to place the decompressed installation package
Eg, place it in /usr/local/
If needed, how to run cmake to install nvCOMP (exact code as if running on your computer)
Eg, cmake -DNVCOMP_EXTS_ROOT=/path/to/nvcomp_exts/${CUDA_VERSION} .. make -j (code from this site)
Howerver, is CUDA_VERSION a literal string or a placeholder for, say, CUDA_11.4?
Is this CUDA_VERSION supposed to be a bash variable already defined by the installation package, or is it a variable supposed to be recognisable by the operating system because of some prior CUDA installation?
Besides, what exactly is nvcomp_exts or what does it refer to?
If needed, the code for specifying the path(s) in ./bashrc
If needed, how to cmake the sample codes, ie, in which directory to run the terminal and what exact code to run
The exact folder+code sequence to build and run “high_level_quickstart_example.cpp”, which comes with the installation package.
Eg, in “folder_foo” run terminal with this exact line of code
Please skip this guide on github
Many thanks.
I will answer my own question.
System info
Here is the system information obtained from the command line:
uname -r: 5.15.0-46-generic
lsb_release -a: Ubuntu 20.04.5 LTS
nvcc --version: Cuda compilation tools, release 10.1, V10.1.243
nvidia-smi:
Two Tesla K80 (2-in-1 card) and one GeForce (Gigabyte RTX 3060 Vision 12G rev . 2.0)
NVIDIA-SMI 470.82.01
Driver Version: 470.82.01
CUDA Version: 11.4
cmake --version: cmake version 3.22.5
make --version: GNU Make 4.2.1
lscpu: Xeon CPU E5-2680 V4 # 2.40GHz - 56 CPU(s)
Observation
Although there are two GPUs installed in the server, nvCOMP only works with the RTX.
The Steps
Perhaps "installation" is a misnomer. One only needs to properly compile the downloaded nvCOMP files and run the resulting executables.
Step 1: The nvCOMP library
Download the nvCOMP library from https://developer.nvidia.com/nvcomp.
The file I downloaded was named nvcomp_install_CUDA_11.x.tgz. And I left the extracted folder in the Downloads directory and renamed it nvcomp.
Step 2: The nvCOMP test package on GitHub
Download it from https://github.com/NVIDIA/nvcomp. Click the green "Code" icon, then click "Download ZIP".
By default, the downloaded zip file is called nvcomp-main.zip. And I left the extracted folder, named nvcomp-main, in the Downloads directory.
Step 3: The NIVIDIA CUB library on GitHub
Download it from https://github.com/nvidia/cub. Click the green "Code" icon, then click "Download ZIP".
By default, the downloaded zip file is called cub-main.zip. And I left the extracted folder, named cub-main, in the Downloads directory.
There is no "installation" of the CUB library other than making the folder path "known", ie available, to the calling program.
Comments: The nvCOMP GitHub site did not seem to explain that the CUB library was needed to run nvCOMP, and I only found that out from an error message during an attempted compilation of the test files in Step 2.
Step 4: "Building CPU and GPU Examples, GPU Benchmarks provided on Github"
The nvCOMP GitHub landing page has a section with the exact name as this Step. The instructions could have been more detailed.
Step 4.1: cmake
All in the Downloads directory are the folders nvcomp(the Step 1 nvCOMP library), nvcomp-main (Step 2), and cub-main (Step 3).
Start a terminal and then go inside nvcomp-main, ie, go to /your-path/Downloads/nvcomp-main
Run cmake -DCMAKE_PREFIX_PATH=/your-path/Downloads/nvcomp -DCUB_DIR=/your-path/Downloads/cub-main
This cmake step sets up the build files for the next make" step.
During cmake, a harmless yellow-colored cmake warning appeared
There was also a harmless printout "-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed" per this thread.
The last few printout lines from cmake variously stated it found Threads, nvcomp, ZLIB (on my system) and it was done with "Configuring" and "Build files have been written".
Step 4.2: make
Run make in the same terminal as above.
This is a screenshot of the make compilation.
Please check the before and after folder tree to see what files have been generated.
Step 5: Running the examples/benchmarks
Let's run the "built-in" example before running the benchmarks with the (now outdated) Fannie Mae single-family loan performance data from NVIDIA's RAPIDS repository.
Check if there are executables in /your-path/Downloads/nvcomp-main/bin. These are the excutables created from the cmake and make steps above.
You can try to run these executables on your to-be-compressed files, which are buit with different compression algorithms and functionalities. The name of the executable indicates the algorithm used and/or its functionality.
Some of the executables require the files to be of a certain size, eg, the "benchmark_cascaded_chunked" executable requires the target file's size to be a multiple of 4 bytes. I have not tested all of these executables.
Step 5.1: CPU compression examples
Per https://github.com/NVIDIA/nvcomp
Start a terminal (anywhere)
Run time /your-path/Downloads/nvcomp-main/bin/gdeflate_cpu_compression -f /full-path-to-your-target/my-file.txt
Here are the results of running gdeflate_cpu_compression on an updated Fannie Mae loan data file "2002Q1.csv" (11GB)
Similarly, change the name of the executable to run lz4_cpu_compression or lz4_cpu_decompression
Step 5.2: The benchmarks with the Fannie Mae files from NVIDIA Rapids
Apart from following the NVIDIA instructions here, it seems the "benchmark" executables in the above "bin" directory can be run with "any" file. Just use the executable in the same way as in Step 5.1 and adhere to the particular executable specifications.
Below is one example following the NVIDIA instruction.
Long story short, the nvcomp-main(Step 2) test package contains the files to (i) extract a column of homogeneous data from an outdated Fannie Mae loan data file, (ii) save the extraction in binary format, and (iii) run the benchmark executable(s) on the binary extraction.
The Fannie Mae single-family loan performance data files, old or new, all use "|" as the delimiter. In the outdated Rapids version, the first column, indexed as column "0" in the code (zero-based numbering), contains the 12-digit loan IDs for the loans sampled from the (real) Fannie Mae loan portfolio. In the new Fannie Mae data files from the official Fannie Mae site, the loan IDs are in column 2 and the data files have a csv file extension.
Download the dataset "1 Year" Fannie Mae data, not the "1GB Splits*" variant, by following the link from here, or by going directly to RAPIDS
Place the downloaded mortgage_2000.tgz anywhere and unzip it with tar -xvzf mortgage_2000.tgz.
There are four txt files in /mortgage_2000/perf. I will use Performance_2000Q1.txt as an example.
Check if python is installed on the system
Check if text_to_binary.py is in /nvcomp-main/benchmarks
Start a terminal (anywhere)
As shown below, use the python script to extract the first column, indexed "0", with format long, from Performance_2000Q1.txt, and put the .bin output file somewhere.
Run time python /your-path/Downloads/nvcomp-main/benchmarks/text_to_binary.py /your-other-path-to/mortgage_2000/perf/Performance_2000Q1.txt 0 long /another-path/2000Q1-col0-long.bin
For comparison of the benchmarks, run time python /your-path/Downloads/nvcomp-main/benchmarks/text_to_binary.py /your-other-path-to/mortgage_2000/perf/Performance_2000Q1.txt 0 string /another-path/2000Q1-col0-string.bin
Run the benchmarking executables with the target bin files as shown at the bottom of the web page of the NVIDIA official guide
Eg, /your-path/Downloads/nvcomp-main/bin/benchmark_hlif lz4 -f /another-path/2000Q1-col0-long.bin
Just make sure the operating system know where the executable and the target file are.
Step 5.3: The high_level_quickstart_example and low_level_quickstart_example
These two executables are in /nvcomp-main/bin
They are completely self contained. Just run eg high_level_quickstart_example without any input arguments. Please see corresponding c++ source code in /nvcomp-main/examples and see the official nvCOMP guides on GitHub.
Observations after some experiments
This could be another long thread but let's keep it short. Note that NVIDIA used various A-series cards for its benchmarks and I used a GeForce RTX 3060.
Speed
The python script is slow. It took 4m12.456s to extract the loan ID column from an 11.8 GB Fannie Mae data file (with 108 columns) using format "string"
In contract, R with data.table took 25.648 seconds to do the same.
With the outdated "Performance_2000Q1.txt" (0.99 GB) tested above, the python script took 32.898s whereas R took 26.965s to do the same extraction.
Compression ratio
"Bloated" python outputs.
The R-output "string.txt" files are generally a quarter of the size of the corresponding python-output "string.bin" files.
Applying the executables to the R-output files achieved much better compression ratio and throughputs than to the python-output files.
Eg, running benchmark_hlif lz4 -f 2000Q1-col0-string.bin with the python output vs running benchmark_hlif lz4 -f 2000Q1-col0-string.txt with the R output
Uncompressed size: 436,544,592 vs 118,230,827 bytes
Compressed size: 233,026,108 vs 4,154,261 bytes
Compressed ratio: 1.87 vs 28.46 bytes
Compression throughput (GB/s): 2.42 vs 18.96
decompression throughput (GB/s): 8.86 vs 91.50
Wall time: 2.805 vs 1.281s
Overall performance: accounting for file size and memory limits
Use of the nvCOMP library is limited by the GPU memory, no more than 12GB for the RTX 3060 tested. And depending on the compression algorithm, an 8GB target file can easily trigger a stop with cudaErrorMemoryAllocation: out of memory
In both speed and compression ratio, pigz trumped the tested nvCOMP excutables when the target files were the new Fannie Mae data files containing 108 columns of strings and numbers.
"ceph" build instructions are here https://docs.ceph.com/en/latest/install/build-ceph/
but they do not work. I get
tjcw#cuttlefisharmserver:~/workspace/ceph_v16.2.6/build$ ninja -j256
ninja: error: loading 'build.ninja': No such file or directory
tjcw#cuttlefisharmserver:~/workspace/ceph_v16.2.6/build$
There is a build.ninja file in the directory tree
tjcw#cuttlefisharmserver:~/workspace/ceph_v16.2.6$ find . -name build.ninja
./src/seastar/build.ninja
tjcw#cuttlefisharmserver:~/workspace/ceph_v16.2.6$
but that does not seem to be the right place.
What should I do next ? I am new to building ceph, so it could easily be finger trouble. I am trying to build on an ARM64 machine, not x86-64
tjcw#cuttlefisharmserver:~/workspace/ceph_v16.2.6$ uname -a
Linux cuttlefisharmserver 5.4.0-91-generic #102-Ubuntu SMP Fri Nov 5 16:30:45 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux
tjcw#cuttlefisharmserver:~/workspace/ceph_v16.2.6$
This is a fully-patched Ubuntu 20.4 machine
The solution was to use 'make' instard of 'ninja'. Ceph is in process of transitioning its build system from make to ninja; the level of Ceph I was trying to build requires 'make'.
After building the RISCV tools and GCC (cloned from lowrisc, isa-sim and not riscv-tools), i'm stuck in the debugging with Gdb phase here.
In the second terminal target remote in gdb times out.
In the first terminal when i run spike --gdb-port 9824 pk tests/debug or spike --gdb-port 9824 pk hello.c it yields:
spike: unrecognized option --gdb-port
usage: spike [host options] <target program> [target options]
Host Options:
-p <n> Simulate <n> processors
-m <n> Provide <n> MB of target memory
-d Interactive debug mode
-g Track histogram of PCs
-h Print this help message
--ic=<S>:<W>:<B> Instantiate a cache model with S sets,
--dc=<S>:<W>:<B> W ways, and B-byte blocks (with S and
--l2=<S>:<W>:<B> B both powers of 2).
--extension=<name> Specify RoCC Extension
--extlib=<name> Shared library to load
I don't know if it has to do with configuring gdb on its own ? Or is it built and configured when i ran ./build.sh for the riscv tools.
If not, could you please correct the --gdb-port command (I'm new to linux) I've tried --gdb-port=9824 or --gdb-port:9824 and it's the same.
Thank you
Message spike: unrecognized option --gdb-port says that spike, not gdb can't recognize option. Spike is from riscv-isa-sim, not from riscv-tools. And LowRisc variant of Spike - https://github.com/lowRISC/riscv-isa-sim is many commits behind master:
This branch is 3 commits ahead, 172 commits behind riscv:master.
Latest commit e220bc4 on May 19, 2016 #wsong83 wsong83 Merge commit '0d084d5' into update
One of not ported commit added gdb support to spike from https://github.com/riscv/riscv-isa-sim (and documented it in https://github.com/riscv/riscv-isa-sim#debugging-with-gdb), but it is not pulled to https://github.com/lowRISC/riscv-isa-sim (and not documented at https://github.com/lowRISC/riscv-isa-sim). gdb-related commits were from Oct 2016, Jun 2016, May 2016, and the --gdb-port was added in d1d8863086c57f04236418f21ef8a7fbfc184b0b (Mar 19, 2016) https://github.com/riscv/riscv-isa-sim/commit/d1d8863086c57f04236418f21ef8a7fbfc184b0b
+ fprintf(stderr, " --gdb-port=<port> Listen on <port> for gdb to connect\n");
+ parser.option(0, "gdb-port", 1, [&](const char* s){gdb_port = atoi(s);});
You can try merging changes between isa sims or ask lowRisc authors to merge or just try to use spike from riscv...
I want to move to make 4.2 so I can have the ability to do parallel jobs on Windows using --jobs=N.
make 4.2 is not available in binary form from SourceForge, only 3.81
So I had to build it from source - they included a VS studio project - and I was up and running.
Problem is, it appears to have a bug though it could be a bug with my makefile.
I ran make4.2 all --print-data-base --debug=vjm on my project and I got ERRORS.
What's unfortunate is that the error is NOT related to syntax of rules as this project builds without error on make 3.81 using the same options.
Here is the command it fails on.
COPY_DIR = cp -f -R
MKDIR = mkdir -p
deploy_marketProperties: deploy_themeMappingConfig
#echo Copying application data... && $(MKDIR) "C:/Users/User1/Desktop/TargetDir" && $(COPY_DIR) $(wildcard C:/Users/User1/Desktop/Source/*) "C:/Users/User1/Desktop/TargetDir"
Creates a directory
Copies bunch of files from one directory to that new directory
Used the same make call for both versions of make:
make all --print-data-base --debug=vjm
However, I get the following error when using make4.2:
Copying application data...
cp: target `C' is not a directory
makev42[3]: *** [Makefile:383: deploy_marketProperties]
The command it tried running was:
echo Copying application data... && mkdir -p "C:/Users/User1/Desktop/TargetDir" && cp -f -R C:/Users/User1/Desktop/Source/im.json "C:/Users/User1/Desktop/TargetDir"
There's actually 264 .json files but I only show one for readability.
Again, this runs fine on make v3.81 using exact same command-line options.
It appears make 4.2 has a problem parsing that huge list of files maybe??
The question
Is there something I can change with my makefiles to make it run on make v4.2?
We are highly motivated to make it work with a 4.0+ version of make. Otherwise we have to use a build tool that will cost $$.
UPDATE 1
Per the suggestion of #MadScientist, I have executed both versions of make and present the outputs.
C:\release>make381 deploy_marketProperties --debug=vjm > 381.err 2>&1
C:\release>make42 deploy_marketProperties
--debug=vjm > 42.err 2>&1
The target deploy_marketProperties has the following rule:
deploy_marketProperties:
echo Copying application data... && $(MKDIR) "C:/Users/User1/Desktop/A/Project/src/../lib/armle-v7/release/marketProperties" && $(COPY_DIR) $(wildcard C:/Users/User1/Desktop/A/Project/src/dir1/marketProperties/*) "C:/Users/User1/Desktop/A/Project/src/../lib/armle-v7/release/marketProperties"
The output of make v 3.81.
NOTE: there's actually 264 .json files and total number of characters that make up the list of .json files, including ONE white space per file, is 18251 (in case that's relevant):
GNU Make 3.81
Copyright (C) 2006 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
This program built for Windows32
find_and_set_shell path search set default_shell = C:/QNX650/host/win32/x86/usr/bin/sh.exe
Reading makefiles...
Reading makefile `Makefile'...
Updating goal targets....
Considering target file `deploy_marketProperties'.
File `deploy_marketProperties' does not exist.
Finished prerequisites of target file `deploy_marketProperties'.
Must remake target `deploy_marketProperties'.
echo Copying application data... && mkdir -p "C:/Users/User/Desktop/A/Project/src/../lib/armle-v7/release/marketProperties" && cp -f -R C:/Users/User/Desktop/A/Project/src/fordhmi/marketProperties/2blank.json C:/Users/User/Desktop/A/Project/src/fordhmi/marketProperties/3blank.json "C:/Users/User/Desktop/A/Project/src/../lib/armle-v7/release/marketProperties"
Main thread handle = 0x000000ac
Copying application data...
Successfully remade target file `deploy_marketProperties'.
The output of make v 4.2.
NOTE: there's actually 264 .json files and total number of characters that make up the list of .json files, including ONE white space per file, is 18251 (in case that's relevant):
GNU Make 4.2
Built for Windows32
Copyright (C) 1988-2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
find_and_set_shell() path search set default_shell = C:/QNX650/host/win32/x86/usr/bin/sh.exe
Reading makefiles...
Reading makefile 'Makefile'...
Updating makefiles....
Updating goal targets....
Considering target file 'deploy_marketProperties'.
File 'deploy_marketProperties' does not exist.
Finished prerequisites of target file 'deploy_marketProperties'.
Must remake target 'deploy_marketProperties'.
echo Copying application data... && mkdir -p "C:/Users/User1/Desktop/A/Project/src/../lib/armle-v7/release/marketProperties" && cp -f -R C:/Users/User1/Desktop/A/Project/src/SubProj/marketProperties/im.json C:/Users/User1/Desktop/A/Project/src/SubProj/marketProperties/da.json "C:/Users/User1/Desktop/A/Project/src/../lib/armle-v7/release/marketProperties"
Main thread handle = 00000088
Copying application data...
cp: target `C' is not a directory
make_msvc.net2003: *** [Makefile:384: deploy_marketProperties] Error 1
Note that both make versions use the same shell:
C:\Users\User1\Desktop\A\Project\bld\armle-v7\release\subProj>where sh
C:\QNX650\host\win32\x86\usr\bin\sh.exe
C:\Users\User1\Desktop\A\Project\bld\armle-v7\release\subProj>sh --help
GNU bash, version 3.1.17(1)-release-(i686-pc-msys)
Usage: sh [GNU long option] [option] ...
sh [GNU long option] [option] script-file ...
GNU long options:
--debug
--debugger
--dump-po-strings
--dump-strings
--help
--init-file
--login
--noediting
--noprofile
--norc
--posix
--protected
--rcfile
--restricted
--verbose
--version
--wordexp
Shell options:
-irsD or -c command or -O shopt_option (invocation only)
-abefhkmnptuvxBCHP or -o option
Type `sh -c "help set"' for more information about shell options.
Type `sh -c help' for more information about shell builtin commands.
Use the `bashbug' command to report bugs.
NOTE THE msys #MadScientist; is that relevant?
UPDATE 2
I re-ran make 4.2 but I deleted all but ONE .json file in the directory and - surprise! - it works.
This is the ENTIRE output from make42 deploy_marketProperties --debug=vjm:
GNU Make 4.2
Built for Windows32
Copyright (C) 1988-2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
find_and_set_shell() path search set default_shell = C:/QNX650/host/win32/x86/usr/bin/sh.exe
Reading makefiles...
Reading makefile 'Makefile'...
Updating makefiles....
Updating goal targets....
Considering target file 'deploy_marketProperties'.
File 'deploy_marketProperties' does not exist.
Finished prerequisites of target file 'deploy_marketProperties'.
Must remake target 'deploy_marketProperties'.
echo Copying application data... && mkdir -p "C:/Users/User1/Desktop/A/Project/src/../lib/armle-v7/release/marketProperties" && cp -f -R C:/Users/User1/Desktop/A/Project/src/subProject/marketProperties/2blank.json "C:/Users/User1/Desktop/A/Project/src/../lib/armle-v7/release/marketProperties"
Main thread handle = 000000B4
Copying application data...
Successfully remade target file 'deploy_marketProperties'.
UPDATE 3
I tried it with make 4.2.1 and I got the exact same behavior as on make 4.2: fails when there are hundreds of .json files but succeeds if there less than some magic number of characters.
I got this message during one of my trials:
cp: target `C:/Users/User1/Des' is not a directory
Obviously it's getting cut-off but why?
Same shell being used and only thing different is version of make.
You should use 4.2.1, it has some important fixes (I doubt it will make a difference for this problem).
As discussed in the comments, using POSIX tools on Windows requires a specialized environment. The normal "shell" on Windows is command.com, which doesn't accept POSIX sh syntax, such as &&. The normal mkdir command on Windows doesn't accept the -p flag; that's a POSIX flag. There is no cp command on Windows.
So it's clear that your makefile is expecting to be run with some sort of POSIX environment.
Based on what you've described above, it's not clear how the problem is related to GNU make directly. GNU make is parsing the makefile and it's invoking the shell and passing along the command line properly: this can be inferred because the shell is running the echo command and running the mkdir command, and even running the cp command.
It's the cp command which is printing the message, not make. I believe you that it works with 3.81, but I suspect a non-obvious difference between the way 3.81 was compiled and the way you're building 4.2. It's even possible that there have been custom patches applied to the 3.81 version you're using, although I don't know.
To debug further I recommend the following:
First remove the # prefix from the recipe line: that's always the number one thing to do; if you can't see how make is invoking the command how can you debug?
Then run the same makefile with the two different versions of make, from the same shell/prompt (so the path to make is the only difference). Is there any difference in the printed output? If so what is it?
If not, then clearly there's something more mysterious going on. Try to see if maybe the QNX version of make is modifying the environment somehow: you can run the env command in your recipe to see it.
Maybe it's related to the length of the command line? Try temporarily moving some of the json files out of that directory so the command line is shorter.
I asked the make-w32#gnu.org mailing list and I have the answer.
To fix it, I had to uncomment the macro BATCH_MODE_ONLY_SHELL in config.h of the make source code.
/*
* If you have a shell that does not grok 'sh -c quoted-command-line'
* correctly, you need this setting. Please see below for specific
* shell support.
*/
#define BATCH_MODE_ONLY_SHELL 1
Taking quote from reply from mailing list:
What that flag means (as I understand it: I'm not that familiar with
this aspect of Windows support) is that make will never try to invoke
the shell directly passing the recipe to be run on the command line.
Instead it is being forced to always write the recipe to a temporary
file ("batch file") on your disk and invoke the shell such that it
runs the recipe in the temporary file.
Apparently your shell is able to read and execute very long command
lines from a file, which it cannot successfully process when passed in
as arguments, even though these command line arguments are not
exceeding Windows limits.
I'm having performance issues with lcov.
I'm executing a program in seven different profiles, collecting the coverage for each of them and then merging the coverage profile with lcov:
lcov --rc lcov_branch_coverage=1 -a coverage_1.dat -a coverage_2.dat -a coverage_3.dat -a coverage_4.dat -a coverage_5.dat -a coverage_6.dat -a coverage_7.dat -o coverage_full.dat
However, this is excruciatingly slow. It takes about 10 minutes to combine my 7 profiles, this is actually longer than it takes to compile and run the 7 profiles. Each dat file is about 1M lines.
The lcov --combine and lcov --remove steps are very slow as well. Around 45 seconds for each of them.
Is there any way to speedup this combine step ? I can use several threads if necessary and I have got plenty of memory. If there are other tool that are able to do this combination correctly, I'd be interest as well (I've tried to convert the files to Cobertura and do the merge with a Python script I found, but it crashes).
If there is an alternative to lcov completely, I'm also interested. I've been using gcovr, but with it, I have to use several other tools to do the combination and it is not optimal, but it is much faster.
If there is an alternative to lcov completely, I'm also interested.
Try fastcov - it will use all available cores in parallel (it can output the report in lcov info format):
https://github.com/RPGillespie6/fastcov
It can also combine files. Note: You need GCC 9+