How to use MPE for MPI c++ project? - c++

MPE is very useful for visualizing MPI programs, but it only provides compiler wrappers for C and Fortran: mpecc and mpef77, respectively. How can I use it if my MPI project is written in C++ and is normally compiled with mpic++, not mpicc (and so it can't be compiled with mpecc)?
How do I setup (1) the MPE library itself and (2) my C++ project?

The answer is actually quite simple, nonetheless I struggled with it for days.
Steps shown below works with MPICH 3.3.2 on Linux (Ubuntu 18) run with WSL2 - some adjustments may be necessary in different evnironments.
MPE library setup for c++
You setup the MPE library normally, the same way you would for a C project - the necessary steps are:
Download and extract the latest MPE archive (I've used MPE 2-1.4.9 from here)
Navigate to extracted directory:
cd mpe2-2.4.9b
Configure library's build process - in my case the following command worked:
./configure MPI_CC=mpicc MPI_F77=mpif77 prefix=$HOME/installs/mpe2 MPI_CFLAGS=-pthread MPI_FFLAGS=-pthread
Explanation:
MPE is written in C, so we use mpicc to compile it - we do not (yet) specify how to build our project, so we do not use mpic++. If we use mpic++ as MPI_CC, the MPE library won't compile.
specifying Fortran flags isn't strictly necessary, but this way we avoid unneccessary errors in compilation output
prefix (installation path) is an arbitrary path of your choice, just remember what you have inserted here as it will be necessary in further steps
I had to provide manual linkage of the pthread library - this may/may not be neccessary depending on your system
Compile the MPE library:
make
Install the compiled library:
make install
Using MPE in c++ project
Since we cannot use a predefined compiler wrapper mpecc to compile c++, we have to link the necessary libraries manually, as we would do with any other library.
Suppose we have a file main.cpp with following content:
#include <mpi.h>
#include <mpe.h>
/* other necessary includes */
int main(int argc, char** argv) {
MPI_Init(&argc, &argv);
MPE_Init_log();
/* some MPI and MPE stuff */
MPE_Finish_log("test");
MPI_Finalize();
}
The specific command which allows to build a c++ file with MPI and MPE calls is:
mpic++ main.cpp -o main -I/$HOME/installs/mpe2/include -L/$HOME/installs/mpe2/lib -lmpe -pthread
Explanation:
We use mpic++ to link the all MPI items automatically
$HOME/installs/mpe2 is an arbitrary installation path you've specified when configuring the MPE library
-I flag tells the compiler where to look for header files (the ones we #include)
-L flag tells the compiler where to find the compiled library items (implementation of functions defined in included header files)
-l flag tells compiler to actually link our executable with specific library (which can be found thanks to us specyfing search location with -L flag)
I had to link pthread manually for MPE to work, but that may depend on your system
If you use cmake for building your project, the following CMakeLists.txt should work:
cmake_minimum_required(VERSION 3.5) # not necessarily 3.5, it;s just my setup
project(mpi_test) # arbitrary project name
find_package(MPI REQUIRED)
set(CMAKE_CXX_COMPILER mpic++)
include_directories($ENV{HOME}/installs/mpe2/include) # again, it's the MPE installation path
link_directories($ENV{HOME}/installs/mpe2/lib) # again, it's the MPE installation path
add_executable(main_exp src/main_exp.cpp)
target_link_libraries(main_exp mpe pthread) # again, pthread may/may not be neccessary

Related

C++ - Why is a Static Library unusable without source files?

So from what I understand, a static compiled mylib.a file should just be usable as plug-and-play. But I can't use it without the source code, I'm trying to avoid that.
I adapted this tutorial to compile using CMake with the following directory structure:
/
lib
libmy_math.a
main.cpp
my_math.h
my_math.cpp
CMakeLists.txt
The contents of the files are exactly as in the tutorial. The only added file is CMakeLists.txt which basically just runs the library compiling part before building:
CMakeLists.txt
cmake_minimum_required(VERSION 3.21)
project(Test)
### Uncomment this ONCE to compile a libmy_math.a file
# add_library(my_math STATIC my_math.cpp)
add_executable(Test main.cpp)
find_library(MY_MATH_LIB my_math HINTS ./lib)
target_link_libraries(Test PUBLIC ${MY_MATH_LIB})
As the CMake file says, uncommenting Line 5 compiles the library, which is then linked to my code when compiling.
I'm trying to package my code so that I don't show my client the source code for the compiled library (my_math), but if I delete the my_math.h and my_math.cpp files, I get a "file not found" error on import:
/Users/Joe/Desktop/Test/main.cpp:1:10: fatal error: 'my_math.h' file not found
#include "my_math.h"
^~~~~~~~~~~
I thought you could compile libraries without needing the source code. What am I missing here?
A static library does not contain each and every definition a header can contain - think of macros, etc. Thus you still need the header. However, you don't need the .cpp anymore to link the library.
Besides the problem stated in Iorro's answer the find_library call here will not work as expected.
find_library(MY_MATH_LIB my_math HINTS ./lib)
target_link_libraries(Test PUBLIC ${MY_MATH_LIB})
What this does is, while running CMake, look for the my_math.a file. However, at that point of the build process, my_math.a has not been built yet. You're just running CMake for the first time, so all you have is source files. Nothing has been compiled yet.
What you want to do instead is link to the library target directly, as CMake already knows about that and can resolve the dependency on its own. So you need to replace the two lines above with:
target_link_libraries(Test PUBLIC my_math)
This now introduces a curious asymmetry in how your test consumes the library (as it is part of the same build) vs. how your clients consume the same library (which are not part of the same build and thus need to do a find_package call before they can use it). Look at CMake's config-file packages for a mechanism that allows you to restore symmetry here, at the cost of a significant amount of boilerplate required in your CMake script.

How to correctly set GSL path on mac

I have 'correctly' installed gsl on mac 10.13.2.
In my c++ program, I am calling like usual, for example:
#include <gsl/gsl_math.h>
However while running the code, it can not find the gsl.
fatal error: 'gsl/gsl_math.h' file not found
I was wondering how to correctly link gsl PATH and libraries.
I have tried,
setting PATH and LD_LIBRARY_PATH in .bash_profile
setting PKG_CONFIG_PATH to .../Gsl2.3/lib/pkgconfig
$which gsl-config returns
/Users/gkdgoutam/Softwares/HEP_Softwares/Install/Gsl2.3/bin/gsl-config
$pkg-config --libs gsl returns
-L/Users/gkdgoutam/Softwares/HEP_Softwares/Install/Gsl2.3/lib -lgsl -lgslcblas -lm
The only solution I can find is to run everytime with gsl linked.
Like:
g++ $(gsl-config --cflags) mycode.cc $(gsl-config --libs) && ./a.out
But I was wondering if the GSL PATH can be set globally so that I can simply run
g++ mycode.cc && ./a.out
This is how c++ code is compiled and built:
COMPILATION
A compilation unit will take each cpp file and work its way through included headers to locate forward declaration of implementations of symbol signatures of used functionality in your code. In your case this involves gsl/gsl.h. If the file cannot be found in the search directories, which you can expand by specifying C_INCLUDE_PATH and or CPLUS_INCLUDE_PATH. If you omit #include <gsl/gsl_math.h>, your code will not compile as there are signatures, which cannot be found for GSL functions, which you use.
LINKING
Once you have compiled all cpp/cc files you need to link the binary, which can be executed. The linking process consists of a search through all symbols in your .o/.obj... files and a matching of the same to what it can find in your object files and the libraries, which you have specified using for example -lgsl. If all goes well, every lookup finds an according binary implementation for your machine's hardware model (i.e. 64bit/32bit ..., Arm, Intel, ... etc). If some are not found you will find linkage errors.
What you are asking is, is there a way that C++ does not work as above? No!
If you leave out #include <gsl/gsl.h> or if said file is not found in your search paths, compilation will not work or. If you omit -lgsl, linking will fail. If you find it annoying to write all the above stuff in the command line, write a Makefile to reduce the building process to ideally a simple command: make.
Don't forget, that if you are linking against the shared library version of GSL, you might need specifying LD_LIBARAY_PATH on Linux and DYLD_LIBRARY_PATH on Macs as well.
TLDR: you cannot ask a c++ compiler / linker to work differently as designed.

Undefined reference when trying to use external library

I am trying to incorporate a C library into some Rcpp code.
I can use the C library in a C++ program easily. I 'make' the C library, which creates the .a and .dll files in the /lib folder. I can then use the package by including the header in the program and running something like this from command line:
cc myfile.cpp -o myfile -Ipath.to.header path.to.lib.a -lz
This essentially tells the compiler to take the .cpp program, include headers from -I, and to link to two libraries.
Getting this to work with Rcpp shouldn't be overly difficult if I understand makevars correctly (which I unfortunately don't seem to).
I add the library to a folder in my package, and in src I add a makevars and makevars.win that look like this:
PKG_CFLAGS=
# specify header location
PKG_CPPFLAGS=-Ipath.to.lib/include
# specify libs to link to
PKG_LIBS=path.to.lib/lib/file.a -lz
# make library
path.to.lib/lib/file.a:
cd path.to.lib;$(MAKE)
This correctly 'makes' the .a and .dll files for the library, however none of the Rcpp magic runs (i.e. in the build I never see the g++ system call that compiles the files in src), so "no Dll was created".
I am fairly certain this is a problem in my makevars target that makes the library. When I remove that portion from the makevars, and 'make' the library from the command line myself before building the package, I get the proper g++ calls with my -I and -l statements, but I get errors about undefined references.
I notice that the -l statements are only included in the final g++ call where the final .dll is made, but isn't included in the earlier g++ calls where the files with the library headers are compiled.
So I have two problems:
How do I fix my makevars so that it 'makes' the library, but doesn't stop Rcpp from compiling the files in src?
How do I deal with the undefined references? The library is clearly not header-only, so I am guessing it needs the -l statement in the earlier g++ calls, but that may not even be possible.
The best approach is to avoid complicated src/Makevars file altogether.
One easy-ish approach around this: use configure to build your static library, then once you actually build just refer to it in src/Makevars.
I use that scheme in Rblpapi (where we copy an externally supplied library in) and in nloptr where we download nlopt sources and build it 'when needed' (ie when no libnlopt is on the system).

How to Specify the Compiler and To Compile Different Main Functions

My project is quite simple but I like to keep files in different folders for clarity. For instance I have three folders.
An Output folder that contains all the classes used for the output. (for now only Output.cc and Output.h).
A Math folder, containing all of the classes related to math. (vector.cc, vector.h, randomgen.h, randomgen.cc, etc)
A Tests folder where there are different cpp files each containing a main function. (Output_test.cc, vector_test.cc, etc)
How can I create a CMake script that complies all of the different main function of the different test programs using the classes that are in different folders?
In addition, I didn't see where the compiler, and its options, are specified in a CMake file.
How to specify the compiler?
There are a few ways to specify the compiler you want to use. Settings environment variables, defining compiler variables, or designating a generator.
Settings Environment Variables
There are two ways to use environment variables to help CMake determine which compiler to use during a CMake configuration. Using the PATH variable or the CC and CXX variables.
PATH
Make sure the path to your desired compiler is first in the list. If you don't want to modify your path, then use the 2nd option.
CC & CXX
CMake reads the variables CC and CXX to determine the path for the C compiler and C++ compiler respectively. Note that the first time CMake is configured for a project it will cache these paths, and look to the cache first for all future configurations. So if you wish to change the compiler path, be sure to delete the cache file CMakeCache.txt. As HughB pointed out, there is a good example of this given by Guillaume
Defining Compiler Variables
Similar to using CC and CXX, there are CMake variables that can be defined at the commandline to choose the desired compiler. They are CMAKE_C_COMPILER and CMAKE_CXX_COMPILER. You can set them using the -D option and they use the same values as CC and CXX. Note, just like CC and CXX these are cached after the first CMake configuration.
Example
cmake -DCMAKE_CXX_COMPILER=/usr/bin/g++4.6/g++ -DCMAKE_C_COMPILER=/usr/bin/gcc4.6/gcc <path to src>
Designating a Generator
The last way to choose the compiler is by using the -G option to select a generator. There are numerous generators to choose and I recently answered a question about them. I'm not going to go into too much detail about them in this answer, but you can read my other answer for more details.
Example
cmake -G "Unix Makefile" <path to src>
Don't Hardcode the Compiler
I recommend resisting the urge to "hardcode" the compiler in the CMakeLists.txt files. CMake is meant to be compiler independent and you should be setting the compiler information external of the CMake files. Use one of the methods mentioned above to keep your CMake files portable and to avoid confusion if yourself or someone else wants to build the project with a different compiler.
Related references. (Disclaimer: Written by me)
What is a CMake generator
Understanding the Purpose Behind CMake
How to Compile Multiple Executables?
As HughB mentioned use add_executables. You could also create separate libraries for each folder group, there are many ways to organize your project. I'm going to keep it simple and give an example that builds two executables in the same project.
For this example I created 5 files:
include/Helloworld.h
#include <stdio.h>
src/HelloWorld.cpp
#include "HelloWorld.h"
int main()
{
printf("Hello World!\n");
return 0;
}
include/HelloWorld2.h
#include <stdio.h>
src/HelloWorld2.cpp
#include "HelloWorld2.h"
int main()
{
printf("Hello World2!\n");
return 0;
}
CMakeLists.txt
cmake_minimum_required(VERSION 3.0)
# This is required to compile and must contain the paths to
# all of the header files
include_directories(include)
# These are optional, but are added to be passed to the
# add_executable command so that the headers are accessible
# from within the project
set(HW_INCLUDES include/HelloWorld.h)
set(HW_INCLUDES2 include/HelloWorld2.h)
project(HelloWorlds)
# Required sources
set(HW_SOURCES src/HelloWorld.cpp)
set(HW_SOURCES2 src/HelloWorld2.cpp)
add_executable(HelloWorld ${HW_SOURCES} ${HW_INCLUDES})
add_executable(HelloWorld2 ${HW_SOURCES2} ${HW_INCLUDES2})
When this project is built there will be two executables: HelloWorld.exe and HelloWorld2.exe.
Consider putting the code that will be used in different program mains in a library. Use the add_library command to do make a library. You could have a directory hierarchy like this:
TopDir
CMakeLists.txt
MyLib
CMakeLists.txt
vector.cc
vector.h
....
MyExe1
CMakeLists.txt
main1.cc
MyExe2
CMakeLists.txt
main2.cc
Use add_subdirectory in your top level cmakelists.txt to traverse the directories. In the dirs that build executables, use add_executable and target_link_libraries. If you named the library MyLib then your target_link_libraries command would look like
target_link_libraries( exe1 MyLib )
In regard to overriding the compiler, see how to specify new gcc path for cmake

How to setup GLFW with eclipse c++ and MinGW compiler?

How do I include glfw.h and link the libraries libglfw.a and libglfadll.a in eclipse juno c++ with MinGW compiler. This is an attempt I made on setting it up:
This is the build command I tried to use:
g++ -o Practice.exe "src\\main.o" "-lC:\\Users\\Kaiden.ZEUS\\Files" & "Folders\\Programming\\C++\\Workspaces\\Practice\\Practice\\lib\\libglfw.a" "-lC:\\Users\\Kaiden.ZEUS\\Files" & "Folders\\Programming\\C++\\Workspaces\\Practice\\Practice\\lib\\libglfwdll.a"
Nothing of this is specific to OpenGL or GLEW, you're dealing with basic programmer craftsmanship skills here: How to configure a compiler linker toolchain to use additional libraries. This is essential knowledge, so please take the patience to properly learn it. The following is just a short list of notes what you should change. But you should really take up some learning material on the compilation and linking process to understand it.
You should place the libraries and headers into system wide directories, but not the standard directories of the compiler suite and configure those as additional search paths for the compiler and linker.
DO NOT put 3rd party library and header files into your project source tree, unless you take proper precautions that it won't interfere with likely installed systemwide instances of them.
Also you must choose between the static or the dynamically linked version of GLFW. If you use both you'll get symbol conflicts (this is something specific to GLFW).
In your build command line you're using the -loption with *directories*. This is wrong, search paths are specified using-L(capital L), while-l(lower l) just specifies library names without the path, prefix and suffix. Also you can replace backslashes` with forward slashes /, saving you some typing, i.e. the \\ escaping to produce a single backslashe to the command. In your case (I shortened the path)
g++ -o Practice.exe "src/main.o" "-LC:/Users/Kaiden.ZEUS/Files/ ... /lib" "-lglfw"
or
g++ -o Practice.exe "src/main.o" "-LC:/Users/Kaiden.ZEUS/Files/ ... /lib" "-lglfwdll"
However this compile command lacks specification of the include files. Say you've got the GLEW headers installed in C:/Users/Kaiden.ZEUS/Files/ ... /include/GL you'd add
"-IC:/Users/Kaiden.ZEUS/Files/ ... /include/GL"
to the command line.