Is there a way to control the naming of symbols in shared libraries? Specifically, I've been using GCC for a project where we access shared library through C-Types in Python. This works great, however recently I've been working with a system where the Intel compiler is suggested. I can build the shared object just fine but I find the symbols have a slightly different naming convention as compared to intel. Specifically when I compile with gcc the symbol names look like:
__test_function_MOD_read_a_file
the Intel compiled shared object has the symbol names like:
test_function_mp_read_a_file__
Is there a way to force consistency of the naming or at least change the name of the symbols after the fact?
For example consider the following bit of code test_function.f90
MODULE test_function
CONTAINS
SUBROUTINE read_a_file
PRINT *,'I did a thing!'
END SUBROUTINE
END
The compile line should look something like
gfortran -fPIC -c test_function.f90
gfortran -shared -o libtest_function.so test_function.o
Both compilers are mangling the names of the subprograms contained in the module. The Fortran standard does not mandate any naming convention. You can prevent the name mangling by using the ISO C Binding feature of Fortran to give the subprogram a specific name. For example,
module bar
contains
function fun(x) bind(c, name='foo')
real fun, x
fun = x
end function fun
function goo(x)
real goo, x
goo = x
end function goo
end module bar
When compiled with gfortran the resulting object file contains
gfortran -c a.f90
nm a.o
00000000 T __bar_MOD_goo
00000013 T foo
Thus, you can reference functionfun as foo in the library. You'll probably also want to use the types defined by the iso_c_binding module.
Related
Background: I am in a situation where I have to make use of an old Fortran95 library in a new C++ project. The F95 library is extensive, has tons of small modules, poorly documented, and it was mostly auto-generated about a decade ago by some obscure computer algebra system (by people on a different continent). So basically heirloom code, but it works and it is currently irreplaceable.
Luckily I have the source code and it can be compiled with current versions of ifort, but I am not too familiar with Fortran, and would rather not touch the old code in any significant way.
So suppose I have this Fortran code (pes_shell.f90):
subroutine pes_init()
use pes,wp=>pes_wp
implicit none
real,parameter::auang=0.5291772083
call pes0_init (dir='coef')
call pes1_init (pes_x3y1z1u1_sysall)
return
end subroutine pes_init
The functions pes0_init(...) and pes1_init(...) lead into the abyssal depths of the Fortran library and they are contained in the pes module.
I can compile this to an object file, if I give ifort the path for the modules:
ifort -r8 -O2 -c pes_shell.f90 -I/home/debianuser/PES/PES_library/lib/mod
My POC C++ code, calling pes_init():
extern "C"{
void pes_init_();
}
int main(){
pes_init_();
return 0;
}
This can also be compiled to an object file, with icpc:
icpc -c PEStest.cpp
However I cannot figure out how to link the two object files, plus the truckload of fortran modules, into a final executable.
I have tried just simply using icpc, but it cannot seem to be able to find the Fortran functions, even if I give it the location of the module files:
icpc -I/home/debianuser/PES/PES_library/lib/mod -o test.x pes_shell.o PEStest.o
pes_shell.o: In function `pes_shell_mp_f_':
pes_shell.f90:(.text+0x595): undefined reference to `pes_x3y1z1u1_mp_pes_x3y1z1u1_pot_'
pes_shell.o: In function `pes_shell_mp_pes_init_':
pes_shell.f90:(.text+0x5f0): undefined reference to `pes0_mp_pes0_init_'
pes_shell.f90:(.text+0x603): undefined reference to `pes1c_mp_pes1_init_'
PEStest.o: In function `main':
PEStest.cpp:(.text+0x2b): undefined reference to `pes_init_'
EDIT:
Pointing the linker to the directory where libpes.a can be found eliminates the problem of locating the function that was referenced in the c++ code, but icpc still cannot find the fortran functions that are being called from fortran codes:
icpc -I/home/debianuser/PES/PES_library/lib/mod -L/home/debianuser/PES/PES_library/lib/pes-xyz -lpes -o test.x PEStest.o pes_shell_new.o
pes_shell_new.o: In function `f_':
pes_shell_new.f90:(.text+0x585): undefined reference to `pes_x3y1z1u1_mp_pes_x3y1z1u1_pot_'
pes_shell_new.o: In function `pes_init_':
pes_shell_new.f90:(.text+0x5e0): undefined reference to `pes0_mp_pes0_init_'
pes_shell_new.f90:(.text+0x5f3): undefined reference to `pes1c_mp_pes1_init_'
I am trying to use FFTW3 with gfortran compiler included in the TDM-GCC-64 suite in Windows 7 platform.
I have downloaded "fftw-3.3.4-dll64.zip" from the following page:
http://www.fftw.org/install/windows.html
I also created a fortran module described in the following tutorial:
http://www.fftw.org/doc/Defining-an-FFTW-module.html
Now, I tried to compile the following fortrans program
program test
use FFTW3
implicit none
integer N
parameter(N=4)
integer*8 plan
double complex in, out
dimension in(N),out(N)
integer i
write(*,*) 'Input array:'
do i = 1,N,1
in(i) = dcmplx(float(i),float(i+1))
write(*,*) ' in(',i,') = ',in(i)
enddo
call dfftw_plan_dft_1d ( plan, N, in, out, FFTW_FORWARD, FFTW_ESTIMATE )
call dfftw_execute ( plan )
write(*,*) 'Output array:'
do i = 1,N,1
write(*,*) ' out(',i,') = ',out(i)
enddo
call dfftw_destroy_plan ( plan )
call dfftw_plan_dft_1d ( plan, N, out, in, FFTW_FORWARD, FFTW_ESTIMATE )
call dfftw_execute ( plan )
write(*,*) 'Output array after inverse FFT:'
do i = 1,N,1
write(*,*) ' ',N,' * in(',i,') = ',in(i)
enddo
call dfftw_destroy_plan ( plan )
end
I tried to compile but the a couple of error messages popped up:
undefined reference to dfftw_plan_dft_1d_'
undefined reference todfftw_execute_'
...
The command used for compiling is:
gfortran test.f90 -ffree-form -o test_fftw.exe
I searched web up and down but have not found solutions to the problem. Could anyone help me out here? Thanks!!!
You need to learn the basics of compiling and linking programs with GCC
This tutorial refers to the GCC C compiler, gcc, and C++ compiler, g++, but the principles are the same for the Fortran compiler, gfortran.
Your program invokes functions, like dfftw_plan_dft_1d that are not defined
in your code but in the fftw3 library. You must therefore tell gfortran
to link that library with your program, after it is compiled, or no definitions will be found for
these functions, and no executable can be built. Instead, the linkage
fails with the undefined reference errors you are seeing.
The command you have used to build the program:
gfortran test.f90 -ffree-form -o test_fftw.exe
attempts to compile and link the program in a single command. You can do this,
if you clearly understand the underlying separate processes of compiling
and linking. If you don't, you should follow the better practice of compiling first,
and if that is successful, then linking. The command you have used fails in
its linkage step because it does not tell the linker that the fftw3 library
is needed (or where to find it) so no attempt is made to find it and link it.
You say you have downloaded fftw-3.3.4-dll64.zip. I presume you have
unzipped it somewhere on your system. Let's say you unzipped it to C:\fftw-3.3.4-dll64.
You also say you have made an FFTW3 module that apparently is being used successfully.
In that case, compile your program first:
gfortran -c -o test.o test.f90
generating an object file test.o. The option -c tells gfortran to compile only,
and not to link. (You do not need to specify --ffree-form: the file-extension .f90
implies it).
Next, link the object file test.o with the fttw3 library, to make an executable
program test_fftw.exe:
gfortran -o test_fftw.exe test.o -LC:\fftw-3.3.4-dll64 -lfftw3-3
The linker option:
-LC:\fftw-3.3.4-dll64
tells the linker to add C:\fftw-3.3.4-dll64 (where you unzipped the package)
to the list of directories that it will search to find libraries. And the linker option -lfftw3-3 tells it to link a library that matches the identifier fftw3-3. On Windows, the GCC linker
will match a library identfier name with any of the filenames:
libname.a (a static library)
libname.dll (a shared library)
name.dll (also a shared library)
name.lib (an import library for a shared library)
So, since C:\fftw-3.3.4-dll64 contains the shared library libfftw3-3.dll,
the linker will identify it in that directory as the one required by -lfftw3-3
Now, unless there are other problems, you program will link and you will have
an executable test_fftw.exe, which will run, provided that libfftw3-3.dll
can be found, at runtime, in one of the places where the Windows
loader will look for it.
Please consider following peace of code:
// 1. Single header file. Imagine that it is some static library.
// Counter.h
#pragma once
struct Counter
{
Counter()
{
++getCount();
}
static int& getCount()
{
static int counter = 0;
return counter;
}
};
// 2. Shared library (!) :
// main_DLL.cpp
#include <iostream>
#include "counter.h"
extern "C"
{
__declspec(dllexport) // for WIN
void main_DLL()
{
Counter c;
std::cout << "main_DLL : ptr = " << &Counter::getCount()<< " value = " << Counter::getCount() << std::endl;
}
}
// 3. Executable. Shared library statically (!) linked to the executable file.
// main.cpp
#include "counter.h"
#include <iostream>
extern "C"
{
__declspec(dllimport) // for WIN
void main_DLL();
}
int main()
{
main_DLL();
Counter c;
std::cout << "main_EXE : ptr = " << &Counter::getCount() << " value = " << Counter::getCount() << std::endl;
}
Results:
Results for WIN (Win8.1 gcc 5.1.0):
main_DLL : ptr = 0x68783030 value = 1
main_EXE : ptr = 0x403080 value = 1
// conclusion: two different counters
Results for UNIX (Red Hat <I don’t remember version exactly> gcc 4.8.3):
main_DLL : ptr = 0x75693214 value = 1
main_EXE : ptr = 0x75693214 value = 2
// conclusion: the same counter addressed
Building:
Building for WIN:
g++ -c -Wall -Werror -o main_DLL.o main_DLL.cpp
g++ -shared -Wl,--out-implib=libsharedLib.a -o libsharedLib.so main_DLL.o
g++ -Wall –Werror -o simpleExample main.cpp -L./ -lsharedLib
Building for UNIX:
g++ -c -Wall -Werror -fPIC -o main_DLL.o main_DLL.cpp
g++ -shared -fPIC -o libsharedLib.so main_DLL.o
g++ -Wall –Werror -fPIC -o simpleExample main.cpp -L./ -lsharedLib
So, you see that I added –fPIC on UNIX and there is no need to create import library for UNIX, because all exports symbols are included inside shared library. On Windows I use __declspec for it.
For me, results on Windows are pretty much expected. Because shared library and executable are building separately and they should know about static variable in Counter::getCount. They should simply allocate memory for it, that’s why they have different static counters.
I did quite some analysis using tools like nm, objdump. Although I’m not a big expert in them, so I haven’t found anything suspicious. I can provide their output if needed.
Using ldd tool I can see that library linked statically in both cases.
Why I can’t see the same results on Unix for me it’s strange. Could the root cause lie in building options (–fPIC, for example), or I’m missing something?
In windows, A DLL is not exporting global and static symbols unless you add the dllexport statement, therefore, the linker doesn't even know they exists, so it allocate new instance for the static member.
In linux/unix a shared lib is exporting all the global and static symbols, so when the linker find the existence of the static member in the shared lib, it just use its address.
That is the reason for the different result.
EDIT: This is a complete rewrite of the answer. With much more details.
I think that this question deserves more elaborated answer. Especially that there are things that were not mentioned so far.
Dependency Walker
Let me start with referring to the “Dependency Walker” program.
It is a nice program (although these days a bit old-schoolish in its look & feel) that allows analyzing Windows binaries (both EXE and DLL) for symbols that they export/import and their own dependencies to other DLLs. Also it allows showing undecorated symbol names but this seems to be working only with MSVC build binaries. (And some more but that is not important here.)
Thanks to this program crucial information (for this question) can be uncovered. So I encourage you to use it during experiments.
Exporting policy on Linux vs. Windows
SHR already pointed this out but I will mention it also for completeness of the answer. And some extra details.
On Linux every symbol gets exported from a shared library by default. On the other hand on Windows you have to explicitly state which symbols to export from a shared library.
GCC seems however to provide some means of controlling exports in "Windows style". See for example Visibility entry on GCC Wiki.
Also note that there are various ways of exporting on both Linux and Windows. For example both seem to support exporting selectively by providing linker with a list of names for symbols to export. But it also seems that nowadays (on Windows at least) this isn't really used much. __declspec approach seems to be preferred.
What can be exported?
After that general introduction let's now stick to Windows case. Nowadays you export/import symbols from shared libraries by using the __declspec. Just as shown in the question. (Well maybe not exactly that - typically you use a #define to handle bi-directionality as shown in already mentioned Visibility entry on GCC Wiki.)
But the declaration can be applied not only to functions, methods and global variables. It can also be applied to types. For example you can have:
class __declspec(dllexport) Counter { /* ... */ };
Such exporting/importing means in general that all members get exported/imported.
Not so easy!
But it would be too easy, wouldn't it? The complication is that GCC and MSVC handle exporting types differently.
My notes here are based mostly on experiments (checks done using Dependency Walker) so I can be wrong or not precise enough. But I did observe differences in behavior.
In tests I used MSVC 2013 from the Express Edition with update 5. For GCC I used MinGW distro from nuwen.net, version 13.0.
MSVC, when exporting entire type, exports each and every member. Including implicitly defined members (like compiler generated copy constructor). And including inlined functions. Furthermore if inlined function has some static local variables they get exported to (!).
GCC on the other hand seems to be far more restrictive. It doesn't export implicitly defined members. Nor it doesn't export inlined members.
Exporting/Importing inline functions
If instead of exporting entire type you would explicitly export an inlined function then and only then will GCC really export it. But still it will not export static local variables in that function.
Further more if you try to import an inlined function GCC will error. With GCC you cannot define symbols that you are importing. And this happens when you import inlined (and so defined) symbol. So in fact it doesn't make any sense to export inlined functions with GCC.
MSVC allows to import inlined functions. In all cases I checked it didn't seem to actually inline the function but instead called the imported version.
Yet note that because MSVC in case of inlined function exports also its static local variables it would be possible for it to really inline the function (rather than import it) while maintaining a single copy of static local variables. For ordinary programs such behavior is mandated by the Standard (N3337, C++11), in point 7.1.2 ([dcl.fct.spec]) at $4 we can read:
(…) A static local variable in an extern inline function always refers to the same object. (…)
But a program and a shared library are actually more like two programs so they are out of scope for the Standard. Yet MSVC even in that case acts (or better to say: could act) as one would expect from a single program.
Solution
Denis Bakhvalov in a comment provided solution for his own question. The solution is to move getCount function from header to source file and export/import it.
This seems to be the only solution portable between GCC and MSVC. Or to be more precise MSVC allows more solutions to this problem but none of them will work when program is build under GCC.
The variable trick
The above is not entirely true. There is another workaround that will work consistently between GCC and MSVC.
This is to stop using static local variable. Instead make it a global variable (most likely by making it static variable in the class) and export it. This will make the trick as well.
Sadly there is no way (or I don't know any) to directly force exporting/importing static local variables. You have to change them to global variables to do that.
MSVC solutions
With MSVC you have more options.
As mentioned before exporting/importing the inlined function itself (whether directly or through type) will do the job.
Summary
As described above even consistency between GCC and MSVC on Windows only requires care. You have to limit yourself to stay in common subset of allowed solutions.
Keeping the program (source) interoperable between Linux and Windows even if with same compiler (GCC) also requires care.
Luckily there is a common subset for all three environments: GCC on Linux, GCC on Windows and MSVC on Windows. That common subset is described already by mentioned Denis' comment.
So do not inline functions that you intend to export/import. Keep them in sources. And on Windows builds (regardless of compiler) export them explicitly (otherwise you will get linker error anyway since the functions in sources of a shared library will not be available when building program).
Note that this is actually a reasonable approach on its own. Inlining function from shared library doesn't seem wise. It freezes not only the interface but also implementation (of that function). You can no longer change this function freely (and deliver new version of your shared library) since all clients would have to be rebuild since they could have inlined that function. So it is a wise approach by itself not to inline from shared library. And as a bonus it assures that your sources are multi-platform friendly.
Also do have a look into the mentioned Visibility entry on GCC Wiki. It might be reasonable to use that approach (of explicit exports) on Linux as well since it seems cleaner (from design point of view) and more efficient at runtime. While it fits well what you have to do for Windows anyway.
I am trying to compile a c++ program but it does not work.
At first, I should said that c++ is not a language I really know, I use Fortran. Anyway, the main c++ program calls a fortran subroutine. I can compile this subroutine but when I want to compile the c++ program there is a link error.
The real program is just huge and calls many subroutines, so I did a simple test (simple program calling just one subroutine) and it does not work as well! When I want to create the .x my subroutine is undefined.
Here is the c++ program (test-TQINIT.cpp) and the subroutine (TQINIT.f).
test-TQINIT.cpp:
#include <iostream>
using namespace std;
extern "C"
{
void TQINIT_(int*, int*);
}
main()
{ int NWG;
NWG=80000;
int *IWSG = new int[NWG];
TQINIT_(IWSG,&NWG);
}
TQINIT.f:
SUBROUTINE TQINIT(IWSG,NWG)
IMPLICIT NONE
INTEGER NWG
INTEGER IWSG(NWG)
LOGICAL TQG2ERR
INTEGER IERR
CALL TQRSERR
CALL TQINI(NWG,IWSG)
IF (TQG2ERR(IERR)) THEN
WRITE(6,*)'!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!'
WRITE(6,*)'ERROR INITIALIZING (TQINIT) !!!'
WRITE(6,*)'!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!'
STOP
ENDIF
END
I compile the subroutine using:
gfortran -c TQINIT.f
and get a TQINIT.o
Then I compile the main program using:
g++ -c test-TQINIT.cpp
and get a test-TQINIT.o.
Then for create the .x I use (TQINIT.f need the library):
g++ test-TQINIT.o TQINIT.o -L/usr/local/thermocalc/3.0/SDK/TQ8 -ltq-linux-x86_64-gfortran44-8 -o test-TQINIT.x
This is what I get:
test-TQINIT.o: In function `main':
test-TQINIT.cpp:(.text+0x33): undefined reference to `TQINIT_'
collect2: ld returned 1 exit status
I hope someone will be able to help me.
Thank you by advance.
Fortran is case-insensitive, and the name of the subroutines in the object file normally (as far as I know) end up in lowercase with an underscore, so yours is probably named tqinit_ and not TQINIT_, so your call in C++ should be lowercase.
You can verify the name in the object file with the objdump -t TQINIT.o command given in Alexander Vogt's answer, or the simpler nm TQINIT.o which is more terse.
If you use the ISO_C_BINDING in Fortran, you will have control over the precise routine names that will be seen by the other language, e.g., case, no underscores. You can also declare the arguments so that consistency with C/C++ will be guaranteed. For more on this topic, see https://stackoverflow.com/questions/tagged/fortran-iso-c-binding
I'm no expert on mixing C and Fortran, but I have always linked the Fortran Code directly using
g++ test-TQINIT.cpp TQINIT.o -L/usr/local/thermocalc/3.0/SDK/TQ8 \
-ltq-linux-x86_64-gfortran44-8 -o test-TQINIT.x
instead of compiling those two files separately and then linking... Perhaps this helps (although your way should work as well, at least it does with the simple example I tried).
BTW:
You can find out how the functions in your Fortran object are called by issuing
objdump -t TQINIT.o
This should give you the correct function name to call.
I am working with two sets of FORTRAN code, for example, codeA.f and codeB.f. In each set of code there are subroutines with the same name, but which perform significantly different tasks (as well as common blocks).
codeA.f:
subroutine init
c do something here ...
end
codeB.f:
subroutine init
c do something else here ...
end
I create two C++ wrappers for both routines (wrapperB is identical, but swap B for A).
wrapperA.cpp:
namespace codeA {
extern "C"{void init_();}
void init() {init_();}
}
wrapperA.h:
namespace codeA {
init();
}
I then compile the two wrappers into shared libraries.
Makefile:
codeA.o : codeA.F
g77 -fno-automatic -fPIC -O3 -c codeA.f -o codeA.o
wrapperA.o : wrapperA.cpp
g++ -fPIC -c wrapperA.cpp -o wrapperA.o
libwrapperA.so : codeA.o wrapperA.o
g++ -fPIC -shared -lg2c codeA.o wrapperA.o -o libwrapperA.so
Finally, I want to be able to call the two different subroutines from the same program.
main.cpp:
#include "wrapperA.h"
#include "wrapperB.h"
main {
wrapperA::init();
wrapperB::init();
}
The main program is linked against the two shared wrapper libraries.
Makefile:
main : main.cpp
g++ main.cpp -I. -L. -lwrapperA -lwrapperB -o main
Everything compiles and there are no complaints. However, when the program "main" is run, the init() commands perform the same action, dictated by which of the wrapper libraries was linked against first. However, if I link against only one of the libraries, the linker complains about missing symbols.
I have two question about this:
a) Is it possible to link against two different FORTRAN routines using only namespaces or some similar method to what I am doing? If so, what am I doing wrong?
b) If a) is not possible, is it possible to resolve the conflicts by instructing the compiler to append the symbols with some unique identifier when creating the FORTRAN objects (I am using gcc version 4.1.2 20080704 (Red Hat 4.1.2-51))?
Thanks for any insight into this.
Your compile commands show the use of g77, which is no longer maintained and also implies the use of a very old version of Fortran (i.e., FORTRAN 77). You are also using an old version of gcc. If you use gcc and gfortran of version 4.3 or higher -- preferably 4.4 or higher -- then you can use the Fortran 2003 ISO C Binding to solve your problem. In the Fortran code you can assign externally visible names to the routines with the "bind" option of the ISO C Binding. Just assign different "bind" names to the two routines. Then they won't clash at the library & linker level:
In one Fortran file use
subroutine init (stuff...) bind (C, name="init_V1")
and the other file
subroutine init (stuff...) bind (C, name="init_V2")
and the routines will be visible to C, C++, the linker, etc. as init_V1 and init_V2 without clashing.
Have a look at this SO question, which is very similar to your problem : Static and shared library symbol conflicts?
Basically, you have two shared libraries defining different symbols : wrapperA::init() and wrapperB::init() ; but also conflicting symbols : _init__(). So when the linker resolves, it use the first of the conflicting symbol it find.
If you remove the unneeded external visibility of _init__() in your libraries by using the techniques given in the SO question : http://www.gnu.org/software/gnulib/manual/html_node/Exported-Symbols-of-Shared-Libraries.html, then I think you will have solved your problem. Each wrapper::init() will then call internally it's proper _init__() without any conflict.