I am compiling a plugin for Nuke8 under linux. All compiling is done without issue but i have the following error when i try to load the plug :
undefined symbol: _ZN9Imath_2_16Rand325nextfEv
When i do "ldd" onto the plugin.so, i have this:
linux-vdso.so.1 => (0x00007fff44869000)
libDDImage.so => not found
libfftw3f.so.3 => /usr/lib64/libfftw3f.so.3 (0x00007f4609bf5000)
libImath.so.6 => /usr/lib64/libImath.so.6 (0x00007f46099f0000)
libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007f46096ea000)
libm.so.6 => /lib64/libm.so.6 (0x00007f4609465000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f460924f000)
libc.so.6 => /lib64/libc.so.6 (0x00007f4608ebb000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f4608c9d000)
libIex.so.6 => /usr/lib64/libIex.so.6 (0x00007f4608a7f000)
/lib64/ld-linux-x86-64.so.2 (0x000000300bc00000
All lib seem to be load ok. I have a "libDDImage.so => not found", but this ok i have the same thing when i do this on exemple plugin.
I think the problem come from the Imath lib, but i don't know how to fix it.
Anybody have an idea? Thanks in advance.
Best
UPDATE
Since this answer was originally posted, The Foundry have made their modified OpenEXR source code available for download, including the custom namespace and some interface extensions. This should make it easy to write and successfully build custom plugins that link against the distributed OpenEXR libraries.
Links to compiled bonaries and source files can be found at: https://www.thefoundry.co.uk/products/nuke/developers/
They also have an open pull request for these changes to be merged into the main OpenEXR project, which can be found here: https://github.com/openexr/openexr/pull/141
Original Answer
Unfortunately this type of issue is hard to nail down without knowing everything about your build and runtime environment, but here's some information and ideas that will hopefully help put you on the right track.
To summarize, I think it's likely one of four things:
Symbol namespace issue
Binary compatibility issue (due to the library version mismatch)
Library load issue
Compiler version issue
Symbol Namespaces
Nuke 8 ships with its own EXR 2 libraries (specifically, version 2.0.1), which you can find in the installation directory. If you look at the exported symbols (using nm -D), you can see that not only are the symbols in a custom namespace, but the library version is different than the one you are linking against.
$ nm -D "/usr/local/Nuke8.0v5/libImath-2_0_1_Foundry.so.10" | grep Rand | c++filt
0000000000012590 T Imath_2_0_1_Foundry::Rand32::nextf()
As you can see, the EXR 2 symbols in Nuke are in the namespace Imath_2_0_1_Foundry, while your library is looking for the Imath_2_1 namespace. This would seem to indicate a library loading issue (either due to it not being found, or to Nuke not being able to load it).
Library Loading
Something that's always important to keep in mind is that the libraries that are resolved by ldd aren't necessarily going to be the same ones Nuke finds. The easiest way to examine what actually happens in Nuke is to run it through strace using something like the this:
$ strace -fqo /var/tmp/nuke_strace_output.txt Nuke
Note that you may need to use the full path to the Nuke binary, depending on your shell environment. You should try and start Nuke with no other custom code running (other than whatever is required to get your plugin onto the plugin path), and without opening any Nuke scripts, in order to prevent anything else from loading the Imath library.
Once you have your empty Nuke session running, just attempt to create an instance of your node and then quit Nuke. Now you can grep through nuke_strace_output.txt and find the point where your plugin is loaded, which should look something like this:
open("/path/to/MyPlugin.so", O_RDONLY|O_CLOEXEC) = 50
After that, if you scroll through the strace output, you'll see what steps Nuke takes when it tries to load the libraries your plugin depends on that have not yet been loaded (which names it tries, where it looks, etc.), which should include libImath (and I'm guessing libfftw3f).
Binary Compatibility
If possible, I would recommend trying to use the same version of OpenEXR that Nuke ships with, so you can just piggy-back on its libraries. You'll need to get your own headers in order to compile your plugin, but that's trivial for something like Imath.
As far as compilers go, you should be using GCC 4.1.2. If you don't, you're very likely going to run into binary compatibility issues at some point.
Anyway, I know this is jumping around into a lot of different areas, but I hope it helps some.
Related
I am working on a C++ project consisting of multiple similar programs. Each use a variety of headers and binaries, around 80. I managed to compile each program, and successfully run each but one of them, one named VerifyServer. When I try to run this, I get the below error:
./VerifyServer: error while loading shared libraries: libboost_system.so.1.57.0: cannot open shared object file: No such file or directory
However, here's the weird thing:
I was getting a very similar error when I tried running the other programs as well, but then I added
-Wl,-rpath=$(BOOST_LIB_HEADER_PATH)/stage/lib
to each of my compiler arguments, and it worked. I managed to run each programm, and even the file indicated in the error message of VerifyServer changed. Note that the variable BOOST_LIB_HEADER_PATH is correctly assigned to the path of my boost libraries.
Since my run time linker was failing to find this binary to correctly link it with my program, I tried checking which dependencies it was unable to find, so when I ran
ldd VerifyServer | grep boost (I use grep for simplicity)
I get this message:
libboost_thread.so.1.57.0 => /path/to/boost_1_57_0/stage/lib/libboost_thread.so.1.57.0 (0x00007f34c9b36000)
libboost_serialization.so.1.57.0 => /path/to/boost_1_57_0/stage/lib/libboost_serialization.so.1.57.0 (0x00007f34c9ad1000)
libboost_system.so.1.57.0 => not found
So, by now I assume that the system binary is not in the same directory as the others. When I manually check, I see that oddly enough, it is right there! So I decide to do the same thing on one of the programs that run without any problems, and I use
ldd IdentityProviderServer | grep boost
and I get the following message:
libboost_thread.so.1.57.0 => /path/to/boost_1_57_0/stage/lib/libboost_thread.so.1.57.0 (0x00007f97b6094000)
libboost_serialization.so.1.57.0 => /path/to/boost_1_57_0/stage/lib/libboost_serialization.so.1.57.0 (0x00007f97b602f000)
libboost_system.so.1.57.0 => /path/to/boost_1_57_0/stage/lib/libboost_system.so.1.57.0 (0x00007f97b602a000)
libboost_filesystem.so.1.57.0 => /path/to/boost_1_57_0/stage/lib/libboost_filesystem.so.1.57.0 (0x00007f97b600d000)
When I see that the system binary can be found in the same location as others, I just get confused. I don't know what the issue is here. Is there something I am missing about how the linker works?
Please note that I have to use the version 1.57.0 of boost here. I cannot update it to the current version, and I doubt my issue has anything to do with the version of boost.
This may be due to the linker unable to find transient dependencies. If you are directly linking against EG: libboost_thread, then the applications runtime linker will use the -rpath path that you defined when compiling.
HOWEVER, the libboost_thread may require ITS OWN library, libboost_system. The question is, should the libboost_thread library's runtime linker use your applications -rpath? or ignore it and use its own system search paths?? You must explicitly tell the runtime linker what it should do. Note, the default runtime linker search behavior changes depending on your compiler version. Google, RUNPATH vs RPATH and -Wl,--disable-new-dtags
I have answered this issue in another similar question: CMake project fails to find shared library
Although that user is using CMake to build their application, the underlying problem may be occurring here too.
Can someone shed some light on what is the best practice for loading plugins into a C++ Linux program?
Say we have a program (editor) with a plugin (libsyntax.so). The config file for editor contains the path to the libsyntax.so library (plugin1=/opt/editor/gizmos/libsyntax.so). The editor reads the config then calls:
void* library = dlopen(path, RTLD_NOW|RTLD_GLOBAL);
MYFUN* function = (MYFUN*)dlsym(library, "function");
All is fine, stuff works.
Now let us assume that (libsyntax.so) depends on a helper library (libcolor.so). When we run readelf we get:
readelf -d libsyntax.so
Dynamic section at offset 0x12b6b48 contains 31 entries:
Tag Type Name/Value
0x0000000000000001 (NEEDED) Shared library: [libcolor.so]
...
However at this point the above dlopen() fails and the error is "No such file or directory". Using LD_DEBUG=all reveals that after libsyntax.so is loaded, messages are:
28664: file=libcolor.so [0]; needed by /home/.../libsyntax.so [0]
28664: find library=libcolor.so [0]; searching
28664: search cache=/etc/ld.so.cache
28664: search path=/lib64/tls/x86_64:/lib64/tls:...:/usr/lib64 (system search path)
28664: trying file=/lib64/tls/x86_64/libcolor.so
... and so on
The loader/linker is looking in standard places and, obviously, not finding my dependency. This can be easily taken care of by ldconfig or LD_LIBRARY_PATH but both solutions feel dirty.
Is there a clean way to load both the plugin and dependencies? How are you doing this?
A clean way to ensure dependencies are found, is to set the run-time search path (RPATH) during linking of the program and its plugins to a sensible location, where the dependencies can be found.
If the rpath was set for a binary (which you can see with readelf), the linker will use the additional paths listed there besides the default system locations, and LD_LIBRARY_PATH.
Additionally, there's a special variable, $ORIGIN, which is always resolved to the location of the current binary during runtime, and you can set paths relative to that!
For example, if your application's root path (that contains the main executable) is /opt/editor, the plugins are under /opt/editor/gizmos, and let's say you have another folder, /opt/editor/lib with additional dependencies, you might use the linker options:
# editor executable
-Wl,-rpath=$ORIGIN/lib
# plugins
-Wl,-rpath=$ORIGIN/../lib,-rpath=$ORIGIN
Persinally, I think that LD_LIBARY_PATH is your friend here. Just define, as the definition for the plugin interface, where libraries the plugin needs should be located, and make sure, in your program, to set the LD_LIBARY_PATH to that location.
Simply setting it before calling dlopen should be enough for the library to load. No other change is needed in the program, and no special linking of the plugin itself.
I know ldd can only take a binary as its parameter what I am asking here is how to run ldd with a binary, say, mybin, with parameter of the binary. For instance, mybin --myparam ./configfile.conf.
The linker will differ if I add the conf file for my binary because it loads some plugins in runtime with the plugins' shared object files, plugin1.so something like this. I have some undefined reference issue but I still got no idea which .so file I was missing.
If I run ldd ./mybin, everything is linked and running the plain binary is fine. Once I add the conf file for my binary, to let it loads some plugins shared library, then my binary will report errors when loading those library(coded exception throw, with some undefined reference error messages).
So if there is way to run ldd with mybin --myparam ./a.file something like this would help a lot.
Use the LD_DEBUG environment variable for this. To see the relevant options, run any command with LD_DEBUG=help. For example, running LD_DEBUG=help ls on my machine gives this output:
LD_DEBUG=help ls
Valid options for the LD_DEBUG environment variable are:
libs display library search paths
reloc display relocation processing
files display progress for input file
symbols display symbol table processing
bindings display information about symbol binding
versions display version dependencies
scopes display scope information
all all previous options combined
statistics display relocation statistics
unused determined unused DSOs
help display this help message and exit
To direct the debugging output into a file instead of standard output
a filename can be specified using the LD_DEBUG_OUTPUT environment variable.
One way of debugging your dlopens, or whatever late loading mechanism you are using, would be to run your executables with the relevant args with LD_DEBUG=all. This would give you a lengthy output detailing symbol lookups and search paths. This output would also tell you about resolution failures.
While running the application developed by other person, getting the following error
./appln: error while loading shared libraries: libxerces-c.so.28: cannot open shared object file: No such file or directory
And if I run ldd command:
# ldd appln
linux-gate.so.1 => (0x00e20000)
libdl.so.2 => /lib/libdl.so.2 (0x00a61000)
libxerces-c.so.28 => not found
I already have that libxerces-c.so.28 file in current folder. Please help me how to resolve that error
You need to put libxerces-c.so somewhere in the library path. Probably current folder is not searched for libraries. Try putting it in /usr/local/lib
Evidently "the current folder" isn't in the run time search path used by your executable. I'm assuming that you are on linux (linux-gate.so.1).
You need to ensure that "the current" directory is under the search path. You can do this at link time by using the -rpath option to the linker (-R is also accepted) or -Wl,-rpath,<dir> if you are invoking the linker through the compiler front end. This embeds a search path to be used at run time into your program.
Consider how you want the program to be installed, though. There's no point in adding a path that is specific to your development environment. You might want to consider using $ORIGIN or a $ORIGIN relative path which tells the run time linker to look for shared objects in the location containing (or relative to) the executable. You should always avoid adding . to the run time search path; your program shouldn't behave differently depending on the current directory of the process which invokes it.
As a temporary measure you can set the environment variable LD_LIBRARY_PATH to override the embedded and system search paths but it is normally a bad idea to rely on LD_LIBRARY_PATH overrides for a final installation.
By default .so files are NOT being searched in the current folder (they should be in /usr/lib, etc).
To add the current directory for .so lookup use:
LD_LIBRARY_PATH=`pwd`:$LD_LIBRARY_PATH ./appln
When adding new "local system" libraries (e.g. in /usr/local/lib/) you better add that directory (/usr/local/lib/) once in your /etc/ld.so.conf and you should run ldconfig to update the linker cache (every time you add something inside /usr/local/lib/)
See ldconfig(8), ld.so(8), ldd(1), dlopen(3)
If you want your own libraries, set LD_LIBRARY_PATH to a directory containing them (e.g. $HOME/lib/ and to standard directories, e.g.
export LD_LIBRARY_PATH=$HOME/lib:/usr/lib:/usr/local/lib
in your ~/.bashrc (but I dislike that confusing practice, and prefer to manage my /usr/local/lib/).
You could also use some -Wl,-rpath argument but I really dislike that also.
Read also Program Library HowTo and Drepper's paper: How To Write Shared Libraries
I am having trouble with very large object files being produced. We are working with VxWorks 5.5.1, but we have a GCC 4.1.2 available.
Our modules are roughly 6.2MB in size, and we are looking for ways to reduce that. The problem seems to be mainly caused by excessive use of templates. When dumping the symbols in the file using nm I get a text-file of 1.8MB. This tells me that almost ⅓ of the file is just the names. Is there any way to reduce the file size?
The following approaches have not worked:
--strip-all seems to have no effect - the output is the same as using --strip-debug
I cannot use --gc-sections, because it is not supported for that platform (the option is simply ignored)
I understand that VxWorks links the code at load time, but all it has to link is the C++ runtime library, and I don't want any symbols to be added to the global symbol table, so there should be a way to strip that information, right?
For reference, here is my linker version:
i386-wrs-vxworks-ld.exe --version
>>> GNU ld (Wind River VxWorks G++ DWARF-EH 4.1-131) 2.17.50.20070509
>>> SPR fixes: cq103489 cq111170 cq116027 cq116652 cq118878 cq125145
and my compiler version:
i386-wrs-vxworks-g++.exe --version
>>> i386-wrs-vxworks-g++.exe (GCC) 4.1.2
I see what you mean by wanting to strip the symbols out of the object. But if you were to strip all the symbols, you wouldn't have any symbol to use as the entry point to start your application. But I believe you still have options. Unfortunately VxWorks 5 is known for having a not very effecient C++ compiler.
If you compiled your application into a *.a (archive - aka static library), you would be able to link this into your operating system at build time, and call this from within usrAppInit.c. This should allow for striping out symbols - or at least moving them to an optional downloaded symbol table. From your application build properties, select the Macros tab and add your archive to the LIBS macro.
To make an archive, goto the build properties and select the Rules tab, then select archive from the dropdown box.
To trim the size of your OS (with your application linked in), modify your vxworks settings. Disable as many components as you can. Also be sure to use a downloaded symbol table (development tool components -> symbol table components -> symbol table initialization componts -> selecte symbol table initialization -> downloaded symbol table).
This will strip all the symbols out of the OS, and make a downloadable symbol table, that can be downloaded after boot time to debug.
Good luck!
PS. Make sure you have turned off debug (-g) in your compiler. Maybe we could help more if you post your compiler switches.
The way we have normally handled this is to compress the image. You will also need to build the bootrom so that it will decompress to RAM before running. I believe that there are standard makefile options that will allow you to do this.