I am just starting out with libtorch and seem to have run into a rather serious linker issue. Previously, I was able to at least build other projects with torch (that others had created), but now that I've tried to create my own simple file, nothing seems to be building anymore.
I have manually installed the latest MacOS build in my root directory:
/Users/jlenz/libtorch/share/cmake/Torch
I am using CLion and have the following CmakeLists.txt:
cmake_minimum_required(VERSION 3.23)
project(midi_plugin)
set(CMAKE_CXX_STANDARD 17)
add_executable(midi_plugin main.cpp)
set(CMAKE_PREFIX_PATH "~/libtorch/share/cmake")
find_package(Torch REQUIRED)
target_link_libraries(midi_plugin "${TORCH_LIBRARIES}")
One thing that is noticeable, although the cmake 'succesfully' builds, it displays this in the process:
-- MKL_ARCH: intel64
-- MKL_ROOT /Users/jlenz/opt/miniconda3
-- MKL_LINK: dynamic
-- MKL_INTERFACE_FULL: intel_ilp64
-- MKL_THREADING: intel_thread
-- MKL_MPI: mpich
Once I try to build the project (which is just 'hello world' with torch included) I get the following error:
FAILED: midi_plugin
: && /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ -g -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.1.sdk -mmacosx-version-min=12.5 -Wl,-search_paths_first -Wl,-headerpad_max_install_names CMakeFiles/midi_plugin.dir/main.cpp.o -o midi_plugin -Wl,-rpath,/Users/jlenz/libtorch/lib /Users/jlenz/libtorch/lib/libc10.dylib /Users/jlenz/libtorch/lib/libkineto.a /Users/jlenz/libtorch/lib/libtorch.dylib /Users/jlenz/libtorch/lib/libtorch_cpu.dylib /Users/jlenz/libtorch/lib/libc10.dylib -lmkl_intel_ilp64 -lmkl_core -lmkl_intel_thread && :
ld: library not found for -lmkl_intel_ilp64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
Would anyone know how to solve this? I have tried addressing it with a colleague who has used libtorch quite a bit and he can't identify the issue either. Many thanks in advance.
Posting my solution here in case someone else runs into a similar issue.
It turns out the issue was with my brew installation of torch, which was being referenced implicitly. To fix it, all I had to do was:
Brew install pytorch
It asked permission to overwrite some files that had issues, which I granted. Afterwards, all of my Torch projects are working perfectly.
Related
I am trying to compile glfw from source on Mac with M1 arm64 processor, and while running the linker, cmake strangely is trying to link the project for x86_64 architecture, while the binaries were built for arm64.
I clone the project, create build folder named cmake-build-debug, generate build system in it with the Makefile etc. as follows:
git clone https://github.com/glfw/glfw.git
cd glfw
mkdir cmake-build-debug
cd cmake-build-debug
cmake -S .. -B . -DCMAKE_BUILD_TYPE=Debug -DCMAKE_HOST_SYSTEM_PROCESSOR=arm64 -DCMAKE_SYSTEM_PROCESSOR=arm64
This works fine. But now that I build it with make or cmake --config Debug --build ., the .o binaries are generated perfectly fine, but linker script is incorrectly invoked by cmake with x86_64 target architecture for some reason:
-- Could NOT find Doxygen (missing: DOXYGEN_EXECUTABLE)
-- Including Cocoa support
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/burkov/Documents/Projects/open-source/glfw/cmake-build-debug
[ 47%] Built target glfw
Scanning dependencies of target wave
[ 50%] Linking C executable wave.app/Contents/MacOS/wave
ld: warning: ignoring file CMakeFiles/wave.dir/wave.c.o, building for macOS-x86_64 but attempting to link with file built for unknown-arm64
ld: warning: ignoring file ../src/libglfw3.a, building for macOS-x86_64 but attempting to link with file built for macOS-arm64
Undefined symbols for architecture x86_64:
"_main", referenced from:
implicit entry/start for main executable
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [examples/wave.app/Contents/MacOS/wave] Error 1
make[1]: *** [examples/CMakeFiles/wave.dir/all] Error 2
make: *** [all] Error 2
I look at the failing Makefile in glfw/cmake-build-debug/examples/CMakeFiles/wave.dir/build.make and see the line, where cmake is crashing:
cd /Users/me/Documents/Projects/open-source/glfw/cmake-build-debug/examples && $(CMAKE_COMMAND) -E cmake_link_script CMakeFiles/wave.dir/link.txt --verbose=$(VERBOSE)
I manually open the file glfw/cmake-build-debug/examples/CMakeFiles/wave.dir/link.txt file and see the following link script code there:
/Library/Developer/CommandLineTools/usr/bin/cc -g -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.1.sdk -Wl,-search_paths_first -Wl,-headerpad_max_install_names CMakeFiles/wave.dir/wave.c.o -o wave.app/Contents/MacOS/wave ../src/libglfw3.a -framework Cocoa -framework IOKit -framework CoreFoundation
If I manually execute this line from shell, it successfully builds my binary for arm64 architecture, as expected.
But when this link.txt script is automatically invoked with cmake via cmake -E cmake_link_script CMakeFiles/wave.dir/link.txt --verbose=$(VERBOSE), it fails, apparently, trying to build the binary for the wrong x86_64 architecture.
Why is this happening and how to fix this?
For anyone running into the same problem, it looks like the first version of cmake with an adequate support for Apple Silicon is 3.19.
I was using 3.17.5 as my slightly out-of-date version of CLion does not support versions of cmake above that.
After an update to cmake 3.22.4 the problem is gone.
I'm using a library that I cannot compile for Apple M1, so I have decided to compile it and use it using (Rosetta 2) for x86_64 which I successfully did following this to install brew and clang for x86_64.
However when I compile my project and try to link it against this library I get this error:
ld: warning: ignoring file ..../libapronxx.a, building for macOS-arm64 but attempting to link with file built for macOS-x86_64
...
ld: symbol(s) not found for architecture arm64
I have tried to the set compiler and linker flags ("-arch x86_64") but still got the same problem.
My question is: What is the proper way to build for macOS-x86_64 using cmake on Apple M1 (arm)?
Additional information: I'm using cmake via CLion.
UPDATE:
I successfully compiled my project using the following commands:
# install a x86 cmake
arch -x86_64 /usr/local/bin/brew install cmake
...
# in the build directory
arch -x86_64 /usr/local/bin/cmake ..
make
arch -x86_64 ./my_exe
I also specified the architecture for clang using -arch flag
string(APPEND CMAKE_CXX_FLAGS_RELEASE " -arch x86_64 -O3")
string(APPEND CMAKE_CC_FLAGS_RELEASE " -arch x86_64 -O3")
# did the same for debug too
While this work fine, I still don't believe it is the proper way to use cmake to build for macOS-x86_64, in fact I cannot take the advantages of my IDE with all this manual approach.
After checking CMake source code, I found that it is enough to set CMAKE_OSX_ARCHITECTURES to x86_64:
set(CMAKE_OSX_ARCHITECTURES "x86_64")
This is the cleanest way so far to solve this issue, and work straight forward with CLion too.
My goal is to alias my cmake command so that it uses my homebrew installation of llvm exclusively, and not the suite installed by Xcode. I would like to leave the system defaults as defaults as well. I am on Catalina with Xcode and homebrew llvm up to date.
My first step is to use the homebrew clang++ and libc++, and I can do it manually
/usr/local/opt/llvm/bin/clang++ -Wall -Wextra -std=c++17 -stdlib=libc++ \
-nostdinc++ -I/usr/local/opt/llvm/include/c++/v1 \
-L/usr/local/opt/llvm/lib -Wl,-rpath,/usr/local/opt/llvm/lib \
main.cc
For completion's sake, here's my CMakeLists.txt; it's pretty barebones as most of the work I want to do doesn't belong in CMakeLists.txt and I'm just trying to compile a toy program.
cmake_minimum_required(VERSION 3.15)
project(test)
set(CMAKE_CXX_STANDARD 17)
add_executable(test
main.cc)
But I'm already stuck with cmake here. I can specify the compiler using
-DCMAKE_CXX_COMPILER="/usr/local/opt/llvm/bin/clang++ and similar for C/clang. I am using the CMAKE_PREFIX_PATH as well, but when I open my build.ninja files, I still see it referencing Xcode directories. Adding a CMAKE_IGNORE_PATH didn't seem to get it to actually ignore /Applications/Xcode.app.
I have also tried adding the flags CMAKE_CXX_STANDARD_LIBRARIES and CMAKE_CXX_STANDARD_INCLUDE_DIRECTORIES to my cmake command, but if I do this I start to get linker errors (still using the system linker and not homebrew's ld64.lld, I believe).
Here is my current working cmake command as a reference:
cmake -DCMAKE_PREFIX_PATH="/usr/local/opt/llvm" \
-DCMAKE_CXX_COMPILER="/usr/local/opt/llvm/bin/clang++" \
-DCMAKE_C_COMPILER="/usr/local/opt/llvm/bin/clang" -G Ninja ..
The specific compiler paths are still required because cmake just breezes straight to AppleClang.
I know that the path repetition is annoying, but I want a full command to work before I start modifying my .zshrc.
Here's the broken command (broken as in building the program no longer succeeds, cmake doesn't complain here):
cmake -DCMAKE_IGNORE_PATH="/Applications/Xcode.app" -DCMAKE_PREFIX_PATH="/usr/local/opt/llvm" \
-DCMAKE_CXX_COMPILER="/usr/local/opt/llvm/bin/clang++" \
-DCMAKE_C_COMPILER="/usr/local/opt/llvm/bin/clang" \
-DCMAKE_CXX_STANDARD_LIBRARIES="/usr/local/opt/llvm/lib" \
-DCMAKE_CXX_STANDARD_INCLUDE_DIRECTORIES="/usr/local/opt/llvm/include/c++/v1" \
-G Ninja ..
With the build error message:
[2/2] Linking CXX executable test
FAILED: test
: && /usr/local/opt/llvm/bin/clang++ -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk -Wl,-search_paths_first -Wl,-headerpad_max_install_names CMakeFiles/test.dir/main.cc.o -o test /usr/local/opt/llvm/lib && :
ld: can't map file, errno=22 file '/usr/local/opt/llvm/lib' for architecture x86_64
clang-9: error: linker command failed with exit code 1 (use -v to see invocation)
ninja: build stopped: subcommand failed.
The variable CMAKE_CXX_LINK_EXECUTABLE just gets ignored, so it's not included.
If anyone has any guidance, I would appreciate it. All links I have found about this usually override the system defaults, and I would like to avoid doing that if possible.
I've reproduced this symptom on two computers now, cmake seems to no longer look in /usr/local/lib (or more properly, $(brew --prefix)/lib) for Homebrew-provided libraries since upgrading my machine to macOS Mojave.
Although there are ways to circumvent this (e.g. search for homebrew prefix using EXECUTE_PROCESS; add the result to LINK_LIBRARIES(...) command) none are ideal. What changed in Mojave to break this behavior?
The temporary workaround is to add the following to CMakeLists.txt:
# WARNING: Don't hard-code this path
LINK_DIRECTORIES(/usr/local/lib)
I've already tried brew doctor and updated all homebrew packages to no avail.
The specific error that cmake (make) shows is:
ld: library not found for -l<somelib>
I've asked the question on the Homebrew forums and the Apple developer forums.
Ran into a related (?) issue while trying to pip install psycopg2 in a Django app under OS X Mojave (10.14). I was getting the following errors:
ld: library not found for -lssl
clang: error: linker command failed with exit code 1 (use -v to see invocation)
error: command 'clang' failed with exit status 1
The short explanation: « As of High Sierra, /usr/local is no longer
chown-able... »
The solution: change permissions for /usr/local to allow Homebrew to
create links.
I adapted the solution to my needs. Then I was finally able to run pip install psycopg2. Here is the sequence of commands (update: inside your project root i.e. where you see manage.py).
First
sudo chown -R $(whoami) $(brew --prefix)/*
Then
brew reinstall openssl
export LDFLAGS="-L/usr/local/opt/openssl/lib"
export CPPFLAGS="-I/usr/local/opt/openssl/include"
pip install psycopg2
I've isolated this to the following change in the VERBOSE=1 make logs...
High Sierra (<=10.13) and below did NOT use the -isysroot command.
Mojave (>=10.14) DOES use the -isysroot command.
From gnu.org:
-isysroot <dir>
This option is like the --sysroot option, but applies only to header files (except for Darwin targets, where it applies to both header files and libraries). See the --sysroot option for more information.
So this flag specifically clobbers the lib search path only on Apple. This results in the compilation never looking in the standard ld locations, which can be seen by typing ld -v dummy.
Library search paths:
/usr/lib
/usr/local/lib
Why does cmake do this? My thought is it was to fix the /usr/local/include issues introduced with the new Mojave SDK behavior.
Unfortunately, I can't find a cmake compile flag to add the default library search paths back in. For now the only solution I've found is to add the following to my project:
IF(APPLE)
# Fix linking on 10.14+. See https://stackoverflow.com/questions/54068035
LINK_DIRECTORIES(/usr/local/lib)
ENDIF()
I'm not sure if this is a behavior that warrants an upstream cmake patch. If there is a better solution, please provide it.
I am installing the Moses Translation Software on my Mac OS X 10.9.5 with Xcode 6.1. The instructions say that I need g++ and Boost installed. Once I do that, I git clone, "cd" into the directory, and then type ./bjam -j8. First, I verified I have the prerequisites. First, g++ (I just clicked the TAB to see what was available):
$ g++
g++ g++-4.9
Then boost:
$ brew install boost
Warning: boost-1.56.0 already installed
Then I tried installing:
$ ./bjam -j8
Tip: install tcmalloc for faster threading. See BUILD-INSTRUCTIONS.txt for more information.
mkdir: bin: File exists
...patience...
...patience...
...found 4469 targets...
...updating 155 targets...
darwin.link lm/bin/darwin-4.2.1/release/debug-symbols-on/link-static/threading-multi/query
ld: library not found for -lboost_thread
clang: error: linker command failed with exit code 1 (use -v to see invocation)
// Additional error messages...
...failed darwin.link mert/bin/darwin-4.2.1/release/debug-symbols-on/link-static/threading-multi/util_test...
...skipped <pmert/bin/darwin-4.2.1/release/debug-symbols-on/link-static/threading-multi>util_test.passed for lack of <pmert/bin/darwin-4.2.1/release/debug-symbols-on/link-static/threading-multi>util_test...
darwin.link mert/bin/darwin-4.2.1/release/debug-symbols-on/link-static/threading-multi/vocabulary_test
ld: library not found for -lboost_thread
clang: error: linker command failed with exit code 1 (use -v to see invocation)
"g++" -o "mert/bin/darwin-4.2.1/release/debug-symbols-on/link-static/threading-multi/vocabulary_test" "mert/bin/darwin-4.2.1/release/debug-symbols-on/link-static/threading-multi/VocabularyTest.o" "mert/bin/darwin-4.2.1/release/debug-symbols-on/link-static/threading-multi/libmert_lib.a" -lboost_unit_test_framework -llzma -lbz2 -ldl -lboost_system -lz -lboost_thread -lm -liconv -g -Wl,-dead_strip -no_dead_strip_inits_and_terms
...failed darwin.link mert/bin/darwin-4.2.1/release/debug-symbols-on/link-static/threading-multi/vocabulary_test...
...skipped <pmert/bin/darwin-4.2.1/release/debug-symbols-on/link-static/threading-multi>vocabulary_test.passed for lack of <pmert/bin/darwin-4.2.1/release/debug-symbols-on/link-static/threading-multi>vocabulary_test...
...failed updating 72 targets...
...skipped 83 targets...
The build failed. If you need support, run:
./jam-files/bjam -j8 --debug-configuration -d2 |gzip >build.log.gz
then attach build.log.gz to your e-mail.
You MUST do 3 things before sending to the mailing list:
1. Subscribe to the mailing list at http://mailman.mit.edu/mailman/listinfo/moses-support
2. Attach build.log.gz to your e-mail
3. Say what is the EXACT command you executed when you got the error
ERROR
There's a ton of error messages not shown (condensed into "// Additional error messages"), and they are all of the form "ld: library not found for -lboost_thread". So it's clear that something can't find a boost-related library, but I don't know how to fix this. Does anyone have suggestions? I literally just copied a couple of lines from the installation instructions, and Moses is popular enough such that an obvious error in the installation instructions would have been caught long ago.
Additional comment: On the installation instructions page, they list a command where they can force an installer to find the boost library:
./bjam --with-boost=~/workspace/temp/boost_1_55_0 -j8
My boost is in
/usr/local/Cellar/boost/1.56.0/
I tried substituting the --with-boost= argument with the above file path, but that did not work either (I got the same errors).
I'm going to post this on the Moses mailing list but I'd also want to ask here because I have gotten this same error ("library not found ... clang: error: linker command failed with exit code 1") with other software and it would be helpful for me to learn a general strategy for making sure that clang can find my libraries.
This package expects a non-standard layout of boost (or maybe that's the standard liayout for installed boost on many systems, but it certainly isn't the default layout for boost in my working directories):
bjam --help says:
--with-boost=/path/to/boost
If Boost is in a non-standard location, specify it here. This directory is
expected to contain include and lib or lib64.
When my Boost build tree is in
/home/sehe/custom/boost/boost // headeers
/home/sehe/custom/boost/stage
/home/sehe/custom/boost/stage/lib // libraries
I have created a "forwarding" directory - so I'm not required to install boost:
mkdir /tmp/boost-moses
cd /tmp/boost-moses/
ln -sfv /home/sehe/custom/boost/stage/lib lib
ln -sfv /home/sehe/custom/boost include
Now I could trigger a build in the mosesdecoder directory with
./bjam --with-boost=/tmp/boost-moses
It's not said here which Moses version ComputerScientist compiled, but his last comment hit the nail. When I was about to compile version 2.1.1, I needed to rewrite Jamroot file and jam-files/boost-build/tools/mpi.jam and change all the boost_mpi mentions to boost_mpi-mt. It seems to me that boost-1.55 used different library style then boost-1.56 and Moses was in that version adapted to 1.55.
However I am not able to compile Moses 3.0 with either boost-1.55 or boost-1.56 now because of undefined reference to boost::filesystem::path::stem().
Unfortunatelly I can not post this as a comment to previous answer...