Makefile for cuda/c++, Cannot successfully compile - c++

Question

I have spend quite some time trying to write a makefile to compile some c++ files along with some .cu files. I cannot successfully compile the target while I do get an output currently, it is not a proper binary file.
So, basically I have all my source files in a src directory, my makefile in the main project folder, one directory up from src. I have a main.cpp, a hostDeviceCom.cu, a myKernel.cu and a cudaErrorCheck.cu that I had tested in a previous project compiled by hand with nvcc in one step. The concept is to have a separate build folder where the .out will be.
So my question is: What am I doing so wrong in the following makefile?
TARGET_EXEC ?= cudaNestim.out
T_CUDA_O ?= cudaTMp.o
NVCC ?= nvcc
BUILD_DIR ?= ./build
SRC_DIRS ?= ./src
CUDA_ARCH ?= -arch=sm_52
NVCCFLAGS ?= $(CUDA_ARCH)
NVCCFLNK ?= $(CUDA_ARCH) --device-link
CXXFLAGS ?= --std=c++11 -MM -MT
CXXOPTS ?= -MM -MT
# System Libraries -------------------------------------------------------------------------------
OCV_DIR ?= -L/user/local/lib
OCV_LIB ?= -lopencv_core -lopencv_viz -lopencv_highgui -lopencv_features2d -lopencv_imgproc
OCV_INC ?= -I/usr/include
OCV_LINK ?= $(OCV_DIR) $(OCV_LIB)
CUDA_DIR ?= -L/user/local/cuda-8.0/lib64
CUDA_LIB ?= -lcutil -lcudpp -lcuda -lcudart
CUDA_INC ?= -I/usr/local/cuda-8.0/include
CUDA_LINK ?= $(CUDA_DIR) $(CUDA_LIB)
#--------------------------------------------------------------------------------------------------
# find all the source files
SRCS := $(shell find $(SRC_DIRS) -name *.cu -or -name *.cpp -or -name *.c -or -name *.s)
#go to build directory and create a .o file for each src file found
OBJS := $(SRCS:%=$(BUILD_DIR)/%.o)
# create a .d file from each .o file. SO one .d for each .source
DEPS := $(OBJS:.o=.d)
LIB_DIRS := $(OCV_LINK) $(CUDA_LINK)
INCS := $(CUDA_INC) $(OCV_INC)
$(BUILD_DIR)/$(TARGET_EXEC): $(OBJS)
$(NVCC) $(NVCCFLNK) $(OBJS) $(LIB_DIRS) -o $#
$(BUILD_DIR)/%.cu.o: %.cu
$(MKDIR_P) $(dir $#)
$(NVCC) $(NVCCFLAGS) -o $# -c $<
#echo ".cu.o rule:" $(NVCCFLAGS) -o $# -c $< $(OCV_INC) $(CUDA_INC)
#$(NVCC) $(NVCCFLAGS) -o $# -c $< $(OCV_INC) $(CUDA_INC)
$(BUILD_DIR)/%.cpp.o: %.cpp
$(MKDIR_P) $(dir $#)
$(CXX) $(CXXFLAGS) $(INCS) -o $# -c $<
# Phony rules -------------------------------------------------------------------------------
.PHONY: clean
clean:
$(RM) -r $(BUILD_DIR)
MKDIR_P ?= mkdir -p
-include $(DEPS)
I know, I am not using any .d rule, I am trying to make this work and figure out later how to incorporate a .d rule for more complex stuff.

Answer 1

it is not a proper binary file
The proximal reason for this is that you are specifying device-link as the final build (ie link phase) operation:
NVCCFLNK ?= $(CUDA_ARCH) --device-link
^^^^^^^^^^^
...
$(BUILD_DIR)/$(TARGET_EXEC): $(OBJS)
$(NVCC) $(NVCCFLNK) $(OBJS) $(LIB_DIRS) -o $#
^^^^^^^^
As a result, the last build step you do is a device link, which does not create an executable. Your Makefile is generally also not set up properly for device-linking, because when separating compile from link, if you want device linking (i.e. generation of relocatable device code) then you must specify -dc, not -c during compile:
$(BUILD_DIR)/%.cu.o: %.cu
$(MKDIR_P) $(dir $#)
$(NVCC) $(NVCCFLAGS) -o $# -c $<
^^
Since you have already specified that the link phase should be done at once, and with nvcc, we can make some relatively simple changes to your makefile to achieve a makefile that will do rdc linking. Here's a fully worked example, derived from yours:
$ cat Makefile
TARGET_EXEC ?= cudaNestim.out
NVCC ?= nvcc
BUILD_DIR ?= ./bld
SRC_DIRS ?= ./src
CUDA_ARCH ?= -arch=sm_61
NVCCFLAGS ?= $(CUDA_ARCH)
#NVCCFLNK ?= $(CUDA_ARCH) --device-link
NVCCFLNK ?= $(CUDA_ARCH)
CXXFLAGS ?= --std=c++11 -MM -MT
CXXOPTS ?= -MM -MT
# System Libraries -------------------------------------------------------------------------------
OCV_DIR ?= -L/user/local/lib
#OCV_LIB ?= -lopencv_core -lopencv_viz -lopencv_highgui -lopencv_features2d -lopencv_imgproc
OCV_LIB ?=
OCV_INC ?= -I/usr/include
OCV_LINK ?= $(OCV_DIR) $(OCV_LIB)
CUDA_DIR ?= -L/usr/local/cuda-8.0/lib64
#CUDA_LIB ?= -lcutil -lcudpp -lcuda -lcudart
CUDA_LIB ?= -lcuda -lcudart
CUDA_INC ?= -I/usr/local/cuda-8.0/include
CUDA_LINK ?= $(CUDA_DIR) $(CUDA_LIB)
#--------------------------------------------------------------------------------------------------
MKDIR_P ?= mkdir -p
# find all the source files
SRCS := $(shell find $(SRC_DIRS) -name *.cu -or -name *.cpp -or -name *.c -or -name *.s)
#go to build directory and create a .o file for each src file found
OBJS := $(SRCS:%=$(BUILD_DIR)/%.o)
# create a .d file from each .o file. SO one .d for each .source
DEPS := $(OBJS:.o=.d)
LIB_DIRS := $(OCV_LINK) $(CUDA_LINK)
INCS := $(CUDA_INC) $(OCV_INC)
$(BUILD_DIR)/$(TARGET_EXEC): $(OBJS)
$(NVCC) $(NVCCFLNK) $(OBJS) $(LIB_DIRS) -o $#
$(BUILD_DIR)/%.cu.o: %.cu
$(MKDIR_P) $(dir $#)
$(NVCC) $(NVCCFLAGS) -o $# -dc $<
#echo ".cu.o rule:" $(NVCCFLAGS) -o $# -c $< $(OCV_INC) $(CUDA_INC)
#$(NVCC) $(NVCCFLAGS) -o $# -c $< $(OCV_INC) $(CUDA_INC)
$(BUILD_DIR)/%.cpp.o: %.cpp
$(MKDIR_P) $(dir $#)
$(CXX) $(CXXFLAGS) $(INCS) -o $# -c $<
# Phony rules -------------------------------------------------------------------------------
.PHONY: clean
clean:
$(RM) -r $(BUILD_DIR)
$ ls
bld Makefile src
$ cat src/hello.cu
#include <stdio.h>
__device__ void hello(){
printf("hello!\n");
}
$ cat src/test.cu
#include <stdio.h>
__device__ void hello();
__global__ void my_hello(){
hello();
}
int main(){
my_hello<<<1,1>>>();
cudaDeviceSynchronize();
}
$ make clean
rm -f -r ./bld
$ make
mkdir -p bld/./src/
nvcc -arch=sm_61 -o bld/./src/hello.cu.o -dc src/hello.cu
mkdir -p bld/./src/
nvcc -arch=sm_61 -o bld/./src/test.cu.o -dc src/test.cu
nvcc -arch=sm_61 ./bld/./src/hello.cu.o ./bld/./src/test.cu.o -L/user/local/lib -L/usr/local/cuda-8.0/lib64 -lcuda -lcudart -o bld/cudaNestim.out
$ bld/cudaNestim.out
hello!
$
Note that if you actually didn't want RDC code generation, you could change the -dc back to -c in the Makefile above. However, for demonstration purposes, my particular code example depends on CUDA RDC.

Answer 2

More of a suggestion than a direct answer:
What you're doing wrong is trying to reinvent the wheel. Use a higher-level build system, don't bother writing the Makefile yourself. There's more than one of these around; I personally use CMake and am sort-of happy with it. CMake has a module for setting up things paths and other variables for CUDA. Here's a short tutorial you could check out:
How to build CUDA programs using CMake
Now, it's not that you'll never run into any issues, but:
There'll be less of them
They would likely be easier to debug (have look at some CMake+CUDA questions here on SO).