CMake : /usr/include not used to find fftw3.h - c++

Question

I'm working on a personal project to cross compile for a Raspberry Pi with CMake. This is the first time I build a project with it.
One of the file I compile include the fftw3.h header. This file is present in the /usr/include folder as shown by the following output :
$ ls /usr/include/fftw3*
/usr/include/fftw3.f
/usr/include/fftw3.f03
/usr/include/fftw3.h
/usr/include/fftw3l.f03
/usr/include/fftw3q.f03
The file is successfully found by cmake with find_path(FFTW_FOLDER NAMES fftw3.h) and even if cmake looks in this path by default, I add it explicitly when constructing my target with target_include_directories(${LIB_NAME} PUBLIC /usr/include).
However, when comes the compilation, I have the classic error message telling me that fftw3.h is not found.
If I copy the file and put it in my home folder and add my home folder in the include list there is no problem and the compilation works.
Any thoughts ?
Thanks,
Damien
Edit 1 to answer #Pablo and #hikerjobs:
This is how my compile line looks like:
/home/littlemonster/projects/rpi-workspace/rpi-tools/arm-bcm2708/gcc-
linaro-arm-linux-gnueabihf-raspbian-x64/bin/arm-linux-gnueabihf-g++
-Dmatrix_hal_EXPORTS -I/home/littlemonster/projects/rpi-
workspace/lib/matrix-hal/matrix-creator-hal -fPIC -Wall -Werror -
Wextra -Wdouble-promotion -std=gnu++11 -o CMakeFiles/matrix-
hal.dir/matrix-creator-hal/cpp/driver/cross_correlation.cpp.o -c
/home/littlemonster/projects/rpi-workspace/lib/matrix-hal/matrix-
creator-hal/cpp/driver/cross_correlation.cpp
In file included from /home/littlemonster/projects/rpi-
workspace/lib/matrix-hal/matrix-creator-
hal/cpp/driver/cross_correlation.cpp:18:0:
/home/littlemonster/projects/rpi-workspace/lib/matrix-hal/matrix-
creator-hal/cpp/driver/cross_correlation.h:21:19: fatal error:
fftw3.h: No such file or directory
#include <fftw3.h>
^
compilation terminated.
Because I am cross compiling and the library is present on the Raspberry Pi I don't link the library.
Edit 2 to answer #Bartłomiej :
The following lines are located in my rpi-toolchain.cmake file which is used when invoking cmake with the following : -DCMAKE_TOOLCHAIN_FILE=rpi-toolchain.cmake. When I try to specify the sysroot by setting CMAKE_SYSROOT the compiler is not even able to compile a test file.
# Define our host system
set(CMAKE_SYSTEM_NAME Linux)
set(CMAKE_SYSTEM_PROCESSOR arm)
SET(CMAKE_SYSTEM_VERSION 1)
SET(rpi-tools-dir ${CMAKE_SOURCE_DIR}/rpi-tools)
# Define the C cross compiler location
SET(CMAKE_C_COMPILER ${rpi-tools-dir}/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian-x64/bin/arm-linux-gnueabihf-gcc)
# Define the CXX cross compiler location
SET(CMAKE_CXX_COMPILER ${rpi-tools-dir}/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian-x64/bin/arm-linux-gnueabihf-g++)
# Define the sysroot path for the RaspberryPi distribution in our tools folder
SET(CMAKE_FIND_ROOT_PATH ${rpi-tools-dir}/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian-x64/arm-linux-gnueabihf/sysroot/)
# Only use binaries from the host and not from the toolchain sysroot
SET(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
# Search for libraries only in the target sysroot
SET(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
# Search for the headers in the target and host directories
SET(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE BOTH)