Cross Compile OpenCV C++ application ARM - c++

Question

I have read all of the related questions with no success trying anything mentioned anywhere. I am new to cross-compiling and have been working on this for over a week with no progress. So please forgive me if you think I am stupid or have overlooked something.
So I have an application running in C++ that works great on my development computer running Ubuntu 14.04 x64. I am trying to cross compile for my Banana Pro running Lubuntu. Based on the documentation from Lemaker I am supposed to cross compile using "arm-linux-gnueabihf-"
So far the farthest I have been able to get is to :
/usr/local/opencv-arm/usr/local/lib/libopencv_calib3d.so: file not recognized: File format not recognized
collect2: error: ld returned 1 exit status
I get this error regardless of what command I run, Here is a list of commands I have tried:
arm-linux-gnueabihf-g++ `arm-linux-gnueabihf-pkg-config arm-opencv --cflags` -Wall test.cpp -o vis-300 `arm-linux-gnueabihf-pkg-config arm-opencv --libs`
arm-linux-gnueabihf-g++ `pkg-config arm-opencv --cflags` -Wall test.cpp -o vis-300 `pkg-config arm-opencv --libs`
arm-linux-gnueabihf-gcc `pkg-config arm-opencv --cflags` -Wall test.cpp -o vis-300 `pkg-config arm-opencv --libs`
arm-linux-gnueabihf-g++ `pkg-config arm-opencv --cflags` test.cpp -o vis-300 `pkg-config arm-opencv --libs`
And there have been many more commands before those with different errors such as:
arm-linux-gnueabihf-gcc: error trying to exec 'cc1plus': execvp: No such file or directory
arm-linux-gnueabihf-cpp fatal error too many input files
I have tried with just normal arm-linux-gnueabihf-gcc/g++, 4.6, 4.7, and 4.8
I have built opencv making small changes for hf using these 2 guides and both produced the same results:
http://processors.wiki.ti.com/index.php/Building_OpenCV_for_ARM_Cortex-A8
http://www.ridgesolutions.ie/index.php/2013/05/24/building-cross-compiling-opencv-for-linux-arm/
and not included in either I install it using this command because it will conflict with my current x86_64 install:
sudo make install DESTDIR=/usr/local/opencv-arm
Also the above pkg-config lines point to my custom pkg config file named arm-opencv.pc
# Package Information for pkg-config
prefix=/usr/local/opencv-arm/usr/local
exec_prefix=${prefix}
libdir=
includedir_old=${prefix}/include/opencv
includedir_new=${prefix}/include
Name: OpenCV-arm
Description: Open Source Computer Vision Library
Version: 2.4.10
Libs: ${exec_prefix}/lib/libopencv_calib3d.so ${exec_prefix}/lib/libopencv_contrib.so ${exec_prefix}/lib/libopencv_core.so ${exec_prefix}/lib/libopencv_features2d.so ${exec_prefix}/lib/libopencv_flann.so ${exec_prefix}/lib/libopencv_gpu.so ${exec_prefix}/lib/libopencv_highgui.so ${exec_prefix}/lib/libopencv_imgproc.so ${exec_prefix}/lib/libopencv_legacy.so ${exec_prefix}/lib/libopencv_ml.so ${exec_prefix}/lib/libopencv_nonfree.so ${exec_prefix}/lib/libopencv_objdetect.so ${exec_prefix}/lib/libopencv_ocl.so ${exec_prefix}/lib/libopencv_photo.so ${exec_prefix}/lib/libopencv_stitching.so ${exec_prefix}/lib/libopencv_superres.so ${exec_prefix}/lib/libopencv_ts.a ${exec_prefix}/lib/libopencv_video.so ${exec_prefix}/lib/libopencv_videostab.so -lrt -lpthread -lm -ldl
Cflags: -I${includedir_old} -I${includedir_new}
Anyways I have tried a lot of stuff short of just installing everything on the board itself and compiling there. Any help is much appreciated and keep in mind I have never successfully cross-compiled before. I always give up and compile on the board.