Windows says 64-bit executable is "Unsupported 16-Bit Application" - c++

Question

I have a C++ program that links Google's WebRTC library that compiles and runs successfully when I target 32-bit but doesn't work at all when I target 64-bit. After some trial and error I created the following program:
#include <media/base/adapted_video_track_source.h>
class AdaptedVideoTrackSource : rtc::AdaptedVideoTrackSource
{
public:
void AddRef() const {}
rtc::RefCountReleaseStatus Release() const { return rtc::RefCountReleaseStatus::kDroppedLastRef; }
bool is_screencast() const {return false;}
absl::optional<bool> needs_denoising() const {return false;}
bool GetStats(webrtc::VideoTrackSourceInterface::Stats* stats) {return false;}
webrtc::MediaSourceInterface::SourceState state() const {return webrtc::MediaSourceInterface::kLive;}
bool remote() const {return false;}
};
int main() {
AdaptedVideoTrackSource source;
}
This program fails with this error:
If I remove the super class the program runs fine.
How exactly do I debug a problem like this? I'm at a loss since I can't exactly debug the program. dumpbin appears to think my webrtc library is fine but says warning LNK4048: Invalid format file; ignored for my executable.
There's a lot of steps in the build process and I don't think I can put all of them here. I use CMake with ExternalProject_Add to download and build webrtc. I generate Ninja makefiles to build my code. Here are the ninja rules used to link the exe.
rule CXX_EXECUTABLE_LINKER__Test
command = cmd.exe /C "$PRE_LINK && "C:\Program Files\CMake\bin\cmake.exe" -E vs_link_exe --intdir=$OBJECT_DIR --rc=C:\PROGRA~2\WI3CF2~1\10\bin\100183~1.0\x64\rc.exe --mt=C:\PROGRA~2\WI3CF2~1\10\bin\100183~1.0\x64\mt.exe --manifests $MANIFESTS -- C:\PROGRA~2\MIB055~1\2017\COMMUN~1\VC\Tools\MSVC\1416~1.270\bin\Hostx64\x64\link.exe /nologo $in /out:$TARGET_FILE /implib:$TARGET_IMPLIB /pdb:$TARGET_PDB /version:0.0 $LINK_FLAGS $LINK_PATH $LINK_LIBRARIES && $POST_BUILD"
description = Linking CXX executable $TARGET_FILE
restat = $RESTAT
build utest\Test.exe: CXX_EXECUTABLE_LINKER__Test utest\CMakeFiles\Test.dir\main.cpp.obj | <OTHER LIBARIES> || <OTHER LIBARIES>
FLAGS = /DWIN32 /D_WINDOWS /GR /EHsc /bigobj /Zi /Ob0 /Od /RTC1 -MTd
LINK_FLAGS = /machine:x64 /debug /INCREMENTAL /subsystem:console
LINK_LIBRARIES = <OTHER LIBARIES> webrtc_bundle.lib <OTHER LIBRARIES>
LINK_PATH = -LIBPATH:D:\Folder\install64\lib
OBJECT_DIR = utest\CMakeFiles\Test.dir
POST_BUILD = cd .
PRE_LINK = cd .
TARGET_COMPILE_PDB = utest\CMakeFiles\Test.dir\
TARGET_FILE = utest\Test.exe
TARGET_IMPLIB = utest\Test.lib
TARGET_PDB = utest\Test.pdb
It's private code so I renamed the executable I build and replaced the non webrtc libraries with <OTHER LIBARIES>.
This is the args.gn I use to build webrtc:
target_cpu="x64"
rtc_enable_protobuf=true
is_official_build=false
rtc_build_examples=false
rtc_include_tests=false
enable_iterator_debugging=true
is_clang=false
Also, I wrote my own BUILD.gn file to bundle webrtc with other libraries that Google's build system can build.
UPDATE
I found that I can manually link my obj files and make a perfectly good exe. Then I started going into the Ninja Makefiles generated by CMake and playing around with the linker rules. I found that if I remove /debug from the linker flags then everything works great. Of course, I want to be able to debug my debug builds.

Answer 1

I moved on and tried to start building my project with clang-cl and lld-link which provided a little more diagnostic output. I started getting warnings about linking different versions of the run time library. MSVC_RUNTIME_LIBRARY was correctly set and I had the /MTd flag in CMAKE_CXX_FLAGS and CMAKE_LINKER_FLAGS but by running ninja with the verbose setting I could see that my flags were being followed by /MDd which overwrote the previous runtime setting. Eventually I appended /MTd to CMAKE_CXX_FLAGS_DEBUG and CMAKE_CXX_FLAGS_RELEASE and now it's the last runtime flag in the compiler and link commands and I haven't had this problem since.