OpenCV read video file give different result in different VS project - c++

Question

I have a simple script to read mp4 file like this.
#include <opencv2/core/core.hpp>
#include <opencv2/highgui/highgui.hpp>
#include <opencv2/videoio/videoio.hpp>
int main()
{
cv::VideoCapture file("test.mp4");
cv::Mat frame;
while (true)
{
file.read(frame);
cv::imshow("Preview", frame);
cv::waitKey(42);
}
file.release();
return 0;
}
This works fine. But when I integrate this script into another project I'm working on. The the image frame shows in wrong aspect ratio.
Correct (Side by Side):
Wrong (Only show one side and aspect ratio is wrong):
I'm running on Windows VS2019. I have remove all other script in my exists project, just leave the above. The only different I can think of is the includes and linker setting. I use ceres, glog, d3d11, realsense2, VTK, pcl, eigen3, OpenXR in the project. Does any of that effects what OpenCV behave? Or what might be the problem?
I've already try setting the frame width and height for VideoCapture and it's not working.
I've test both OpenCV 4.1 and 4.6.
When accessing frame.cols, frame.rows, I got the correcy resolution.
UPDATE
The metadata of the file I'm trying to read is as below. This is a side by side 3D video. It also can display correctly in player such as VLC.
mediainfo test.mp4
General
Complete name : test.mp4
Format : MPEG-4
Format profile : Base Media
Codec ID : isom (isom/iso2/avc1/mp41)
File size : 181 MiB
Duration : 31 s 339 ms
Overall bit rate : 48.4 Mb/s
Writing application : Lavf58.20.100
Video
ID : 1
Format : AVC
Format/Info : Advanced Video Codec
Format profile : High#L5.1
Format settings : CABAC / 1 Ref Frames
Format settings, CABAC : Yes
Format settings, Reference frames : 1 frame
Format settings, GOP : M=1, N=30
Codec ID : avc1
Codec ID/Info : Advanced Video Coding
Duration : 31 s 317 ms
Bit rate : 48.0 Mb/s
Width : 3 840 pixels
Height : 1 080 pixels
Display aspect ratio : 16:9
Original display aspect ratio : 3.556
Frame rate mode : Variable
Frame rate : 60.000 FPS
Minimum frame rate : 59.920 FPS
Maximum frame rate : 60.080 FPS
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bits
Scan type : Progressive
Bits/(Pixel*Frame) : 0.193
Stream size : 179 MiB (99%)
Title : SStar Video
Codec configuration box : avcC
Audio
ID : 2
Format : AAC LC
Format/Info : Advanced Audio Codec Low Complexity
Codec ID : mp4a-40-2
Duration : 31 s 339 ms
Bit rate mode : Constant
Bit rate : 140 kb/s
Channel(s) : 2 channels
Channel layout : L R
Sampling rate : 48.0 kHz
Frame rate : 46.875 FPS (1024 SPF)
Compression mode : Lossy
Stream size : 534 KiB (0%)
Title : SStar Audio
Default : Yes
Alternate group : 1