Reading h264 or h265 stream with ffmpeg/OpenCV: Which is faster? - c++

Question

I'm using OpenCV with ffmpeg support to read a RTSP stream coming from an IP camera and then to write the frames to a video. The problem is that the frame size is 2816x2816 at 20 fps i.e. there's a lot of data coming in.
I noticed that there was a significant delay in the stream, so I set the buffer size of the cv::VideoCapture object to 1, because I thought that the frames might just get stuck in the buffer instead of being grabbed and processed. This however just caused for frames to be dropped instead.
My next move was to experiment a bit with the frame size/fps and the encoding of the video that I'm writing. All of those things helped to improve the situation, but in the long run I still have to use a frame size of 2816x2816 and support up to 20 fps, so I can't set it lower sadly.
That's where my question comes in: given the fact that the camera stream is going to be either h264 or h265, which one would be read faster by the cv::VideoCapture object? And how should I encode the video I'm writing in order to minimize the time spent decoding/encoding frames?
That's the code I'm using for reference:
using namespace cv;
int main(int argc, char** argv)
{
VideoCapture cap;
cap.set(CAP_PROP_BUFFERSIZE, 1); // internal buffer will now store only 1 frames
if (!cap.open("rtsp://admin:admin#1.1.1.1:554/stream")) {
return -1;
}
VideoWriter videoWr;
Mat frame;
cap >> frame;
//int x264 = cv::VideoWriter::fourcc('x', '2', '6', '4'); //I was trying different options
int x264 = cv::VideoWriter::fourcc('M', 'J', 'P', 'G');
videoWr = cv::VideoWriter("test_video.avi", 0, 0, 20, frame.size(), true);
namedWindow("test", WINDOW_NORMAL);
cv::resizeWindow("test", 1024, 768);
for (;;)
{
cap >> frame;
if (frame.empty()) break; // end of video stream
imshow("test", frame);
if (waitKey(10) == 27) break;
videoWr << frame;
}
return 0;
}