OpenCV FFMPEG RTSP Camera Feed Errors - c++

Question

I'm getting these errors at random times when saving frames from an rtsp camera feed. The errors happen at different times, usually after 100-200 images have been saved, and the errors themselves are not always exactly the same. They cause the images that are saved at the time of the error to be distorted either to the point of being completely grey or contain distorted pixels.
#Frame_142 - [hevc # 0c3bf800] The cu_qp_delta 29 is outside the valid range [-26, 25].
#Frame_406 - [hevc # 0b6bdb80] Could not find ref with POC 41
I've tried implementing the code in both python and c++ with the same result. Also tried saving as .png instead of .jpg. The rtsp feed works fine when using imshow to display the camera, the problem only appears to happen when trying to save the frames. From what I can gather the errors have to do with ffmpeg but google isn't much help for these types of errors.
#include <iostream>
#include <opencv2\opencv.hpp>
#include <chrono>
#include <thread>
using namespace std;
using namespace cv;
int main() {
VideoCapture cap("rtsp://admin:admin#192.168.88.97/media/video1");
if (!cap.isOpened())
return -1;
for (int i = 0; i < 500; i++)
{
Mat frame;
cap >> frame;
imwrite("C:\\Users\\Documents\\Dev\\c++\\OpenCVExample\\frames\\frame" + std::to_string(i) + ".png", frame);
cout << i << "\n";
std::this_thread::sleep_for(std::chrono::milliseconds(10));
}
return 0;
}

Answer 1

OpenCV and RTSP streams are a little tricky. This usually happens when the frames of the stream cannot be read fast enough. In that case (somewhere) a buffer receiving RTSP frames will be filled faster than is can be emptied; when the buffer is full it will be flushed and will restart filling from the most recent frame. This will corrupt the stream decoding until the next keyframe.
My suggestion is to remove the sleep even if 10ms is pretty small. Since this won't be enough (:P), the only solution is to implement a thread constantly reading frames from the cv::VideoCapture and discarding the ones grabbed while the main thread is busy saving the previous frame.
A simple implementation could be:
#include <opencv2/opencv.hpp>
#include <thread>
class VideoSourceHandler
{
public:
// Video file or IP camera
VideoSourceHandler( const std::string & source ) :
mCapture( source ),
mRun( false )
{
}
// USB camera
VideoSourceHandler( int webcamID ) :
mCapture( webcamID ),
mRun( false )
{
}
// start and stopCapture can be squashed into C'tor and D'tor if you want RTTI
void startCapture()
{
mRun = true;
mLoopThread = std::thread( &VideoSourceHandler::threadLoop, this );
}
void stopCapture()
{
mRun = false;
mLoopThread.join();
}
cv::Mat getFrame()
{
std::this_thread::yield(); // Be nice
const std::lock_guard<std::mutex> lock( mMutex );
return mCurrentFrame;
}
private:
void threadLoop()
{
while( mRun )
{
// Sleep if you want to "control" FPS
{
const std::lock_guard<std::mutex> lock( mMutex );
mCapture >> mCurrentFrame;
}
std::this_thread::yield(); // Be nice
}
}
cv::VideoCapture mCapture;
std::thread mLoopThread;
std::mutex mMutex;
bool mRun;
cv::Mat mCurrentFrame;
};
int main()
{
VideoSourceHandler vsh( 0 );
vsh.startCapture();
while( true )
{
cv::Mat frame = vsh.getFrame();
if( frame.empty() )
continue;
cv::imshow( "Test", frame );
char key = cv::waitKey( 1 );
if( key == 'q' || key == 27 )
break;
}
vsh.stopCapture();
return 0;
}

Answer 2

Changing the video compression to H.264 resolved the issue for me.