How to efficiently seek in video files with OpenCV? - c++

Question

I want to make a common video player with opencv C++. Users should be able to freely move between frames of the video(commonly slider). So I simply wrote and tested two methods and both has problems.
My first solution:
int frameIdx = 0; /// This is accessed by other thread
cv::VideoCapture cap("video.mp4");
while (true) {
cv::Mat frame;
cap.set(cv::CAP_PROP_POS_FRAMES, frameIdx);
cap.read(frame);
showFrameToWindow(frame);
frameIdx++;
}
My second solution:
int frameIdx = 0; /// This is accessed by other thread
std::vector<cv::Mat> buffer;
cv::VideoCapture cap("video.mp4")
while (true) {
cv::Mat frame;
cap >> frame;
if (frame.empty()) break;
buffer.push_back(frame);
}
while (true) {
cv::Mat frame = buffer[frameIdx].clone();
showFrameToWindow(frame);
frameIdx++;
}
The first solution is too slow. I think there is an overhead cap.read(cv::Mat). It's not possible to play video more than 20~30fps in my computer.
The second solution, it is satisfactory for speed, but it requires a lot of memory space.
So I imagine what if I change std::vector to std::queue of the buffer, limit its size, and update it in other thread while playing the video.
I'm not sure it's gonna work, and I wonder there's an common algorithm to seek in large video file. Any comments will save me. Thanks.
I developed my second solution to limitied size of frame buffer and handling it with other thread.
/// These variables are accessed from all threads.
#define BUF_MAX 64
bool s_videoFileLoaded;
double s_videoFileHz; // milliseconds per frame
int s_videoFileFrameCount;
bool s_seek;
int s_seekingIndex; // indexing video file, binded with UI
std::queue<cv::Mat> s_frameBuffer;
std::queue<int> s_indexBuffer;
std::mutex s_mtx;
///
/// !!! When seeking slider in UI moved manually,
/// !!! #s_seek turns to True.
///
/// Start of main thread.
cv::VideoCapture cap("video.mp4");
s_videoFileLoaded = true;
s_videoFileFrameCount = cap.get(cv::CAP_PROP_FRAME_COUNT);
s_videoFileHz = 1000.0 / cap.get(cv::CAP_PROP_FPS);
s_seekingIndex = 0;
runThread( std::bind(&_videoHandlingThreadLoop, cap) );
Timer timer;
while (s_videoFileLoaded) {
timer.restart();
{
std::lock_guard<std::mutex> lock(s_mtx);
if (s_frameBuffer.empty())
continue;
cv::Mat frame = s_frameBuffer.front();
s_seekingIndex = s_indexBuffer.front();
s_frameBuffer.pop();
s_indexBuffer.pop();
showFrameToWindow(frame);
}
int remain = s_videoFileHz - timer.elapsed();
if (remain > 0) Sleep(remain);
}
void _videoHandlingThreadLoop(cv::VideoCapture& cap) {
s_seek = true;
int frameIndex = -1;
while (s_videoFileLoaded) {
if (s_frameBuffer.size() > BUF_MAX) {
Sleep(s_videoFileHz * BUF_MAX);
continue;
}
// Check whether it's time to 'seeking'.
if (s_seek) {
std::lock_guard<std::mutex> lock(s_mtx);
// Clear buffer
s_frameBuffer = std::queue<cv::Mat>();
s_indexBuffer = std::queue<int>();
frameIndex = s_seekingIndex;
cap.set(cv::CAP_PROP_POS_FRAMES, frameIndex);
s_seek = false;
}
// Read frame from the file and push to buffer.
cv::Mat frame;
if (cap.read(frame)) {
std::lock_guard<std::mutex> lock(s_mtx);
s_frameBuffer.push(frame);
s_indexBuffer.push(frameIndex);
frameIndex++;
}
// Check whether the frame is end of the file.
if (frameIndex >= s_videoFileFrameCount) {
s_seekingIndex = 0;
s_seek = true;
}
}
}
and this worked. I could play the video file with stable playback speed. But still had some lag when seeking manually.