I'm implementing a 3d VR player and for the video decoding I'm using Libvlc.
To render the video I have a texture buffer and a memory buffer, when VLC decodes a frame the data is copied by the main loop to the texture.
I've tested two different mechanisms to lock the memory buffer when vlc is dumping a frame, first I tried a mutex as the examples show but the performance is awful, also I've tested a lock-free mechanism using atomic operations, the performance is not perfect but better.
The best results I get are just not locking at all the buffer, it works very smooth but some times the main loop and vlc loop gets desinchronized and then tearing and stuttering are noticeable.
So my question is, what is the best approach to lock the buffer? any way to don't loss performance?
Here are the important code parts:
-Video context struct
typedef struct _videoContext
{
libvlc_instance_t* instance;
libvlc_media_t* media;
libvlc_media_player_t* player;
unsigned char *pixeldata;
unsigned char currentFrame;
int width;
int height;
int duration;
int time;
int volume;
bool finished;
std::atomic<bool> lock;
} videoContext;
-Video event functions
static void *lock(void *data, void **p_pixels)
{
videoContext* context = (videoContext*)data;
bool expected = false;
while (!context->lock.compare_exchange_strong(expected, true))
Sleep(0);
p_pixels[0] = context->pixeldata;
return NULL;
}
static void unlock(void *data, void *id, void *const *p_pixels)
{
videoContext* context = (videoContext*)data;
context->time = libvlc_media_player_get_time(context->player);
context->lock.store(false);
context->currentFrame++;
}
-Main loop function
if (vctx->currentFrame != currentFrame && vctx->lock.compare_exchange_strong(exchangeExpected, true))
{
currentFrame = vctx->currentFrame;
Texture::setData(&videoTexture, vctx->pixeldata, vctx->width, vctx->height);
vctx->lock.store(false);
}
Related
I want to make a common video player with opencv C++. Users should be able to freely move between frames of the video(commonly slider). So I simply wrote and tested two methods and both has problems.
My first solution:
int frameIdx = 0; /// This is accessed by other thread
cv::VideoCapture cap("video.mp4");
while (true) {
cv::Mat frame;
cap.set(cv::CAP_PROP_POS_FRAMES, frameIdx);
cap.read(frame);
showFrameToWindow(frame);
frameIdx++;
}
My second solution:
int frameIdx = 0; /// This is accessed by other thread
std::vector<cv::Mat> buffer;
cv::VideoCapture cap("video.mp4")
while (true) {
cv::Mat frame;
cap >> frame;
if (frame.empty()) break;
buffer.push_back(frame);
}
while (true) {
cv::Mat frame = buffer[frameIdx].clone();
showFrameToWindow(frame);
frameIdx++;
}
The first solution is too slow. I think there is an overhead cap.read(cv::Mat). It's not possible to play video more than 20~30fps in my computer.
The second solution, it is satisfactory for speed, but it requires a lot of memory space.
So I imagine what if I change std::vector to std::queue of the buffer, limit its size, and update it in other thread while playing the video.
I'm not sure it's gonna work, and I wonder there's an common algorithm to seek in large video file. Any comments will save me. Thanks.
I developed my second solution to limitied size of frame buffer and handling it with other thread.
/// These variables are accessed from all threads.
#define BUF_MAX 64
bool s_videoFileLoaded;
double s_videoFileHz; // milliseconds per frame
int s_videoFileFrameCount;
bool s_seek;
int s_seekingIndex; // indexing video file, binded with UI
std::queue<cv::Mat> s_frameBuffer;
std::queue<int> s_indexBuffer;
std::mutex s_mtx;
///
/// !!! When seeking slider in UI moved manually,
/// !!! #s_seek turns to True.
///
/// Start of main thread.
cv::VideoCapture cap("video.mp4");
s_videoFileLoaded = true;
s_videoFileFrameCount = cap.get(cv::CAP_PROP_FRAME_COUNT);
s_videoFileHz = 1000.0 / cap.get(cv::CAP_PROP_FPS);
s_seekingIndex = 0;
runThread( std::bind(&_videoHandlingThreadLoop, cap) );
Timer timer;
while (s_videoFileLoaded) {
timer.restart();
{
std::lock_guard<std::mutex> lock(s_mtx);
if (s_frameBuffer.empty())
continue;
cv::Mat frame = s_frameBuffer.front();
s_seekingIndex = s_indexBuffer.front();
s_frameBuffer.pop();
s_indexBuffer.pop();
showFrameToWindow(frame);
}
int remain = s_videoFileHz - timer.elapsed();
if (remain > 0) Sleep(remain);
}
void _videoHandlingThreadLoop(cv::VideoCapture& cap) {
s_seek = true;
int frameIndex = -1;
while (s_videoFileLoaded) {
if (s_frameBuffer.size() > BUF_MAX) {
Sleep(s_videoFileHz * BUF_MAX);
continue;
}
// Check whether it's time to 'seeking'.
if (s_seek) {
std::lock_guard<std::mutex> lock(s_mtx);
// Clear buffer
s_frameBuffer = std::queue<cv::Mat>();
s_indexBuffer = std::queue<int>();
frameIndex = s_seekingIndex;
cap.set(cv::CAP_PROP_POS_FRAMES, frameIndex);
s_seek = false;
}
// Read frame from the file and push to buffer.
cv::Mat frame;
if (cap.read(frame)) {
std::lock_guard<std::mutex> lock(s_mtx);
s_frameBuffer.push(frame);
s_indexBuffer.push(frameIndex);
frameIndex++;
}
// Check whether the frame is end of the file.
if (frameIndex >= s_videoFileFrameCount) {
s_seekingIndex = 0;
s_seek = true;
}
}
}
and this worked. I could play the video file with stable playback speed. But still had some lag when seeking manually.
By using the following code, I can successfully capture and return frame from a separate thread to main(). But now I cannot write to the frame/cv::Mat (contour in main()) for display. If I do not use separate thread to for webcam then I can successfully write to the frame/cv::Mat. What am I doing wrong?
VideoCapture cap(0);
class Camera
{
public:
Camera(void);
~Camera(void);
Mat captureVideo(void);
private:
Mat frame;
double dWidth;
double dHeight;
double fps;
};
Camera::Camera(void) {
int isrunning = 0;
usleep(10);
if (!cap.open(2))
{
cout << "Cannot open the video camera.\n" << endl;
}
else
{
isrunning = 1;
}
if(isrunning == 0) {
this->~Camera();
}
cap >> frame;
}
Camera::~Camera(void) {
cap.release();
}
Mat Camera::captureVideo(void) {
cap >> frame;
return frame;
}
Camera cam1;
const int frameBuffer = 20; // Frame buffer
std::vector<cv::Mat> frameStack;
int stopSig = 0; // Global stop signal...
void grabFrame(void) {
Mat frame;
::frameStack.clear();
while(!::stopSig) {
frame = ::cam1.captureVideo();
// 1. Remove one frame from the back, if the stack has more then 2 frames...
if(::frameStack.size() > 2) { //If the framestack has more then 2 frames...
::frameStack.pop_back();
}
// 2. Add a frame at the front of the stack if the stack is not full...
if (::frameStack.size() < ::frameBuffer) {
::frameStack.push_back(frame); // Put new frame on stack on the computer's RAM...
} else {
::frameStack.clear();
}
}
return;
}
int main(int argc, char* argv[])
{
Mat fr, outImg; // Captured single frames...
Mat contour; // Video stream showin countours of objects...
::frameStack.clear();
thread t1(grabFrame);
cv::namedWindow("MyWindow", 1);
while(1) {
if(::frameStack.size() >= 2) {
contour = ::frameStack.back();
circle(contour, Point(150,150),50, Scalar(0,255,255),cv::FILLED, 8);// PROBLEM -> NO EFFECT
putText(contour, "some text", Point(100,100), FONT_HERSHEY_DUPLEX, 1, Scalar(0,143,143), 2);// PROBLEM -> NO EFFECT
imshow("Contour Video", contour);
}
if (waitKey(1) == 27)
{
::stopSig = 1; // Signal to threads to end their run...
frameStack.clear();
break;
}
}
t1.join();
return 0;
}
In your example, you are sharing stopSig, frameStack and frameBuffer variables between the two threads.
When you are using shared memory between threads, you have to properly synchronize the access to that shared memory. One of the reasons why synchronization is needed is to prevent two threads modifying a variable at the same time. Another reason is visibility - a write to a variable by one thread can be cached inside the CPU cache, and a thread running on another CPU core won't see the variable change.
Perhaps the easiest way to synchronize memory access between threads is by using std::mutex. I'm not gonna post the whole code here, but you can read about mutexes in C++ online, for example, here.
First off i know the title is horribly worded I counldn't think of a better way to phrase it. Bassically I am using sfml and running multiple windows. I have a function that runs through a vector and that works like the while loop typically would. As the vector gets big windows update slowly to fix that I am trying to impliment multithreading. I have never tried to use multithreading before and this is probably a easy question but none of the basic tutorials I found coverd it.
this is the function im trying to run WindowArray[i]->setPosition(Gravity(KeyArray[i], WindowArray[i])).
If I do it like this it gets errors
std::thread thread1;
thread1 = thread(WindowArray[i]->setPosition, Gravity(KeyArray[i], WindowArray[i]));
How should I go about doing this?
something to put in a compiler
class NewKey {
public:
sf::Image Img;
sf::Texture Tex;
sf::Sprite Sprite;
sf::Vector2i Velocity;
sf::Vector2i Acceleration;
void StepVelocity() {
Velocity += Acceleration;
Acceleration = sf::Vector2i(0, 0);
}
};
vector <NewKey> KeyArray;
vector <unique_ptr <sf::RenderWindow>> WindowArray;
unsigned maxThreads = std::thread::hardware_concurrency();
int activeThreads;
int dropRate;
sf::Vector2i Gravity(NewKey& Key, unique_ptr <sf::RenderWindow>& window) {
dropRate = 10;
Key.Acceleration += sf::Vector2i(0, dropRate);
Key.StepVelocity;
sf::Vector2i Result = sf::Vector2i(window->getPosition() + Key.Velocity);
return Result;
}
void StepWindows()
{
for (int i{ 0 }; i < WindowArray.size(); i++)
{
while (!(activeThreads + 1) >= maxThreads)
{
activeThreads++;
std::thread thread1;
thread1 = thread(WindowArray[i]->setActive(true));
thread1 = thread(WindowArray[i]->setPosition, Gravity(KeyArray[i], WindowArray[i]));
activeThreads--;
}
}
}
int main() {
unique_ptr<sf::RenderWindow> window = make_unique<sf::RenderWindow>();
WindowArray.emplace_back(window);
StepWindows();
}
I'm trying to create a world generation system.
I have a world class that contain a map of terrain that can be increased during the program.
When I do it in the main thread it work fine but their is little freeze when he calculate the vertex and everything, so I tried to use Thread.
So to generate a terrain I do that :
std::thread newThread(&World::addTerrain, this, xIndex, zIndex, 64, 64);
newThread.detach();
(The addTerrain method)
void World::addTerrain(int x, int z, size_t len, size_t col) {
//allTerrain_[x].emplace(std::make_pair(z, Terrain(x*size_, z*size_, size_ / 2.0f, len, col, &shader_, &heightGenerator_)));
allTerrain_[x].emplace(std::make_pair(z, Terrain(x*size_, z*size_, size_/2.0f, len, col, &shader_, &waterShader_, &heightGenerator_)));
}
but when I do this, the new terrain is added to the map, but he look empty (nothing is drawn).
I'm not sure I'm using the right approach, so if you can help me it will be great !
Try this:
Add a mutex to World (terrain_mutex_)
Use a std::lock_guard<std::mutex> when reading from/writing to allTerrain_. Your painting code needs to use this mutex too.
In addTerrain:
void World::addTerrain(int x, int z, size_t len, size_t col) {
{ // lock while copying the terrain
std::lock_guard<std::mutex> guard(terrain_mutex_);
auto terrain_copy = allTerrain_;
}
// add more terrain to the copy
terrain_copy[x].emplace(z, Terrain(x*size_, z*size_, size_/2.0f, len, col,
&shader_, &waterShader_, &heightGenerator_));
{ // lock when swapping in the copy
std::lock_guard<std::mutex> guard(terrain_mutex_);
std::swap(terrain_copy, allTerrain_);
}
}
I am making a simple 2D game engine with OpenGL, and my SpriteRenderer class makes batches of sprites which use the same shader or texture. Once I sort them first by texture, then by shader, and then by depth, the uploadData() method of my SpriteRenderer class uploads the CPU generated vertex data to the GPU. For this I am using the orphaning method, which is said to be faster than a simple glBufferData call.
My problem is, when I have small numbers of sprites (<15), I have frames with huge stalls, but with numbers >20 up to 10 000 it runs smoothly. I wrapped opengl function into c++ classes. If I replace the VertexBufferObject::uploadOrphaned() method by the VertexBufferObject::upload() method, the stall goes away. Here is my vbo class:
//vbo.h
class VertexBufferObject
{
public:
VertexBufferObject();
VertexBufferObject(size_t allocSize);
~VertexBufferObject();
inline ui32 id(){ return (ui32)m_id; }
template <class T>
void addData(const T& data)
{
const byte* convertedData = reinterpret_cast<const byte*>(&data);
size_t size = sizeof(T);
for (ui32 i = 0; i < size; i++)m_data.add(convertedData[i]);
}
void initialize();
void resetData();
void upload(GLenum drawType);
void uploadOrphaned(GLenum drawType);
inline void bind() { ::glBindBuffer(GL_ARRAY_BUFFER, m_id); }
inline void unbind(){ ::glBindBuffer(GL_ARRAY_BUFFER, 0); }
private:
GLuint m_id;
//this is my custom dynamic array class, this is a hobby project
//and I enjoy reinventing the wheel :)
Array<byte> m_data;
and below is the cpp file:
VertexBufferObject::VertexBufferObject() :m_id(0)
{
}
VertexBufferObject::VertexBufferObject(size_t allocSize) : m_id(0), m_data(allocSize)
{
}
VertexBufferObject::~VertexBufferObject()
{
if (m_id != 0) ::glDeleteBuffers(1, &m_id);
}
void VertexBufferObject::initialize()
{
if (m_id == 0) ::glGenBuffers(1, &m_id);
}
void VertexBufferObject::upload(GLenum drawType)
{
glBufferData(GL_ARRAY_BUFFER, m_data.size(), &m_data[0], drawType);
}
void VertexBufferObject::uploadOrphaned(GLenum drawType)
{
glBufferData(GL_ARRAY_BUFFER, m_data.size() * 2, NULL, drawType);
glBufferSubData(GL_ARRAY_BUFFER, 0, m_data.size(), &m_data[0]);
}
void VertexBufferObject::resetData()
{
m_data.strip();
}
I am learning opengl, so my VertexBufferObject class contains what I already know. I know there are other methods to upload data, but I want to understand the reason of the current problem with this simple method. I looked around forums including here, and nobody had a similar issue that I found of. Maybe my class has some errors?