Display FFMPEG decoded frame in a GLFW window - c++

Question

I am implementing the client program of a game where the server sends encoded frames of the game to the client (via UDP), while the client decodes them (via FFMPEG) and displays them in a GLFW window.
My program has two threads:
Thread 1: renders the content of the uint8_t* variable dataToRender
Thread 2: keeps obtaining frames from the server, decodes them and updates dataToRender accordingly
Thread 1 does the typical rendering of a GLFW window in a while-loop. I have already tried to display some dummy frame data (a completely red frame) and it worked:
while (!glfwWindowShouldClose(window)) {
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);
...
glBindTexture(GL_TEXTURE_2D, tex_handle);
glTexImage2D(GL_TEXTURE_2D, 0, GL_RGB, window_width, window_height, 0, GL_RGB, GL_UNSIGNED_BYTE, dataToRender);
...
glfwSwapBuffers(window);
}
Thread 2 is where I am having trouble. I am unable to properly store the decoded frame into my dataToRender variable. On top if it, the frame data is originally in YUV format and needs to be converted to RGB. I use FFMPEG's sws_scale for that, which also gives me a bad dst image pointers error output in the console. Here's the code snippet responsible for that part:
size_t data_size = frameBuffer.size(); // frameBuffer is a std::vector where I accumulate the frame data chunks
uint8_t* data = frameBuffer.data(); // convert the vector to a pointer
picture->format = AV_PIX_FMT_RGB24;
av_frame_get_buffer(picture, 1);
while (data_size > 0) {
int ret = av_parser_parse2(parser, c, &pkt->data, &pkt->size,
data, data_size, AV_NOPTS_VALUE, AV_NOPTS_VALUE, 0);
if (ret < 0) {
fprintf(stderr, "Error while parsing\n");
exit(1);
}
data += ret;
data_size -= ret;
if (pkt->size) {
swsContext = sws_getContext(
c->width, c->height,
AV_PIX_FMT_YUV420P, c->width, c->height,
AV_PIX_FMT_RGB24, SWS_BILINEAR, NULL, NULL, NULL
);
uint8_t* rgb24[1] = { data };
int rgb24_stride[1] = { 3 * c->width };
sws_scale(swsContext, rgb24, rgb24_stride, 0, c->height, picture->data, picture->linesize);
decode(c, picture, pkt, outname);
// TODO: copy content of picture->data[0] to "dataToRender" maybe?
}
}
I have already tried doing another sws_scale to copy the content to dataToRender and I cannot get rid of the bad dst image pointers error. Any advice or solution to the problem would be greatly appreciated as I have been stuck for days on this.

Answer 1

I think you should convert YUV to RGB using OpenGL. That is much high efficiency and simple. The fragment shader looks like below:
precision mediump float;
varying vec2 v_texPo;
uniform sampler2D sampler_y;
uniform sampler2D sampler_u;
uniform sampler2D sampler_v;
void main() {
float y, u, v;
vec3 rgb;
y = texture2D(sampler_y, v_texPo).r;
u = texture2D(sampler_u, v_texPo).r - 0.5;
v = texture2D(sampler_v, v_texPo).r - 0.5;
rgb.r = y + 1.403 * v;
rgb.g = y - 0.344 * u - 0.714 * v;
rgb.b = y + 1.770 * u;
gl_FragColor = vec4(rgb, 1);
}
And you should upload three texture to OpenGL.