I'm writing a C++ program to encode frames from a DirectX game to the H.264/MPEG-4 AVC format. I am using libx264 alone with no other dependencies at the moment.
I have a ID3D11Texture2D* resolved back buffer of the next game frame. I need to somehow copy this into the x264_picture input (apparently YUV420P format according to limited help I've found) but I cannot find any way to do so online.
Here is my code at the moment:
void Fx264VideoEncoder::Fx264VideoEncoderImpl::InitFrameInputBuffer(const FTexture2DRHIRef& BackBuffer, FFrame& Frame)
{
x264_picture_alloc(Frame.InputPicture, X264_CSP_I420, x264Parameters.i_width, x264Parameters.i_height);
// We need to take the back buffer and convert it to an input format that libx264 can understand
{
ID3D11Texture2D* ResolvedBackBufferDX11 = (ID3D11Texture2D*)(GetD3D11TextureFromRHITexture(Frame.ResolvedBackBuffer)->GetResource());
EPixelFormat PixelFormat = Frame.ResolvedBackBuffer->GetFormat();
// ...?
}
}
Related
Background
I have a .webm file (pix_fmt: yuva420p) converted from .mov video file in order to reduce file size and I would like to read the video data using c++, so I followed using this repo as a reference.
This works perfectly on .mov video.
Problem
By using same repo, however, there is no alpha channel data (pure zeros on that channel) for .webm video but I can get the alpha data from .mov video.
Apparently many people already noticed that after the video conversion, ffmpeg somehow detect video as yuv420p + alpha_mode : 1 and thus alpha channel is not used but there is no one discuss about workaround of this.
I tried forcing pixel format during this part to use yuva420p but that just broke the whole program.
// Set up sws scaler
if (!sws_scaler_ctx) {
auto source_pix_fmt = correct_for_deprecated_pixel_format(av_codec_ctx->pix_fmt);
sws_scaler_ctx = sws_getContext(width, height, source_pix_fmt,
width, height, AV_PIX_FMT_RGB0,
SWS_BILINEAR, NULL, NULL, NULL);
}
I also verified my video that it contains alpha channel using other source so I am sure there is alpha channel in my webm video but I cannot fetch the data using ffmpeg.
Is there a way to fetch the alpha data? Other video format or using other libraries work as well as long as it does have some file compression but I need to access the data in c++.
Note: This is the code I used for converting video to webm
ffmpeg -i input.mov -c:v libvpx-vp9 -pix_fmt yuva420p output.webm
You have to force the decoder.
Set the following before avformat_open_input()
AVCodec *vcodec;
vcodec = avcodec_find_decoder_by_name("libvpx-vp9");
av_fmt_ctx->video_codec = vcodec;
av_fmt_ctx->video_codec_id = vcodec->id;
You don't need to set pixel format or any scaler args.
This assumes that your libavcodec is linked with libvpx.
Hello I am working on a Pylon Application, and I want to know how to draw image which is in array.
Basler's Pylon SDK, on memory Image saving
In this link, it shows I can save image data in array(I guess) Pylon::CImageFormatConverter::Convert(Pylon::CPylonImage[i],Pylon::CGrabResultPtr)
But the thing is I can not figure out how to draw that images which is in array.
I Think it would be easy problem, but I'd appreciate it if you understand because it's my first time doing this.
For quick'n dirty displaying of Pylon buffers, you have a helper class available in Pylon SDK to put somewhere in buffer grabbing loop:
//initialization
CPylonImage img;
CPylonImageWindow wnd;
//somewhere in grabbing loop
wnd.SetImage(img);
wnd.Show();
If you want to utilize in your own display procedures (via either GDI or 3rd party libraries like OpenCV etc.), you can get image buffer with GetBuffer() method:
CGrabResultPtr ptrGrabResult;
camera.RetrieveResult( 5000, ptrGrabResult, TimeoutHandling_ThrowException);
const uint8_t *pImageBuffer = (uint8_t *) ptrGrabResult->GetBuffer();
Of course, you have to be aware of your target pixel alignment and optionally use CImageFormatConverter for proper output pixel format of your choice.
CImageFormatConverter documentation
I have a simple C++ application that uses FFmpeg 3.2 to receive an H264 RTP stream. In order to save CPU, I'm doing the decoding part with the codec h264_cuvid. My FFmpeg 3.2 is compiled with hw acceleration enabled. In fact, if I do the command:
ffmpeg -hwaccels
I get
cuvid
This means that my FFmpeg setup has everything OK to "speak" with my NVIDIA card.
The frames that the function avcodec_decode_video2 provides me have the pixel format AV_PIX_FMT_CUDA. I need to convert those frames to new ones with AV_PIX_FMT_RGB. Unfortunately, I can't do the conversion using the well knwon functions sws_getContext and sws_scale because the pixel format AV_PIX_FMT_CUDA is not supported. If I try with swscale I get the error:
"cuda is not supported as input pixel format"
Do you know how to convert an FFmpeg AVFrame from AV_PIX_FMT_CUDA to AV_PIX_FMT_RGB ?
(pieces of code would be very appreciated)
This my understanding of the hardware decoding on the latest FFMPeg 4.1 version. Below are my conclusion after studying the source code.
First I recommend to inspire yourself from the hw_decode example:
https://github.com/FFmpeg/FFmpeg/blob/release/4.1/doc/examples/hw_decode.c
With the new API, when you send a packet to the encoder using avcodec_send_packet(), then use avcodec_receive_frame() to retrieve the decoded frame.
There are two different kinds of AVFrame: software one, which is stored in the "CPU" memory (a.k.a RAM), and hardware one, which is stored in the graphic card memory.
Getting AVFrame from the hardware
To retrieve the hardware frame and get it into a readable, convertible (with swscaler) AVFrame, av_hwframe_transfer_data() needs to be used to retrieve the data from the graphic card. Then look at the pixel format of the retrieved frame, it is usually NV12 format when using nVidia decoding.
// According to the API, if the format of the AVFrame is set before calling
// av_hwframe_transfer_data(), the graphic card will try to automatically convert
// to the desired format. (with some limitation, see below)
m_swFrame->format = AV_PIX_FMT_NV12;
// retrieve data from GPU to CPU
err = av_hwframe_transfer_data(
m_swFrame, // The frame that will contain the usable data.
m_decodedFrame, // Frame returned by avcodec_receive_frame()
0);
const char* gpu_pixfmt = av_get_pix_fmt_name((AVPixelFormat)m_decodedFrame->format);
const char* cpu_pixfmt = av_get_pix_fmt_name((AVPixelFormat)m_swFrame->format);
Listing supported "software" pixel formats
Side note here if you want to select the pixel format, not all AVPixelFormat are supported. AVHWFramesConstraints is your friend here:
AVHWDeviceType type = AV_HWDEVICE_TYPE_CUDA;
int err = av_hwdevice_ctx_create(&hwDeviceCtx, type, nullptr, nullptr, 0);
if (err < 0) {
// Err
}
AVHWFramesConstraints* hw_frames_const = av_hwdevice_get_hwframe_constraints(hwDeviceCtx, nullptr);
if (hw_frames_const == nullptr) {
// Err
}
// Check if we can convert the pixel format to a readable format.
AVPixelFormat found = AV_PIX_FMT_NONE;
for (AVPixelFormat* p = hw_frames_const->valid_sw_formats;
*p != AV_PIX_FMT_NONE; p++)
{
// Check if we can convert to the desired format.
if (sws_isSupportedInput(*p))
{
// Ok! This format can be used with swscale!
found = *p;
break;
}
}
// Don't forget to free the constraint object.
av_hwframe_constraints_free(&hw_frames_const);
// Attach your hw device to your codec context if you want to use hw decoding.
// Check AVCodecContext.hw_device_ctx!
Finally, a quicker way is probably the av_hwframe_transfer_get_formats() function, but you need to decode at least one frame.
Hope this will help!
You must use vf_scale_npp to do this. You can use either nppscale_deinterleave or nppscale_resize depend on your needs.
Both has same input parameters, which are AVFilterContext that should be initialize with nppscale_init, NPPScaleStageContext which takes your in/out pixel format and two AVFrames which of course are your input and output frames.
For more information you can see npplib\nppscale definition which will do the CUDA-accelerated format conversion and scaling since ffmpeg 3.1.
Anyway, I recommend to use NVIDIA Video Codec SDK directly for this purpose.
I am not an ffmpeg expert, but I had a similar problem and managed to solve it. I was getting AV_PIX_FMT_NV12 from cuvid (mjpeg_cuvid decoder), and wanted AV_PIX_FMT_CUDA for cuda processing.
I found that setting the pixel format just before decoding the frame worked.
pCodecCtx->pix_fmt = AV_PIX_FMT_CUDA; // change format here
avcodec_decode_video2(pCodecCtx, pFrame, &frameFinished, &packet);
// do something with pFrame->data[0] (Y) and pFrame->data[1] (UV)
You can check which pixel formats are supported by your decoder using pix_fmts:
AVCodec *pCodec = avcodec_find_decoder_by_name("mjpeg_cuvid");
for (int i = 0; pCodec->pix_fmts[i] != AV_PIX_FMT_NONE; i++)
std::cout << pCodec->pix_fmts[i] << std::endl;
I'm sure there's a better way of doing this, but I then used this list to map the integer pixel format ids to human readable pixel formats.
If that doesn't work, you can do a cudaMemcpy to transfer your pixels from device to host:
cudaMemcpy(pLocalBuf pFrame->data[0], size, cudaMemcpyDeviceToHost);
The conversion from YUV to RGB/RGBA can be done many ways. This example does it using the libavdevice API.
I am currently trying to encode some raw audio data with some video inside an avi container.
The video codec used is mpeg4 and I would like to use the PCM_16LE for the audio codec but I am facing a problem regarding the AVCodec->frame_size parameter for the audio samples.
After doing all the correct allocation, I try allocating the audio frame and for AV_CODEC_ID_PCM_S16LE codec I don't have the codec frame_size needed to get the samples buffer size. Therefore the sample buffer size is huge and I simply can't allocate such quantity of memory.
Does someone know how to bypass this issue and how to manually compute the frame_size ?
frame = av_frame_alloc();
if(!frame)
{
return NULL;
}
//Problem is right here with the frame_size
frame->nb_samples = m_pAudioCodecContext->frame_size;
frame->format = m_pAudioStream->codec->sample_fmt;
frame->channel_layout = m_pAudioStream->codec->channel_layout;
//The codec gives us the frame size, in samples, so we can calculate the size of the samples buffer in bytes
//This returns a huge value due to a null frame_size
m_audioSampleBufferSize = av_samples_get_buffer_size(NULL,
m_pAudioCodecContext->channels,
m_pAudioCodecContext->frame_size,
m_pAudioCodecContext->sample_fmt,
0);
Thank you for your help,
Robert
As you can see in pcm_encode_init function in pcm.c
All pcm encoders have frame_size = 0;. Why?
Because in all PCM formats 'De facto' no such thing like frame, there is no compression by nature of PCM.
So you should decide by your own how many samples you wanna to store in buffer
how do i take Data from a FIBITMAP (Free-Image image) object and store it into a binary file.
i already know how to store the data as an image but i would like to instead store it as a video akin to Fraps. my goal is to create a list of FIBITMAP images and write to them in a binary (.avi or .mp4 format) file.
this is my ScreenShot Code:
BYTE* pixels = new BYTE[ 3 * width * height];
glReadPixels(0, 0, width, height, GL_BGR, GL_UNSIGNED_BYTE, pixels);
// Convert to FreeImage format & save to file
FIBITMAP* image = FreeImage_ConvertFromRawBits(pixels, width, height, 3 * width, 24, 0x0000FF, 0xFF0000, 0x00FF00);
FreeImage_Save(FIF_PNG, image, "./images/test.avi");
// Free resources
FreeImage_Unload(image);
delete[] pixels;
Let's say you want to save it in avi. That is a container format, which specifies the stored video stream's meta-data (resolution of the frames, codec information, etc) and the stream itself.
There are many places where you can study the format of it, and also RIFF in general.
The other thing is that if you want to play your video file with an "outsider" software, then you have to provide a codec for the system, what the player software will find through your OS (you register the codec somehow, that's OS dependent), and the codec will decode the video stream contained in the avi and supply the frames to the player.
(Actually the player calls the decode function with chunks of the stream, and your codec should co-operate).
Of course in your video stream each frame is independent and independently coded from the others. But that doesn't matter from the OS-codec point of view. Your codec has an identifier (FOURCC), which should be set into the AVI codec section, and the player will search the codec using that identifier.
Maybe in your place I may think of motion JPEG. That's the closest existing thing compared to what you do. Only in that case the individual frames are not pngs but jpegs.