I suffered some choppy audio when i try to capture audio from a live stream.
Another essential problem which could explain the problem is that the Wav file created is twice longer than the capture time.
The audio is perfect when i play the avs input file with ffplay, so the avs is ok, the problem is after whether in the capture or in the Wav writing.
To capture :
av_read_frame(pFormatCtx, &packet)
if(packet.stream_index == mAudioStream)
{
int buff_size = sizeof(mAudioBuffer);
std::cout << "Buff_size " << buff_size << std::endl;
len = avcodec_decode_audio3(pAudioCodecCtx,(int16_t*)mAudioBuffer, &buff_size,&packet);
if(len < 0){
qDebug("Extractor - Audio isEnd = -1;");
mAudioBufferSize = 0;
isEnd = ERROR_;
return isEnd;
}
// Set packet result type
mFrameType = AUDIO_PKT;
mAudioBufferSize = buff_size;
//store audio synchronization informations:
if(packet.pts != AV_NOPTS_VALUE) {
mAudioPts_ = av_q2d(pFormatCtx->streams[mAudioStream]->time_base);
mAudioPts_ *= packet.pts;
}
}
// store a copy of current audio frame in _frame
_frame.audioFrame = new decoded_frame_t::audio_frame_t();
_frame.audioFrame->sampleRate = mediaInfos.audioSampleRate;
_frame.audioFrame->sampleSize = mediaInfos.audioSampleSize;
_frame.audioFrame->nbChannels = mediaInfos.audioNbChannels;
_frame.audioFrame->nbSamples = mAudioBufferSize / ((mediaInfos.audioSampleSize/8) * mediaInfos.audioNbChannels);
_frame.audioFrame->buf.resize(mAudioBufferSize);
memcpy(&_frame.audioFrame->buf[0],mAudioBuffer,mAudioBufferSize);
Then i store in a Wav File using libsndfile :
SNDFILE* fd;
SF_INFO sfInf;
sfInf.frames = 0;
sfInf.channels = p_capt->ui_nbChannels;
sfInf.samplerate = p_capt->ui_sampleRate;
sfInf.format = SF_FORMAT_WAV | SF_FORMAT_PCM_U8;
sfInf.sections = 0;
sfInf.seekable = 0;
if (sf_format_check(&sfInf) == FALSE)
std::cout << "Format parameter are uncorrect ! Exit saving !" << std::endl;
else
{
fd = sf_open(fileName.toStdString().c_str(), SFM_WRITE, &sfInf);
if (fd == NULL)
{
std::cout << "Unable to open the file " << fileName.toStdString() << std::endl;
return GRAB_ST_NOK;
}
//little trick because v_buf is a uint8_t vector
sf_count_t l = sf_write_short(fd, (const short *)(&(p_capt->v_buf[0])), p_capt->v_buf.size()/2);
if (l != p_capt->v_buf.size()/2)
{
std::cout << "sf_write didn't write the right amoung of bits " << l << " != " << p_capt->v_buf.size()/2 << std::endl;
ret = GRAB_ST_NOK;
}
else
{
sf_write_sync(fd);
sf_close(fd);
ret = GRAB_ST_OK;
}
}
I hope it's understandable. Waiting for remarks.
Kurt
Ok problem solved.
There were two main problems :
resize DO add n element and is not just preparing the vector for further push etc...
the buff_size of avcodec_decode_audio3 return a length in bytes but is copying in a int16_t array so it can be disturbing.
Related
I am using libx264 compiled from source. It was configured to get both .dll and .lib by this command
./configure --disable-cli --enable-shared --extra-ldflags=-Wl,--output-def=libx264.def`
I am using the libx264 API in my screen-sharing program with the preset - "veryfast", tune - "zerolatency", profile - "high" and also the following settings.
param.i_csp = X264_CSP_BGRA;
param.i_threads = 1;
param.i_width = width;
param.i_height = height;
param.i_fps_num = fps;
param.i_fps_den = 1;
param.rc.i_bitrate = bitrate;
param.rc.i_rc_method = X264_RC_ABR;
param.rc.b_filler = true;
param.rc.f_rf_constant = (float)0;
param.rc.i_vbv_max_bitrate = param.rc.i_bitrate;
param.rc.i_vbv_buffer_size = param.rc.i_bitrate;
param.b_repeat_headers = 0;
param.b_annexb = 1;
For these settings the program works fine. I specified it as single threaded by setting param.i_threads = 1.
If this is removed, x264 defaults to using multiple threads and sets param.i_threads as 1.5x of number of cores in the CPU automatically. This will give faster performance than running in single thread.
But when I remove the param.i_threads = 1 to make it multi-threaded, the generated output is fully grey. I cannot see any output when I view the live stream with VLC or some times I can view a weird output.
I am using this bitmap image as an example (https://imgur.com/a/l8LCd1l). Only this same image is being encoded multiple times. When it is saved into .h264 video, it is viewable clearly. But when the encoded payload is sent through rtmp, the live stream produces very bad and weird output (or sometimes no output). This is the weird output which im seeing most of the time for this image: https://imgur.com/a/VdyX1Zm
This is the full example code in which I am both streaming and writing video file of the same picture. This is using the srs librtmp library. There is no error but the stream has weird output.
In this code if you set add param.i_threads = 1; then only the output stream will be viewable. The problem is that it should be viewable in both single-threaded and multi-threaded encoding.
#include <iostream>
#include <stdio.h>
#include <sstream>
#include <x264.h>
#include "srs_librtmp.h"
#pragma comment(lib, "C:/Softwares/x264/libx264.lib")
using namespace std;
int check_ret(int ret);
int main()
{
int dts = 0;
x264_param_t param;
x264_t* h;
x264_nal_t* nals;
int i_nal;
int pts = 0;
int i_frame_size;
x264_picture_t picIn;
x264_picture_t picOut;
x264_param_default_preset(¶m, "veryfast", "zerolatency");
//x264 settings
param.i_csp = X264_CSP_BGRA;
param.i_width = 1920;
param.i_height = 1080;
param.i_fps_num = 30;
param.i_fps_den = 1;
param.rc.i_bitrate = 2500;
param.rc.i_rc_method = X264_RC_ABR;
param.rc.b_filler = true;
param.rc.f_rf_constant = (float)0;
param.rc.i_vbv_max_bitrate = param.rc.i_bitrate;
param.rc.i_vbv_buffer_size = param.rc.i_bitrate;
param.b_repeat_headers = 0;
param.b_annexb = 1;
x264_param_apply_profile(¶m, "high");
h = x264_encoder_open(¶m);
//allocate picture
x264_picture_alloc(&picIn, param.i_csp, param.i_width, param.i_height);
//picture settings
picIn.img.i_plane = 1;
picIn.img.i_stride[0] = 4 * param.i_width;
picIn.i_type = X264_TYPE_AUTO;
int header_size = x264_encoder_headers(h, &nals, &i_nal);
FILE* fptr;
fopen_s(&fptr, "example1.h264", "wb");
// write sps and pps in the video file
fwrite(nals->p_payload, header_size, 1, fptr);
int size = 1920 * 1080 * 4;
char* bmp = new char[size];
FILE* bitptr;
errno_t err = fopen_s(&bitptr, "flower.bmp", "rb");
fseek(bitptr, 54, SEEK_SET);
fread(bmp, size, 1, bitptr);
fclose(bitptr);
srs_rtmp_t rtmp = srs_rtmp_create("127.0.0.1:1935/live/test");
if (srs_rtmp_handshake(rtmp) != 0)
{
std::cout << "Simple handshake failed.";
return -1;
}
std::cout << "Handshake completed successfully.\n";
if (srs_rtmp_connect_app(rtmp) != 0) {
std::cout << "Connecting to host failed.";
return -1;
}
std::cout << "Connected to host successfully.\n";
if (srs_rtmp_publish_stream(rtmp) != 0) {
std::cout << "Publish signal failed.";
}
std::cout << "Publish signal success\n";
// write sps and pps in the live stream
int ret = srs_h264_write_raw_frames(rtmp, reinterpret_cast<char*>(nals->p_payload), header_size, 0, 0);
ret = check_ret(ret);
if (!ret)
return -1;
std::cout << "SPS and PPS sent.\n";
// main loop
std::cout << "Now streaming and encoding\n";
int i = 1800;
while (i--)
{
picIn.img.plane[0] = reinterpret_cast<uint8_t*>(bmp);
picIn.i_pts = pts++;
i_frame_size = x264_encoder_encode(h, &nals, &i_nal, &picIn, &picOut);
if (i_frame_size)
{
for (int j = 0; j < i_nal; j++)
{
x264_nal_t* nal = nals + j;
// write data in the video file
fwrite(nal->p_payload, nal->i_payload, 1, fptr);
// write data in the live stream
ret = srs_h264_write_raw_frames(rtmp, reinterpret_cast<char*>(nal->p_payload), nal->i_payload, dts, dts);
ret = check_ret(ret);
if (!ret)
{
return -1;
}
}
}
else
{
std::cout << "i_frame_size = 0 (encoder failed)\n";
}
dts += 33;
}
while (x264_encoder_delayed_frames(h))
{
i_frame_size = x264_encoder_encode(h, &nals, &i_nal, NULL, &picOut);
if (i_frame_size)
{
fwrite(nals->p_payload, i_frame_size, 1, fptr);
}
}
std::cout << "\nAll done\n";
std::cout << "Output video is example1.h264 and it is viewable in VLC";
return 0;
}
int check_ret(int ret)
{
if (ret != 0) {
if (srs_h264_is_dvbsp_error(ret)) {
srs_human_trace("ignoring drop video error, code=%d", ret);
}
else if (srs_h264_is_duplicated_sps_error(ret)) {
srs_human_trace("ignoring duplicated sps, code=%d", ret);
}
else if (srs_h264_is_duplicated_pps_error(ret)) {
srs_human_trace("ignoring duplicated pps, code=%d", ret);
}
else {
srs_human_trace("sending h264 raw data failed. ret=%d", ret);
return 0;
}
}
return 1;
}
If you would like to download the original flower.bmp file, here is the link: https://gofile.io/d/w2kX56
This error can be reproduced in any other bmp file also.
Please tell me what is causing this problem when multi-threading is enabled. Am I setting wrong values? Is the code in which I am streaming the encoded data wrong?
I am trying to decode a video stream from the browser using the ffmpeg API. The stream is produced by the webcam and recorded with MediaRecorder as webm format. What I ultimately need is a vector of opencv cv::Mat objects for further processing.
I have written a C++ webserver using the uWebsocket library. The video stream is sent via websocket from the browser to the server once per second. On the server, I append the received data to my custom buffer and decode it with the ffmpeg API.
If I just save the data on the disk and later I play it with a media player, it works fine. So, whatever the browser sends is a valid video.
I do not think that I correctly understand how should the custom IO behave with network streaming as nothing seems to be working.
The custom buffer:
struct Buffer
{
std::vector<uint8_t> data;
int currentPos = 0;
};
The readAVBuffer method for custom IO
int MediaDecoder::readAVBuffer(void* opaque, uint8_t* buf, int buf_size)
{
MediaDecoder::Buffer* mbuf = (MediaDecoder::Buffer*)opaque;
int count = 0;
for(int i=0;i<buf_size;i++)
{
int index = i + mbuf->currentPos;
if(index >= (int)mbuf->data.size())
{
break;
}
count++;
buf[i] = mbuf->data.at(index);
}
if(count > 0) mbuf->currentPos+=count;
std::cout << "read : "<<count<<" "<<mbuf->currentPos<<", buff size:"<<mbuf->data.size() << std::endl;
if(count <= 0) return AVERROR(EAGAIN); //is this error that should be returned? It cannot be EOF since we're not done yet, most likely
return count;
}
The big decode method, that's supposed to return whatever frames it could read
std::vector<cv::Mat> MediaDecoder::decode(const char* data, size_t length)
{
std::vector<cv::Mat> frames;
//add data to the buffer
for(size_t i=0;i<length;i++) {
buf.data.push_back(data[i]);
}
//do not invoke the decoders until we have 1MB of data
if(((buf.data.size() - buf.currentPos) < 1*1024*1024) && !initializedCodecs) return frames;
std::cout << "decoding data length "<<length<<std::endl;
if(!initializedCodecs) //initialize ffmpeg objects. Custom I/O, format, decoder, etc.
{
//these are just members of the class
avioCtxPtr = std::unique_ptr<AVIOContext,avio_context_deleter>(
avio_alloc_context((uint8_t*)av_malloc(4096),4096,0,&buf,&readAVBuffer,nullptr,nullptr),
avio_context_deleter());
if(!avioCtxPtr)
{
std::cerr << "Could not create IO buffer" << std::endl;
return frames;
}
fmt_ctx = std::unique_ptr<AVFormatContext,avformat_context_deleter>(avformat_alloc_context(),
avformat_context_deleter());
fmt_ctx->pb = avioCtxPtr.get();
fmt_ctx->flags |= AVFMT_FLAG_CUSTOM_IO ;
//fmt_ctx->max_analyze_duration = 2 * AV_TIME_BASE; // read 2 seconds of data
{
AVFormatContext *fmtCtxRaw = fmt_ctx.get();
if (avformat_open_input(&fmtCtxRaw, "", nullptr, nullptr) < 0) {
std::cerr << "Could not open movie" << std::endl;
return frames;
}
}
if (avformat_find_stream_info(fmt_ctx.get(), nullptr) < 0) {
std::cerr << "Could not find stream information" << std::endl;
return frames;
}
if((video_stream_idx = av_find_best_stream(fmt_ctx.get(), AVMEDIA_TYPE_VIDEO, -1, -1, nullptr, 0)) < 0)
{
std::cerr << "Could not find video stream" << std::endl;
return frames;
}
AVStream *video_stream = fmt_ctx->streams[video_stream_idx];
AVCodec *dec = avcodec_find_decoder(video_stream->codecpar->codec_id);
video_dec_ctx = std::unique_ptr<AVCodecContext,avcodec_context_deleter> (avcodec_alloc_context3(dec),
avcodec_context_deleter());
if (!video_dec_ctx)
{
std::cerr << "Failed to allocate the video codec context" << std::endl;
return frames;
}
avcodec_parameters_to_context(video_dec_ctx.get(),video_stream->codecpar);
video_dec_ctx->thread_count = 1;
/* video_dec_ctx->max_b_frames = 0;
video_dec_ctx->frame_skip_threshold = 10;*/
AVDictionary *opts = nullptr;
av_dict_set(&opts, "refcounted_frames", "1", 0);
av_dict_set(&opts, "deadline", "1", 0);
av_dict_set(&opts, "auto-alt-ref", "0", 0);
av_dict_set(&opts, "lag-in-frames", "1", 0);
av_dict_set(&opts, "rc_lookahead", "1", 0);
av_dict_set(&opts, "drop_frame", "1", 0);
av_dict_set(&opts, "error-resilient", "1", 0);
int width = video_dec_ctx->width;
videoHeight = video_dec_ctx->height;
if(avcodec_open2(video_dec_ctx.get(), dec, &opts) < 0)
{
std::cerr << "Failed to open the video codec context" << std::endl;
return frames;
}
AVPixelFormat pFormat = AV_PIX_FMT_BGR24;
img_convert_ctx = std::unique_ptr<SwsContext,swscontext_deleter>(sws_getContext(width, videoHeight,
video_dec_ctx->pix_fmt, width, videoHeight, pFormat,
SWS_BICUBIC, nullptr, nullptr,nullptr),swscontext_deleter());
frame = std::unique_ptr<AVFrame,avframe_deleter>(av_frame_alloc(),avframe_deleter());
frameRGB = std::unique_ptr<AVFrame,avframe_deleter>(av_frame_alloc(),avframe_deleter());
int numBytes = av_image_get_buffer_size(pFormat, width, videoHeight,32 /*https://stackoverflow.com/questions/35678041/what-is-linesize-alignment-meaning*/);
std::unique_ptr<uint8_t,avbuffer_deleter> imageBuffer((uint8_t *) av_malloc(numBytes*sizeof(uint8_t)),avbuffer_deleter());
av_image_fill_arrays(frameRGB->data,frameRGB->linesize,imageBuffer.get(),pFormat,width,videoHeight,32);
frameRGB->width = width;
frameRGB->height = videoHeight;
initializedCodecs = true;
}
AVPacket pkt;
av_init_packet(&pkt);
pkt.data = nullptr;
pkt.size = 0;
int read_frame_return = 0;
while ( (read_frame_return=av_read_frame(fmt_ctx.get(), &pkt)) >= 0)
{
readFrame(&frames,&pkt,video_dec_ctx.get(),frame.get(),img_convert_ctx.get(),
videoHeight,frameRGB.get());
//if(cancelled) break;
}
avioCtxPtr->eof_reached = 0;
avioCtxPtr->error = 0;
//flush
// readFrame(frames.get(),nullptr,video_dec_ctx.get(),frame.get(),
// img_convert_ctx.get(),videoHeight,frameRGB.get());
avioCtxPtr->eof_reached = 0;
avioCtxPtr->error = 0;
if(frames->size() <= 0)
{
std::cout << "buffer pos: "<<buf.currentPos<<", buff size:"<<buf.data.size()
<<",read_frame_return:"<<read_frame_return<< std::endl;
}
return frames;
}
What I would expect to happen would be for a continuous extraction of cv::Mat frames as I feed it more and more data. What actually happens is that after the the buffer is fully read I see:
[matroska,webm # 0x507b450] Read error at pos. 1278266 (0x13813a)
[matroska,webm # 0x507b450] Seek to desired resync point failed. Seeking to earliest point available instead.
And then no more bytes are read from the buffer even if later I increase the size of it.
There is something terribly wrong I'm doing here and I don't understand what.
What I ended up doing was to do the reading of the incoming data and actual decoding in a different thread. The read method, however, will just block if there are no more bytes available, waiting until anything is coming.
When new bytes are arriving, they're added to the buffer and the conditional_variable signals the waiting thread to wake up and start reading data again from the buffer.
It works well enough.
I'm currently working on a project similar to OBS, where I'm capturing screen data, encoding it with the x264 library, and then broadcasting it to a twitch server.
Currently, the servers are accepting the data, but no video is being played - it buffers for a moment, then returns an error code "2000: network error"
Like OBS Classic, I'm dividing each NAL provided by x264 by its type, and then making changes to each
int frame_size = x264_encoder_encode(encoder, &nals, &num_nals, &pic_in, &pic_out);
//sort the NAL's into their types and make necessary adjustments
int timeOffset = int(pic_out.i_pts - pic_out.i_dts);
timeOffset = htonl(timeOffset);//host to network translation, ensure the bytes are in the right format
BYTE *timeOffsetAddr = ((BYTE*)&timeOffset) + 1;
videoSection sect;
bool foundFrame = false;
uint8_t * spsPayload = NULL;
int spsSize = 0;
for (int i = 0; i<num_nals; i++) {
//std::cout << "VideoEncoder: EncodedImages Size: " << encodedImages->size() << std::endl;
x264_nal_t &nal = nals[i];
//std::cout << "NAL is:" << nal.i_type << std::endl;
//need to account for pps/sps, seems to always be the first frame sent
if (nal.i_type == NAL_SPS) {
spsSize = nal.i_payload;
spsPayload = (uint8_t*)malloc(spsSize);
memcpy(spsPayload, nal.p_payload, spsSize);
} else if (nal.i_type == NAL_PPS){
//pps always happens after sps
if (spsPayload == NULL) {
std::cout << "VideoEncoder: critical error, sps not set" << std::endl;
}
uint8_t * payload = (uint8_t*)malloc(nal.i_payload + spsSize);
memcpy(payload, spsPayload, spsSize);
memcpy(payload, nal.p_payload + spsSize, nal.i_payload);
sect = { nal.i_payload + spsSize, payload, nal.i_type };
encodedImages->push(sect);
} else if (nal.i_type == NAL_SEI || nal.i_type == NAL_FILLER) {
//these need some bytes at the start removed
BYTE *skip = nal.p_payload;
while (*(skip++) != 0x1);
int skipBytes = (int)(skip - nal.p_payload);
int newPayloadSize = (nal.i_payload - skipBytes);
uint8_t * payload = (uint8_t*)malloc(newPayloadSize);
memcpy(payload, nal.p_payload + skipBytes, newPayloadSize);
sect = { newPayloadSize, payload, nal.i_type };
encodedImages->push(sect);
} else if (nal.i_type == NAL_SLICE_IDR || nal.i_type == NAL_SLICE) {
//these packets need an additional section at the start
BYTE *skip = nal.p_payload;
while (*(skip++) != 0x1);
int skipBytes = (int)(skip - nal.p_payload);
std::vector<BYTE> bodyData;
if (!foundFrame) {
if (nal.i_type == NAL_SLICE_IDR) { bodyData.push_back(0x17); } else { bodyData.push_back(0x27); } //add a 17 or a 27 as appropriate
bodyData.push_back(1);
bodyData.push_back(*timeOffsetAddr);
foundFrame = true;
}
//put into the payload the bodyData followed by the nal payload
uint8_t * bodyDataPayload = (uint8_t*)malloc(bodyData.size());
memcpy(bodyDataPayload, bodyData.data(), bodyData.size() * sizeof(BYTE));
int newPayloadSize = (nal.i_payload - skipBytes);
uint8_t * payload = (uint8_t*)malloc(newPayloadSize + sizeof(bodyDataPayload));
memcpy(payload, bodyDataPayload, sizeof(bodyDataPayload));
memcpy(payload + sizeof(bodyDataPayload), nal.p_payload + skipBytes, newPayloadSize);
int totalSize = newPayloadSize + sizeof(bodyDataPayload);
sect = { totalSize, payload, nal.i_type };
encodedImages->push(sect);
} else {
std::cout << "VideoEncoder: Nal type did not match expected" << std::endl;
continue;
}
}
The NAL payload data is then put into a struct, VideoSection, in a queue buffer
//used to transfer encoded data
struct videoSection {
int frameSize;
uint8_t* payload;
int type;
};
After which it is picked up by the broadcaster, a few more changes are made, and then I call rtmp_send()
videoSection sect = encodedImages->front();
encodedImages->pop();
//std::cout << "Broadcaster: Frame Size: " << sect.frameSize << std::endl;
//two methods of sending RTMP data, _sendpacket and _write. Using sendpacket for greater control
RTMPPacket * packet;
unsigned char* buf = (unsigned char*)sect.payload;
int type = buf[0]&0x1f; //I believe &0x1f sets a 32bit limit
int len = sect.frameSize;
long timeOffset = GetTickCount() - rtmp_start_time;
//assign space packet will need
packet = (RTMPPacket *)malloc(sizeof(RTMPPacket)+RTMP_MAX_HEADER_SIZE + len + 9);
memset(packet, 0, sizeof(RTMPPacket) + RTMP_MAX_HEADER_SIZE);
packet->m_body = (char *)packet + sizeof(RTMPPacket) + RTMP_MAX_HEADER_SIZE;
packet->m_nBodySize = len + 9;
//std::cout << "Broadcaster: Packet Size: " << sizeof(RTMPPacket) + RTMP_MAX_HEADER_SIZE + len + 9 << std::endl;
//std::cout << "Broadcaster: Packet Body Size: " << len + 9 << std::endl;
//set body to point to the packetbody
unsigned char *body = (unsigned char *)packet->m_body;
memset(body, 0, len + 9);
//NAL_SLICE_IDR represents keyframe
//first element determines packet type
body[0] = 0x27;//inter-frame h.264
if (sect.type == NAL_SLICE_IDR) {
body[0] = 0x17; //h.264 codec id
}
//-------------------------------------------------------------------------------
//this section taken from https://stackoverflow.com/questions/25031759/using-x264-and-librtmp-to-send-live-camera-frame-but-the-flash-cant-show
//in an effort to understand packet format. it does not resolve my previous issues formatting the data for twitch to play it
//sets body to be NAL unit
body[1] = 0x01;
body[2] = 0x00;
body[3] = 0x00;
body[4] = 0x00;
//>> is a shift right
//shift len to the right, and AND it
/*body[5] = (len >> 24) & 0xff;
body[6] = (len >> 16) & 0xff;
body[7] = (len >> 8) & 0xff;
body[8] = (len) & 0xff;*/
//end code sourced from https://stackoverflow.com/questions/25031759/using-x264-and-librtmp-to-send-live-camera-frame-but-the-flash-cant-show
//-------------------------------------------------------------------------------
//copy from buffer into rest of body
memcpy(&body[9], buf, len);
//DEBUG
//save individual packet body to a file with name rtmp[packetnum]
//determine why some packets do not have 0x27 or 0x17 at the start
//still happening, makes no sense given the above code
/*std::string fileLocation = "rtmp" + std::to_string(packCount++);
std::cout << fileLocation << std::endl;
const char * charConversion = fileLocation.c_str();
FILE* saveFile = NULL;
saveFile = fopen(charConversion, "w+b");//open as write and binary
if (!fwrite(body, len + 9, 1, saveFile)) {
std::cout << "VideoEncoder: Error while trying to write to file" << std::endl;
}
fclose(saveFile);*/
//END DEBUG
//other packet details
packet->m_hasAbsTimestamp = 0;
packet->m_packetType = RTMP_PACKET_TYPE_VIDEO;
if (rtmp != NULL) {
packet->m_nInfoField2 = rtmp->m_stream_id;
}
packet->m_nChannel = 0x04;
packet->m_headerType = RTMP_PACKET_SIZE_LARGE;
packet->m_nTimeStamp = timeOffset;
//send the packet
if (rtmp != NULL) {
RTMP_SendPacket(rtmp, packet, TRUE);
}
I can see that Twitch is receiving the data in the inspector, at a steady 3kbps. so I'm sure something is wrong with how I'm adjusting the data before sending it. Can anyone advise me on what I'm doing wrong here?
The problems start before the code you included even. When you configure x264 be sure to set:
b_aud = 0;
b_repeat_headers = 0;
b_annexb = 0;
This will tell x264 to generate the format needed by rtmp, Then you can skip all the per-nal preprocessing.
For sps/pps use x264_encoder_headers to retrieve them after x264_encoder_open. Encode them into an "extradata" buffer as documented here Possible Locations for Sequence/Picture Parameter Set(s) for H.264 Stream. This extradata goes into an rtmp "sequence header" packet before any frames are sent. Set the frame the AVCPacketType accordingly body[1] in your case, 0 for sequence header 1 for everything else,
body[0] = 0x27;
body[1] = 0;
body[2] = 0;
body[3] = 0;
body[4] = 0;
memcpy(&body[5], extradata, extradata_size);
body[2] through body[4] MUST be set to the frame cts (pts - dts) if you have b frames. If you want to set it to zero, configure x264 for baseline profile, but this will result in reduced image quality. Use the return code from x264_encoder_encode as the frame size, and write the whole frame in one go.
int frame_size = x264_encoder_encode(encoder, &nals, &num_nals, &pic_in, &pic_out);
if(frame_size) {
int cts = pic_out->i_pts - pic_out->i_dts;
body[0] = pic_out->b_keyframe ? 0x27 : 0x17;
body[1] = 1;
body[2] = cts>>16;
body[3] = cts>>8;
body[4] = cts;
memcpy(&body[5], nals->p_payload, frame_size);
}
Finally, Twitch requires you also send an AAC audio stream. and be sure to set the keyframe interval to 2 seconds.
I want to read and open a video in encoded domain without decoding. I have written the code up to now and it works without errors. But the output of the method av_read_frame() just gives number of zeros and same negative integer value is repeating.
I'm not sure whether I passed the parameters correctly to the method. Please help.
void CFfmpegmethods::VideoRead(){
av_register_all();
const char *url = "H:\\Sanduni_projects\\ad_1.mp4";
AVDictionary *options = NULL;
AVFormatContext *s = avformat_alloc_context(); //NULL;
//AVFormatContext *avfmt = NULL;
//avformat_alloc_context();
AVPacket pkt;
//AVFormatContext *avformat_alloc_context();
//AVIOContext *avio_alloc_context();
//open an input stream and read the header
int ret = avformat_open_input(&s, url, NULL, NULL);
//avformat_find_stream_info(s, &options); //finding the missing information
if (ret < 0)
abort();
av_dict_set(&options, "video_size", "640x480", 0);
av_dict_set(&options, "pixel_format", "rgb24", 0);
if (avformat_open_input(&s, url, NULL, &options) < 0){
abort();
}
av_dict_free(&options);
AVDictionaryEntry *e;
if (e = av_dict_get(options, "", NULL, AV_DICT_IGNORE_SUFFIX)) {
fprintf(stderr, "Option %s not recognized by the demuxer.\n", e->key);
abort();
}
//int i = 0;
while (1){
//Split what is stored in the file into frames and return one for each call
//returns the next frame of the stream
int frame = av_read_frame(s, &pkt);
//cout <<i << " " << frame << endl;
waitKey(30);
//i++;
}
//make the packet free
av_packet_unref(&pkt);
//Close the file after reading
avformat_close_input(&s);
}
Method av_read_frame() output zeros while reading the packets and after that gives negative values. In my code loop runs infinitely. Therefore gives infinite number of negative values.
This is the modified code
while (1){
//Split what is stored in the file into frames and return one for each call
//returns the next frame of the stream
int frame = av_read_frame(s, pkt);
duration = pkt->duration;
size = pkt->size;
total_size = total_size + size;
total_duration = total_duration + duration;
i++;
if (frame < 0) break;
cout << "frame" << i << " " << size << " "<< duration << endl;
}
I have a data buffer which contains multiple compressed members, it could be deflate or zlib compressed member.
I found that zlib inflate call returns Z_STREAM_END after processing the first compressed block, Here multiple compressed member can be in any number(here in my example Its 3). But this data comes from other sides which doesn't communicated detail about number of compressed member in a data.
So how could I implement the use of zlib inflate functionality so that it could work over multiple compressed member ?
Following is a sample quick & dirty example in which I try to elaborate my problem.
This referred the case with zlib 1.2.5 library.
/* example.c -- understanding zlib inflate/decompression operation
*/
#define CHECK_ERR(err, msg) { \
if (err != Z_OK) { \
std::cerr << msg << " error: " << err << std::endl; \
exit(1); \
} \
}
/* ===========================================================================
* deflate() to create compressed data
*/
void test_deflate(std::vector<uint8_t> & input_data, std::vector<uint8_t>& compr)
{
z_stream c_stream; /* compression stream */
int err;
compr.clear();
c_stream.zalloc = (alloc_func)0;
c_stream.zfree = (free_func)0;
c_stream.opaque = (voidpf)0;
err = deflateInit(&c_stream, Z_DEFAULT_COMPRESSION);
CHECK_ERR(err, "deflateInit");
c_stream.next_in = &input_data[0];
c_stream.avail_in = input_data.size();
for (;;) {
uint8_t c_buffer[10] = {};
c_stream.next_out = &c_buffer[0];
c_stream.avail_out = 10;
err = deflate(&c_stream, Z_FINISH);
if (err == Z_STREAM_END)
{
for (int i = 0; i < (10 - c_stream.avail_out); i++)
compr.push_back(c_buffer[i]);
break;
}
CHECK_ERR(err, "deflate");
for (int i = 0; i < (10 - c_stream.avail_out); i++)
compr.push_back(c_buffer[i]);
}
std::cout << "Compressed data (size = " << std::dec << compr.size() << ") = ";
for (int i = 0; i < compr.size(); i++)
std::cout << (uint32_t) compr[i];
std::cout << std::endl;
err = deflateEnd(&c_stream);
CHECK_ERR(err, "deflateEnd");
}
/* ===========================================================================
* Test inflate()
*/
void test_inflate(std::vector<uint8_t> &compr,
std::vector<uint8_t> &uncompr)
{
int err;
z_stream d_stream; /* decompression stream */
uncompr.clear();
d_stream.zalloc = Z_NULL;
d_stream.zfree = Z_NULL;
d_stream.opaque = Z_NULL;
d_stream.avail_in = 0;
d_stream.next_in = Z_NULL;
err = inflateInit(&d_stream);
CHECK_ERR(err, "inflateInit");
d_stream.avail_in = compr.size();
d_stream.next_in = &compr[0];
for(;;) {
uint8_t d_buffer[10] = {};
d_stream.next_out = &d_buffer[0];
d_stream.avail_out = 10;
err = inflate(&d_stream, Z_NO_FLUSH);
if (err == Z_STREAM_END) {
for (int i = 0; i < (10 - d_stream.avail_out); i++)
uncompr.push_back(d_buffer[i]);
if (d_stream.avail_in == 0)
break;
}
CHECK_ERR(err, "inflate");
for (int i = 0; i < (10 - d_stream.avail_out); i++)
uncompr.push_back(d_buffer[i]);
}
err = inflateEnd(&d_stream);
CHECK_ERR(err, "inflateEnd");
std::cout << "Uncompressed data (size = " << std::dec << uncompr.size() << ") = ";
for (int i = 0; i < uncompr.size(); i++)
std::cout << (uint32_t) uncompr[i];
std::cout << std::endl;
}
/* ===========================================================================
* Usage: example
*/
int main(int argc, char **argv)
{
std::vector<uint8_t> input_data;
std::vector<uint8_t> compr, multiple_compr;
std::vector<uint8_t> uncompr;
std::cout << "Input Data (in hex) = ";
for (int i=0; i<32; i++) {
input_data.push_back((uint8_t)i);
if( i && (i % 2 == 0))
std::cout << " ";
std::cout << std::hex << (uint32_t)input_data[i];
}
std::cout << std::endl;
// create compressed buffer-1 from input data
test_deflate(input_data, compr);
// copy compressed buffer-1 data into multiple compressed member buffer
multiple_compr = compr;
compr.clear();
// create compressed buffer-2 from input data
test_deflate(input_data, compr);
// append data of compressed buffer-2 into multiple compressed member buffer
for(int i=0; i< compr.size(); i++)
{
multiple_compr.push_back(compr[i]);
}
// create decompressed output
test_inflate(multiple_compr, uncompr);
// compare decompressed data with input data
std::vector<uint8_t> final_data;
final_data.push_back(input_data);
final_data.push_back(input_data);
if (final_data == uncompr)
std::cout << "Matched" << std::endl;
else
std::cout << "Not Matched" << std::endl;
return 0;
}
1) Here second time inflate call returns error, But I wants it proceed successfully why it work like this ?
2) When I use Z_FINISH in the inflate call argument it returns with error, why can't I use Z_FINISH here ?
Kindly correct my example and suggest some optimized approach to do the same.
Simply repeat the inflate operation on the remaining data.
You can save some unnecessary free's and malloc's by using inflateReset() instead of inflateEnd() and inflateInit(). You may have some leftover data from the last inflate in next_in and avail_in, so use that first, and then reload.