I have a playbin2 pipeline playing a video file with multiple tracks. I can change the currently playing track by setting the current-video property:
current-video : Currently playing video stream (-1 = auto)
flags: readable, writable
Integer. Range: -1 - 2147483647 Default: -1
However, once I set this, the video hangs and then eventually turns gray. The audio keeps playing. A coworker suggested that flushing the pipeline might fix it, but the only way to do that seems to be seeking to the current position with GST_SEEK_FLAG_FLUSH. Using the seek does work, but it seems messy:
g_object_set(m_playBin, "current-video", video, NULL);
GstClockTime = getCurrentTime();
gst_element_seek(m_playBin, m_player->rate(),
GST_FORMAT_TIME,
(GstSeekFlags)(GST_SEEK_FLAG_FLUSH | GST_SEEK_FLAG_ACCURATE),
GST_SEEK_TYPE_SET, clockTime,
GST_SEEK_TYPE_NONE, GST_CLOCK_TIME_NONE)
Am I doing something wrong when I set the current-video? Is there a simpler way to flush the pipeline?
Note: I'm using an Ogg Theora file for testing since it was easy to build, so I'm not sure if this problem shows up in other formats.
Related
How should I validate pts/dts after demuxing and then after decoding?
For me it is significant to have valid pts all the time for days and
possibly weeks of continuous streaming.
After demuxing I check:
dts <= pts
prev_packet_dts < next_packet_pts
I also discard packets with AV_NOPTS_VALUE and wait for packets with
proper pts, because I don't know video duration at this case.
pts of packets can be not increasing because of I-P-B frames
Is it all right?
What about decoded AVFrames?
Should 'pts' be increasing all the time?
Why at some point 'pts' could lag behind 'dts'?
Why pict_type is a parameter of AVFrame? Should be at AVPacket, because
AVPacket is a compressed frame, not the opposite?
Ideally, yes. Unless if your format allows discontinuities, or wraps timestamps around due to overflow, like MPEG-TS.
Writing error.
It is an informational field, indicating the provenance of the frame. It can be used by filters or encoders, e.g. keyframe alignment during a re-encode.
At libav support I was advised to not rely on decoder output. It is more solid to produce pts/dts for encoding/muxing manually and I should search for ffmpeg tools sources to proper implementation. I will search for this approach.
For now I discard AVFrames only with AV_NOPTS_VALUE, and the rest of encoding/muxing works fine.
Validation of AVPackets after Demuxing remains the same, as described above.
Did anybody encounter the above? I was doing dynamic pipeline involving attach/removing a bin containing a queue and a multifilesink plugin. Was trying to record and split the video/audio file in 60 seconds segment. video/audio stream was of H264 video and G711 audio and muxed by matroskamux, which has the property streamable set to be true
Understand that multifilesink could only take streaming stream and there was no timing indexes. But that was fine with me and I could open the file to play them using VLC. However there is the problem of playing the first file produced
It seem to missing header or metadata. Have used mediaInfo to query them (as below)
first file video0.mp4 (cannot be played)
The 2nd-last file (can be played)
The properties i have set for multifilesink
g_object_set(G_OBJECT(multifilesink),
"aggregate-gops", TRUE,
"location", "video%d",
"max-file-duration", 60000000000,
"next-file", 5,
"index", 0,
"post-messages", TRUE,
NULL);
Have tried
adjusting "next-file" and "aggregate-gops", it does not seem to change the matters much
changing file format(cross-finger, though I do not believe it will help). It doesnt help
Have tried with a static pipeline... It seems that all the file can be played properly
Just wonder what did i do wrongly?
We are encoding sample frames using NvEncoder with HEVC. Since HEVC frames do not have any timestamps, in order to perform seek operation on the video, we wrote a remuxer in C++ that creates timestamps for the frames in an orderly fashion and writes the encoded frames in a video container (mp4,mov). The output video in mp4 container looks fine when played with ffplay, and timestamps seems correct when checked with ffprobe. However, when we try to play the video in Gstreamer pipeline, 2nd and 3rd frames seem to have the exactly same timestamp. So when the video is played, 3rd frame is skipped and 2nd frame is shown twice. We cannot tolerate any frame loss, so we need to solve this problem, which we think is due to an incompatibility between ffmpeg and gstreamer regarding frame timestamps. I can also provide the source-code of our remuxer and example outputs if that would help.
I used the following Gstreamer pipeline to play the mp4:
gst-launch-1.0 filesrc location=5_fps.mp4 ! qtdemux name=demux demux.video_0 ! queue! decodebin ! videoconvert ! videoscale ! autovideosink
Following command also gives the same mismatching frame timestamps:
ffmpeg -i 5_fps.bin -vcodec copy -acodec copy 5_fps.mp4
Many thanks!
Edit: I am adding the part of remuxer where each frame from the input stream is read and timestamps are added.
int frame_no=-1; //starting with -1 gives the same ffprobe results as command line ffmpeg container conversion, starting with 0 again causes the same timestamp problem
while (1) {
AVStream *in_stream, *out_stream;
_status = av_read_frame(_ifmt_ctx, &_pkt);
if (_status < 0) break;
in_stream = _ifmt_ctx->streams[_pkt.stream_index];
if (_pkt.stream_index >= _stream_mapping_size ||
_stream_mapping[_pkt.stream_index] < 0) {
av_packet_unref(&_pkt);
continue;
}
double inputFPS=av_q2d(in_stream->r_frame_rate);
double outputFPS=av_q2d(in_stream->r_frame_rate);
_pkt.stream_index = _stream_mapping[_pkt.stream_index];
out_stream = _ofmt_ctx->streams[_pkt.stream_index];
_pkt.pts=frame_no*in_stream->time_base.den/inputFPS;
_pkt.dts=_pkt.pts;
_pkt.duration = in_stream->time_base.den/inputFPS;
_pkt.pos = -1;
std::cout<<"rescaled pts: "<<_pkt.pts<<" dts: "<<_pkt.dts<<" frame no: "<< frame_no<<std::endl;
std::cout<<"input time_base den: "<<in_stream->time_base.den<<" output time_base den: "<<out_stream->time_base.den<<std::endl;
frame_no++;
_status = av_interleaved_write_frame(_ofmt_ctx, &_pkt);
if (_status < 0) {
cout<<"Error muxing packet\n";
break;
}
av_packet_unref(&_pkt);}
I first tried this method where each frame timestamp (pts and dts) is incremented by packet duration. At first I thought this method would not work since B-frames are decoded in different order, so I first tried videos with no b-frames. However, when I tried with videos with b-frames, it again worked. I thought decoded frames would be in a different order, however that was not the case. The only issue is that only the second and third frames appear to have the same timestamps in Gstream (not in Ffmpeg), other than these two frames remaining video plays just fine. Overall, I am also confused that b-frames do not cause any frame order problem.
Example encoded input, example output video if you want to examine the frames. (I don't know if it's okay to share files over google drive, please correct me if there is a better way to share, or not.)
As #AlanBirtles mentioned, assigning timestamps of b-frames in my naive way is not correct at all. I assumed that since the video was playable, timestamp are somehow corrected by Ffmpeg or Gstreamer, and I did not relate my main problem to this. However, when I tried to convert container of a video with no b-frames, the problem of 3rd frame being lost is solved. So I understand that either set timestamps of b-frames correctly, or I should use videos without b-frames. Even though it is not the viable solution, for the time being we will not use b-frames, however in the future I will try to reimplement the remuxer so that any video is remuxed OK.
I have a gstreamer pipeline created from the python gst bindings, which is set up to play a headset's microphone back to the headset's speaker. This works fine and is playing in a pipeline like this:
JackAudioSrc -> GstAudioMixer -> Queue -> GstJackAudioSink
Then many seconds later I want to play a short 10 second .wav file into the pipeline so the wav file is mixed with the microphone and heard on the headset. To do this, a GstFileSrc is dynamically added to the GstAudioMixer to mix in a short 10 second wav file to the headset's speaker, which gives pipeline like this:
GstJackAudioSrc -> GstAudioMixer -> Queue -> GstJackAudioSink
/
Gstfilesrc -> Gstwavparse ->/
When the Gstfilesrc and Gstwavparse file is dynamically added to a sink pad of the mixer, at a time say 6 seconds since the start of the pipeline, only the last 4 seconds of the wav is heard.
The problem seems to be that the wav file seeks to the time relative to when the pipeline started PLAYING.
I have tried changing "do-timestamp" in a multifilesrc, and GstIndentity "sync"=True, and can't find a way to set "live" on a filesrc, and many others but to no avail.
However, the whole 10 second wav file will play nicely if the pipeline is set to Gst.State.NULL then back to Gst.State.PLAYING when the filesrc is added at 6 seconds. This works as the pipeline time gets set back to zero, but this produces a click on the headset, which is unacceptable.
How can I ensure that the wav file starts playing from the start of the wav file, so that the whole 10 seconds is heard on the headset, if added to the pipeline at any random time?
An Update:
I can now get the timing of the wave file correct by adding a clocksync and setting its timestamp offset, before the wavparse:
nanosecs = pipeline.query_position(Gst.Format.TIME)[1]
clocksync.set_property("ts-offset", nanosecs)
Although the start/stop times are now correct, the wav audio is corrupted and heard as nothing but clicks and blips, but at least it starts playing at the correct time and finishes at the correct time. Note that without the clocksync the wav file audio is perfectly clear, it just starts and stops at the wrong time. So the ts-offset is somehow corrupting the audio.
Why is the audio being corrupted?
So I got this working and the answer is not to use the clocksync, but instead request a mixer sink pad, then call set_offset(nanosecs) on the mixer sink pad, before linking the wavparse to the mixer:
sink_pad = audio_mixer.get_request_pad("sink_%u")
nanosecs = pipeline.query_position(Gst.Format.TIME)[1]
sink_pad.set_offset(nanosecs)
sink_pad.add_probe(GstPadProbeType.IDLE, wav_callback)
def wav_callback(pad, pad_probe_info, userdata):
wavparse.link(audio_mixer)
wav_bin.set_state(Gst.State.PLAYING)
return Gst.PadProbeReturn.REMOVE
Then if the wav file needs to be rewound/replayed:
def replay_wav():
global wav_bin
global sink_pad
wav_bin.seek_simple(Gst.Format.TIME, Gst.SeekFlags.FLUSH, 0)
nanosecs = pipeline.query_position(Gst.Format.TIME)[1]
sink_pad.set_offset(nanosecs)
I am trying to create a webcam chat program in C++, and while I have been able to get the images to be captured sent and played, I am having trouble with doing the same with the audio: the audio lags and very quickly goes out of sync with the video, even when I just played it to myself.
I found this answer and sample code to be really useful.
Are there any modifications I can make to this code to get it to be nearly lag free, or is OpenAL not right for this? I am using Windows, but I plan on making a linux version later.
From the code linked:
ALCdevice* inputDevice = alcCaptureOpenDevice(NULL,FREQ,AL_FORMAT_MONO16,FREQ/2);
Try using a larger buffer:
ALCdevice* inputDevice = alcCaptureOpenDevice(NULL,FREQ,AL_FORMAT_MONO16,FREQ*4);
The polling is very aggressive. Try sleeping in the loop:
while (!done) {
...
}
To:
int sleepSeconds = 1;
while (!done) {
...
Sleep(sleepSeconds/10) //windows, miliseconds
//sleep(sleepSeconds) //linux, seconds
}