I am trying to create a webcam chat program in C++, and while I have been able to get the images to be captured sent and played, I am having trouble with doing the same with the audio: the audio lags and very quickly goes out of sync with the video, even when I just played it to myself.
I found this answer and sample code to be really useful.
Are there any modifications I can make to this code to get it to be nearly lag free, or is OpenAL not right for this? I am using Windows, but I plan on making a linux version later.
From the code linked:
ALCdevice* inputDevice = alcCaptureOpenDevice(NULL,FREQ,AL_FORMAT_MONO16,FREQ/2);
Try using a larger buffer:
ALCdevice* inputDevice = alcCaptureOpenDevice(NULL,FREQ,AL_FORMAT_MONO16,FREQ*4);
The polling is very aggressive. Try sleeping in the loop:
while (!done) {
...
}
To:
int sleepSeconds = 1;
while (!done) {
...
Sleep(sleepSeconds/10) //windows, miliseconds
//sleep(sleepSeconds) //linux, seconds
}
Related
I've been having a tough time getting my webcam working quickly with opencv. Frames take a very long time to read, (a recorded average of 124ms across 500 frames) I've tried on three different computers (running Windows 10) with a logitech C922 webcam. The most recent machine I tested on has a Ryzen 9 3950X, with 32gbs of ram; no lack of power.
Here is the code:
cv::VideoCapture cap = cv::VideoCapture(m_cameraNum);
// Check if camera opened successfully
if (!cap.isOpened())
{
m_logger->critical("Error opening video stream or file\n\r");
return -1;
}
bool result = true;
result &= cap.set(cv::CAP_PROP_FRAME_WIDTH, 1280);
result &= cap.set(cv::CAP_PROP_FRAME_HEIGHT, 720);
bool ready = false;
std::vector<string> timeLog;
timeLog.reserve(50000);
int i = 0;
while (i < 500)
{
auto start = std::chrono::system_clock::now();
cv::Mat img;
ready = cap.read(img);
// If the frame is empty, break immediately
if (!ready)
{
timeLog.push_back("continue");
continue;
}
i++;
auto end = std::chrono::system_clock::now();
timeLog.push_back(std::to_string(std::chrono::duration_cast<std::chrono::milliseconds>(end - start).count()));
}
for (auto& entry : timeLog)
m_logger->info(entry);
cap.release();
return 0;
Notice that I write the elapsed time to a log file at the end of execution. The average time is 124ms for debug and release, and not one instance of "continue" after half a dozen runs.
It doesn't matter if I use USB 2 or USB 3 ports (the camera is USB2) or if I run a debug build or a release build, the log file will show anywhere from 110ms to 130ms of time for each frame. The camera works fine in other app, OBS can get a smooth 1080#30fps or 720#60fps.
Stepping through the debugger and doing a lot of Googling, I've learned the following about my system:
The backend chosen by default is DSHOW. GStreamer and FFMPEG are also available.
DSHOW uses FFMPEG somehow (it needs the FFMPEG dll) but I cannot use FFMPEG directly through opencv. Attempting to use cv::VideoCapture(m_cameraNum, cv::CAP_FFMPEG) always fails. It seems like Opencv's interface to FFMPEG is only capable of opening video files.
Microsoft really screwed up camera devices in Windows a few years back, not sure if this is related to my problem.
Here's a short list of the fixes I have tried, most taken from older SO posts:
result &= cap.set(cv::CAP_PROP_FRAME_COUNT, 30); // Returns false, does nothing
result &= cap.set(cv::CAP_PROP_CONVERT_RGB, 0); // Returns true, does nothing
result &= cap.set(cv::CAP_PROP_MODE, cv::VideoWriter::fourcc('M', 'J', 'P', 'G')); // Returns false, does nothing
Set registry key from http://alax.info/blog/1693 that should disable the new Windows camera server.
Updated from 4.5.0 to 4.5.2, no change.
Asked device manager to find a newer driver, no newer driver found.
I'm out of ideas. Any help?
I am developing a clone of Asteroid in pure C++ and for that purpose, I need to add sounds to different events such as when a bullet is fired and when an explosion occurs. The issue however is that I don't have any experience with audio libraries.
I am using Simple DirectMedia Layer (SDL) and wrote a function named playsound() to play a sound in case a certain event occurs. The problem however is the fact that if an event occurs, playsound() is called and the code execution stops until the sound is wholly played out or until I return from the function (I delay the return using delay func).
What I would want to do is that the sound plays in the background without creating any lag for the rest of the Game. I am developing on Ubuntu 16.04 and can't use Windows PlaySound() either to call in the ASYNC flag.
Here is the function:
void playsound(string path) {
// Initialize SDL.
if (SDL_Init(SDL_INIT_AUDIO) < 0)
return;
// local variables
Uint32 wav_length; // length of our sample
Uint8 *wav_buffer; // buffer containing our audio file
SDL_AudioSpec wav_spec;
if(SDL_LoadWAV(path.c_str(), &wav_spec, &wav_buffer, &wav_length) == NULL){
return;
}
SDL_AudioDeviceID deviceId = SDL_OpenAudioDevice(NULL, 0, &wav_spec, NULL, 0);
SDL_QueueAudio(deviceId, wav_buffer, wav_length);
SDL_PauseAudioDevice(deviceId, 0);
SDL_Delay(50);
SDL_CloseAudioDevice(deviceId);
SDL_FreeWAV(wav_buffer);
SDL_Quit();
}
Your delay is stopping your code from executing, 50ms of delay is almost 2 frames at 33ms per frame or 3 frames at 16ms per frame, having a frame drop here and there might not be a problem, but you could see how calling several sounds in succession will slow your program down.
This is how I play sounds in my engine, using SDL2_mixer, (short sounds, for music you have another method called Mix_PlayMusic), it might be helpful to you. I have no lag (and I don't use any sleep or delays in my code). Once you call play() the sound should be played in full, unless there is something else pausing your code.
#pragma once
#include <string>
#include <memory>
#include <SDL2/SDL_mixer.h>
class sample {
public:
sample(const std::string &path, int volume);
void play();
void play(int times);
void set_volume(int volume);
private:
std::unique_ptr<Mix_Chunk, void (*)(Mix_Chunk *)> chunk;
};
And the cpp file
#include <sample.h>
sample::sample(const std::string &path, int volume)
: chunk(Mix_LoadWAV(path.c_str()), Mix_FreeChunk) {
if (!chunk.get()) {
// LOG("Couldn't load audio sample: ", path);
}
Mix_VolumeChunk(chunk.get(), volume);
}
// -1 here means we let SDL_mixer pick the first channel that is free
// If no channel is free it'll return an err code.
void sample::play() {
Mix_PlayChannel(-1, chunk.get(), 0);
}
void sample::play(int times) {
Mix_PlayChannel(-1, chunk.get(), times - 1);
}
void sample::set_volume(int volume) {
Mix_VolumeChunk(chunk.get(), volume);
}
Notice that I don't need to thread my model, every time something triggers a sound play the program keeps execution. (I guess SDL_Mixer plays in the main SDL thread).
For this to work, where you init SDL you'll also have to init the mixer as
if (Mix_OpenAudio(44100, MIX_DEFAULT_FORMAT, 2, 1024) < 0) {
// Error message if can't initialize
}
// Amount of channels (Max amount of sounds playing at the same time)
Mix_AllocateChannels(32);
And an example of how to play a sound would be
// at some point loaded a sample s with sample(path to wave mp3 or whatever)
s.play();
A few remarks, you don't need to use, but can, the code as it is, it is more of a simple example of using SDL2_mixer.
This mean functionality is lacking, you might want a tighter handling of sound, for example to stop a sound mid play (for some reason), you can do this if you play your sounds in different channels with the Mix_HaltChannel function, and the play() function could receive the channel where you want it to be played.
All these functions return error values, for example if no unreserved channel is available Mix_PlayChannel will return an error code.
Another thing you want to keep in mind is if you play the same sound several times it'll start to get blurry/you would not notice if the same sound is being played again. So you could add an integer to sample to count how many times a sample can be played.
If you REALLY want to thread your mixer/audio from the main SDL thread (and still only use SDL), you can just spawn a new SDL context in a thread and send in some way signals to play audio.
You want to load all necessary assets when initializing the game. Then, when you want to play them, they're loaded into the game memory and there will be no lags. And also play the sounds in a separate thread maybe, so it won't block your main thread.
There are several tools in C++ for asynchronous operations. You can try the most simple std::async:
auto handle = std::async(std::launch::async,
playsound, std::string{"/path/to/cute/sound"});
// Some other stuff. Your game logic doesn't blocked here.
handle.get(); // This can actually block.
You should specify flag std::launch::async, which means, that new thread will be used. Then name of callable needed to be executed and its parameters. Don't forget to include <future> header.
The following code is inside a thread and reads input data coming over usb. Approximately every 80 readings it misses one of the packets coming from an stm32 board. The board is programmed to send data packets to the android tablet every one second.
// Non Working Code
while(running){
int resp = bulktransfer(mInEp,mBuf,mBuf.lenght,1000);
if(resp>0){
dispatchMessage(mBuf);
}else if(resp<0)
showsBufferEmptyMessage();
}
I was looking the Missile Launcher example in android an other libraries on the internet and they put a delay of 50ms between each poll. Doing this it solves the missing package problem.
//Working code
while(running){
int resp = bulktransfer(mInEp,mBuf,mBuf.lenght,1000);
if(resp>0){
dispatchMessage(mBuf);
}else if(resp<0)
showsBufferEmptyMessage();
try{
Thread.sleep(50);
}catch(Exception e){}
}
Does anyone knows the reason why the delay works. Most of the libraries on github has this delay an as I mention before the google example too.
I am putting down my results regarding this problem. After all seems that the UsbConnection.bulkTransfer(...) method has some bug when called continuously. The solution was to use the asynchronous API, UsbRequest class. Using this method I could read from the input endpoint without delay and no data was lost during the whole stress test. So the direction to take is asynchronous UsbRequest instead of synchronously bulktransfer.
I have searched all around and can not find any examples or tutorials on how to access a webcam using ffmpeg in C++. Any sample code or any help pointing me to some documentation, would greatly be appreciated.
Thanks in advance.
I have been working on this for months now. Your first "issue" is that ffmpeg (libavcodec and other ffmpeg libs) does NOT access web cams, or any other device.
For a basic USB webcam, or audio/video capture card, you first need driver software to access that device. For linux, these drivers fall under the Video4Linux (V4L2 as it is known) category, which are modules that are part of most distros. If you are working with MS Windows, then you need to get an SDK that allows you to access the device. MS may have something for accessing generic devices, (but from my experience, they are not very capable, if they work at all) If you've made it this far, then you now have raw frames (video and/or audio).
THEN you get to the ffmpeg part - libavcodec - which takes the raw frames (audio and/or video) and encodes them into a streams, which ffmpeg can then mux into your final container.
I have searched, but have found very few examples of all of these, and most are piece-meal.
If you don't need to actually code of this yourself, the command line ffmpeg, as well as vlc, can access these devices, capture and save to files, and even stream.
That's the best I can do for now.
ken
For windows use dshow
For Linux (like ubuntu) use Video4Linux (V4L2).
FFmpeg can take input from V4l2 and can do the process.
To find the USB video path type : ls /dev/video*
E.g : /dev/video(n) where n = 0 / 1 / 2 ….
AVInputFormat – Struct which holds the information about input device format / media device format.
av_find_input_format ( “v4l2”) [linux]
av_format_open_input(AVFormatContext , “/dev/video(n)” , AVInputFormat , NULL)
if return value is != 0 then error.
Now you have accessed the camera using FFmpeg and can continue the operation.
sample code is below.
int CaptureCam()
{
avdevice_register_all(); // for device
avcodec_register_all();
av_register_all();
char *dev_name = "/dev/video0"; // here mine is video0 , it may vary.
AVInputFormat *inputFormat =av_find_input_format("v4l2");
AVDictionary *options = NULL;
av_dict_set(&options, "framerate", "20", 0);
AVFormatContext *pAVFormatContext = NULL;
// check video source
if(avformat_open_input(&pAVFormatContext, dev_name, inputFormat, NULL) != 0)
{
cout<<"\nOops, could'nt open video source\n\n";
return -1;
}
else
{
cout<<"\n Success !";
}
} // end function
Note : Header file < libavdevice/avdevice.h > must be included
This really doesn't answer the question as I don't have a pure ffmpeg solution for you, However, I personally use Qt for webcam access. It is C++ and will have a much better API for accomplishing this. It does add a very large dependency on your code however.
It definitely depends on the webcam - for example, at work we use IP cameras that deliver a stream of jpeg data over the network. USB will be different.
You can look at the DirectShow samples, eg PlayCap (but they show AmCap and DVCap samples too). Once you have a directshow input device (chances are whatever device you have will be providing this natively) you can hook it up to ffmpeg via the dshow input device.
And having spent 5 minutes browsing the ffmpeg site to get those links, I see this...
I am writing client-server system that uses FFMPEG library to parse H.264 stream into NAL units on the server side, then uses channel coding to send them over network to client side, where my application must be able to play video.
The question is how to play received AVPackets (NAL units) in my application as video stream.
I have found this tutorial helpful and used it as base for both server and client side.
Some sample code or resource related to playing video not from file, but from data inside program using FFMPEG library would be very helpful.
I am sure that received information will be sufficient to play video, because I tried to save received data as .h264 or .mp4 file and it can be played by VLC player.
Of what I understand from your question, you have the AVPackets and want to play a video. In reality this is two problems; 1. decoding your packets, and 2. playing the video.
For decoding your packets, with FFmpeg, you should take a look at the documentation for AVPacket, AVCodecContext and avcodec_decode_video2 to get some ideas; the general idea is that you want to do something (just wrote this in the browser, take with a grain of salt) along the lines of:
//the context, set this appropriately based on your video. See the above links for the documentation
AVCodecContext *decoder_context;
std::vector<AVPacket> packets; //assume this has your packets
...
AVFrame *decoded_frame = av_frame_alloc();
int ret = -1;
int got_frame = 0;
for(AVPacket packet : packets)
{
avcodec_get_frame_defaults(frame);
ret = avcodec_decode_video2(decoder_context, decoded_frame, &got_frame, &packet);
if (ret <= 0) {
//had an error decoding the current packet or couldn't decode the packet
break;
}
if(got_frame)
{
//send to whatever video player queue you're using/do whatever with the frame
...
}
got_frame = 0;
av_free_packet(&packet);
}
It's a pretty rough sketch, but that's the general idea for your problem of decoding the AVPackets. As for your problem of playing the video, you have many options, which will likely depend more on your clients. What you're asking is a pretty large problem, I'd advise familiarizing yourself with the FFmpeg documentation and the provided examples at the FFmpeg site. Hope that makes sense