I have made a soft synthesizer in Visual Studio 2012 with C++, MFC and DirectX. Despite having added code to rapidly fade out the sound I am experiencing popping / clicking when stopping playback (also when starting).
I copied the DirectX code from this project: http://www.codeproject.com/Articles/7474/Sound-Generator-How-to-create-alien-sounds-using-m
I'm not sure if I'm allowed to cut and paste all the code from the Code Project. Basically I use the Player class from that project as is, the instance of this class is called m_player in my code. The Stop member function in that class calls the Stop function of LPDIRECTSOUNDBUFFER:
void Player::Stop()
{
DWORD status;
if (m_lpDSBuffer == NULL)
return;
HRESULT hres = m_lpDSBuffer->GetStatus(&status);
if (FAILED(hres))
EXCEP(DirectSoundErr::GetErrDesc(hres), "Player::Stop GetStatus");
if ((status & DSBSTATUS_PLAYING) == DSBSTATUS_PLAYING)
{
hres = m_lpDSBuffer->Stop();
if (FAILED(hres))
EXCEP(DirectSoundErr::GetErrDesc(hres), "Player::Stop Stop");
}
}
Here is the notification code (with some supporting code) in my project that fills the sound buffer. Note that the rend function always returns a double between -1 to 1, m_ev_smps = 441, m_n_evs = 3 and m_ev_sz = 882. subInit is called from OnInitDialog:
#define FD_STEP 0.0005
#define SC_NOT_PLYD 0
#define SC_PLYNG 1
#define SC_FD_OUT 2
#define SC_FD_IN 3
#define SC_STPNG 4
#define SC_STPD 5
bool CMainDlg::subInit()
// initialises various variables and the sound player
{
Player *pPlayer;
SOUNDFORMAT format;
std::vector<DWORD> events;
int t, buf_sz;
try
{
pPlayer = new Player();
pPlayer->SetHWnd(m_hWnd);
m_player = pPlayer;
m_player->Init();
format.NbBitsPerSample = 16;
format.NbChannels = 1;
format.SamplingRate = 44100;
m_ev_smps = 441;
m_n_evs = 3;
m_smps = new short[m_ev_smps];
m_smp_scale = (int)pow(2, format.NbBitsPerSample - 1);
m_max_tm = (int)((double)m_ev_smps / (double)(format.SamplingRate * 1000));
m_ev_sz = m_ev_smps * format.NbBitsPerSample/8;
buf_sz = m_ev_sz * m_n_evs;
m_player->CreateSoundBuffer(format, buf_sz, 0);
m_player->SetSoundEventListener(this);
for(t = 0; t < m_n_evs; t++)
events.push_back((int)((t + 1)*m_ev_sz - m_ev_sz * 0.95));
m_player->CreateEventReadNotification(events);
m_status = SC_NOT_PLYD;
}
catch(MATExceptions &e)
{
MessageBox(e.getAllExceptionStr().c_str(), "Error initializing the sound player");
EndDialog(IDCANCEL);
return FALSE;
}
return TRUE;
}
void CMainDlg::Stop()
// stop playing
{
m_player->Stop();
m_status = SC_STPD;
}
void CMainDlg::OnBnClickedStop()
// causes fade out
{
m_status = SC_FD_OUT;
}
void CMainDlg::OnSoundPlayerNotify(int ev_num)
// render some sound samples and check for errors
{
ScopeGuardMutex guard(&m_mutex);
int s, end, begin, elapsed;
if (m_status != SC_STPNG)
{
begin = GetTickCount();
try
{
for(s = 0; s < m_ev_smps; s++)
{
m_smps[s] = (int)(m_synth->rend() * 32768 * m_fade);
if (m_status == SC_FD_IN)
{
m_fade += FD_STEP;
if (m_fade > 1)
{
m_fade = 1;
m_status = SC_PLYNG;
}
}
else if (m_status == SC_FD_OUT)
{
m_fade -= FD_STEP;
if (m_fade < 0)
{
m_fade = 0;
m_status = SC_STPNG;
}
}
}
}
catch(MATExceptions &e)
{
OutputDebugString(e.getAllExceptionStr().c_str());
}
try
{
m_player->Write(((ev_num + 1) % m_n_evs)*m_ev_sz, (unsigned char*)m_smps, m_ev_sz);
}
catch(MATExceptions &e)
{
OutputDebugString(e.getAllExceptionStr().c_str());
}
end = GetTickCount();
elapsed = end - begin;
if(elapsed > m_max_tm)
m_warn_msg.Format(_T("Warning! compute time: %dms"), elapsed);
else
m_warn_msg.Format(_T("compute time: %dms"), elapsed);
}
if (m_status == SC_STPNG)
Stop();
}
It seems like the buffer is not always sounding out when the stop button is clicked. I don't have any specific code for waiting for the sound buffer to finish playing before the DirectX Stop is called. Other than that the sound playback is working just fine, so at least I am initialising the player correctly and notification code is working in that respect.
Try replacing 32768 with 32767. Not by any means sure this is your issue, but it could overflow the positive short int range (assuming your audio is 16-bit) and cause a "pop".
I got rid of the pops / clicks when stopping playback, by filling the buffer with zeros after the fade out. However I still get pops when re-starting playback, despite filling with zeros and then fading back in (it is frustrating).
Related
I'm writing a little Console-Game-Engine and for better performance I wanted 2 threads (or more but 2 for this task) using two buffers. One thread is drawing the next frame in the first buffer while the other thread is reading the current frame from the second buffer. Then the buffers get swapped.
Of cause I can only swap them if both threads finished their task and the drawing/writing thread happened to be the one waiting. But the time it is waiting systematicly switches more or less between two values, here a few of the messurements I made (in microseconds):
0, 36968, 0, 36260, 0, 35762, 0, 38069, 0, 36584, 0, 36503
It's pretty obvious that this is not a coincidence but I wasn't able to figure out what the problem was as this is the first time I'm using threads.
Here the code, ask for more if you need it, I think it's too much to post it all:
header-file (Manager currently only adds a pointer to my WinAppBase-class):
class SwapChain : Manager
{
WORD *pScreenBuffer1, *pScreenBuffer2, *pWritePtr, *pReadPtr, *pTemp;
bool isRunning, writingFinished, readingFinished, initialized;
std::mutex lockWriting, lockReading;
std::condition_variable cvWriting, cvReading;
DWORD charsWritten;
COORD startPosition;
int screenBufferWidth;
// THREADS (USES NORMAL THREAD AS SECOND THREAD)
void ReadingThread();
// THIS FUNCTION IS ONLY FOR INTERN USE
void SwapBuffers();
public:
// USE THESE TO CONTROL WHEN THE BUFFERS GET SWAPPED
void BeginDraw();
void EndDraw();
// PUT PIXEL | INLINED FOR BETTER PERFORMANCE
inline void PutPixel(short xPos, short yPos, WORD color)
{
this->pWritePtr[(xPos * 2) + yPos * screenBufferWidth] = color;
this->pWritePtr[(xPos * 2) + yPos * screenBufferWidth + 1] = color;
}
// GENERAL CONTROL OVER SWAP CHAIN
void Initialize();
void Run();
void Stop();
// CONSTRUCTORS
SwapChain(WinAppBase * pAppBase);
virtual ~SwapChain();
};
Cpp-file
SwapChain::SwapChain(WinAppBase * pAppBase)
:
Manager(pAppBase)
{
this->isRunning = false;
this->initialized = false;
this->pReadPtr = NULL;
this->pScreenBuffer1 = NULL;
this->pScreenBuffer2 = NULL;
this->pWritePtr = NULL;
this->pTemp = NULL;
this->charsWritten = 0;
this->startPosition = { 0, 0 };
this->readingFinished = 0;
this->writingFinished = 0;
this->screenBufferWidth = this->pAppBase->screenBufferInfo.dwSize.X;
}
SwapChain::~SwapChain()
{
this->Stop();
if (_CrtIsValidHeapPointer(pReadPtr))
delete[] pReadPtr;
if (_CrtIsValidHeapPointer(pScreenBuffer1))
delete[] pScreenBuffer1;
if (_CrtIsValidHeapPointer(pScreenBuffer2))
delete[] pScreenBuffer2;
if (_CrtIsValidHeapPointer(pWritePtr))
delete[] pWritePtr;
}
void SwapChain::ReadingThread()
{
while (this->isRunning)
{
this->readingFinished = 0;
WriteConsoleOutputAttribute(
this->pAppBase->consoleCursor,
this->pReadPtr,
this->pAppBase->screenBufferSize,
this->startPosition,
&this->charsWritten
);
memset(this->pReadPtr, 0, this->pAppBase->screenBufferSize);
this->readingFinished = true;
this->cvWriting.notify_all();
if (!this->writingFinished)
{
std::unique_lock<std::mutex> lock(this->lockReading);
this->cvReading.wait(lock);
}
}
}
void SwapChain::SwapBuffers()
{
this->pTemp = this->pReadPtr;
this->pReadPtr = this->pWritePtr;
this->pWritePtr = this->pTemp;
this->pTemp = NULL;
}
void SwapChain::BeginDraw()
{
this->writingFinished = false;
}
void SwapChain::EndDraw()
{
TimePoint tpx1, tpx2;
tpx1 = Clock::now();
if (!this->readingFinished)
{
std::unique_lock<std::mutex> lock2(this->lockWriting);
this->cvWriting.wait(lock2);
}
tpx2 = Clock::now();
POST_DEBUG_MESSAGE(std::chrono::duration_cast<std::chrono::microseconds>(tpx2 - tpx1).count(), "EndDraw wating time");
SwapBuffers();
this->writingFinished = true;
this->cvReading.notify_all();
}
void SwapChain::Initialize()
{
if (this->initialized)
{
POST_DEBUG_MESSAGE(Result::CUSTOM, "multiple initialization");
return;
}
this->pScreenBuffer1 = (WORD *)malloc(sizeof(WORD) * this->pAppBase->screenBufferSize);
this->pScreenBuffer2 = (WORD *)malloc(sizeof(WORD) * this->pAppBase->screenBufferSize);
for (int i = 0; i < this->pAppBase->screenBufferSize; i++)
{
this->pScreenBuffer1[i] = 0x0000;
}
for (int i = 0; i < this->pAppBase->screenBufferSize; i++)
{
this->pScreenBuffer2[i] = 0x0000;
}
this->pWritePtr = pScreenBuffer1;
this->pReadPtr = pScreenBuffer2;
this->initialized = true;
}
void SwapChain::Run()
{
this->isRunning = true;
std::thread t1(&SwapChain::ReadingThread, this);
t1.detach();
}
void SwapChain::Stop()
{
this->isRunning = false;
}
This is where I run the SwapChain-class from:
void Application::Run()
{
this->engine.graphicsmanager.swapChain.Initialize();
Sprite<16, 16> sprite(&this->engine);
sprite.LoadSprite("engine/resources/TestData.xml", "root.test.sprites.baum");
this->engine.graphicsmanager.swapChain.Run();
int a, b, c;
for (int i = 0; i < 60; i++)
{
this->engine.graphicsmanager.swapChain.BeginDraw();
for (c = 0; c < 20; c++)
{
for (a = 0; a < 19; a++)
{
for (b = 0; b < 10; b++)
{
sprite.Print(a * 16, b * 16);
}
}
}
this->engine.graphicsmanager.swapChain.EndDraw();
}
this->engine.graphicsmanager.swapChain.Stop();
_getch();
}
The for-loops above simply draw the sprite 20 times from the top-left corner to the bottom-right corner of the console - the buffers don't get swapped during that, and that again for a total of 60 times (so the buffers get swapped 60 times).
sprite.Print uses the PutPixel function of SwapChain.
Here the WinAppBase (which consits more or less of global-like variables)
class WinAppBase
{
public:
// SCREENBUFFER
CONSOLE_SCREEN_BUFFER_INFO screenBufferInfo;
long screenBufferSize;
// CONSOLE
DWORD consoleMode;
HWND consoleWindow;
HANDLE consoleCursor;
HANDLE consoleInputHandle;
HANDLE consoleHandle;
CONSOLE_CURSOR_INFO consoleCursorInfo;
RECT consoleRect;
COORD consoleSize;
// FONT
CONSOLE_FONT_INFOEX fontInfo;
// MEMORY
char * pUserAccessDataPath;
public:
void reload();
WinAppBase();
virtual ~WinAppBase();
};
There are no errors, simply this alternating waitng time.
Maybe you'd like to start by looking if I did the synchronisation of the threads correctly? I'm not exactly sure how to use a mutex or condition-variables so it might comes from that.
Apart from that it is working fine, the sprites are shown as they should.
The clock you are using may have limited resolution. Here is a random example of a clock provided by Microsoft with 15 ms (15000 microsecond) resolution: Why are .NET timers limited to 15 ms resolution?
If one thread is often waiting for the other, it is entirely possible (assuming the above clock resolution) that it sometimes waits two clockticks and sometimes none. Maybe your clock only has 30 ms resolution. We really can't tell from the code. Do you get more precise measurements elsewhere with this clock?
There are also other systems in play such as the OS scheduler or whatever controls your std::threads. That one is (hopefully) much more granular, but how all these interactions play out doesn't have to be obvious or intuitive.
I'm using glQuery for getting info about FPS count in my application:
CollectDataBegin();
/*all drawing operations with OpenGL*/
CollectDataEnd();
Where:
void RenderingInfo::CollectDataBegin()
{
//FPS begin
available = 0;
GLenum eError;
// UPDATE 1 START
if (!bQueryGenerated){
glGenQueries(1, queries);
bQueryGenerated = true;
}
// UPDATE 1 END
//GL_NO_ERROR from glGetError();
glBeginQuery(GL_TIME_ELAPSED, queries[0]);
//GL_OUT_OF_MEMORY error from glGetError();
//FPS end
}
void RenderingInfo::CollectDataEnd()
{
//FPS begin
glEndQuery(GL_TIME_ELAPSED);
iFramesCount++;
if (iFramesCount == 20)
{
iFramesCount = 0;
while (!available) {
glGetQueryObjectiv(queries[0], GL_QUERY_RESULT_AVAILABLE, &available);
}
glGetQueryObjectui64v(queries[0], GL_QUERY_RESULT, &timeElapsed);
float jeb = static_cast<float>(timeElapsed) / std::pow(10, 9);
xRenderStats.fFPS = static_cast<float>(1.0 / jeb);
sFPS = std::to_string(xRenderStats.fFPS);
// UPDATE 1 START
if (bQueryGenerated){
glDeleteQueries(1, queries);
bQueryGenerated = false;
}
// UPDATE 1 END
}
}
Private members of RenderingInfo class:
GLuint queries[] = {0};
GLint available = 0;
GLuint64 timeElapsed;
int iFramesCount = 0;
bool bQueryGenerated = false; //UPADTE 1
I wrote and tested this code using Nvidia GeForce GTX760 with newest drivers and I had no problems at all.
But after switching to my integrated Intel HD Graphics 4600 I'm recieving GL_OUT_OF_MEMORY after calling glBeginQuery(). Interesting thing is that I'm not getting this error right away but after making some calls to glBeginQuery().
I wasn't been able to find any posts related to this matter so I'm asking for Your help is solving this issue.
UPDATE 1:
I modified my code accoring to #Ike advices, but I'm still recieving an GL_OUT_OF_MEMORY error.
After removing the code associated with glQuery my app no longer produces GL_OUT_OF_MEMORY errors. Since I was using this funcionality for counting time in which single frame is rendered, I replaced it with more reliable method:
void RenderingInfo::CollectDataBegin()
{
//FPS begin
ctTimeBegin = clock();
//FPS end
}
void RenderingInfo::CollectDataEnd()
{
//FPS begin
ctTimeEnd = clock();
dElapsedTime += (static_cast<double>((ctTimeEnd - ctTimeBegin))/CLOCKS_PER_SEC);
iFramesCount++;
if (iFramesCount == 20)
{
if ((dElapsedTime / iFramesCount) < (1.0 / CLOCKS_PER_SEC)){
xRenderStats.fFPS = 60.0f;
}
else{
xRenderStats.fFPS = static_cast<float>(iFramesCount / dElapsedTime);
}
sFPS = std::to_string(xRenderStats.fFPS);
dElapsedTime = 0.0;
iFramesCount = 0;
}
//FPS end
}
Private RenderingInfo members:
int iFramesCount =0;
clock_t ctTimeBegin = 0;
clock_t ctTimeEnd = 0;
double dElapsedTime = 0.0;
This is not the answer for a question why I received GL_OUT_OF_MEMORY, but it's for showing a possible way out when someone will end up with the similar problem.
I wrote a little debugger for analysing and looging certain problems. Now I implemented a hardwarebreakpoint for detecting the access of a memory address being overwritten. When I run my debugger with a test process, then everything works fine. When I access the address, the breakpoint fires and the callstack is logged. The problem is, when I run the same against an application running multiple threads. I'm replicating the breakpoint into every thread that gets created and also the main thread. None of the functions report an error and everything looks fine, but when the address is accessed, the breakpoint never fires.
So I wonder if there is some documentation where this is described or if there are additionaly things that I have to do in case of a multithreaded application.
The function to set the breakpoint is this:
#ifndef _HARDWARE_BREAKPOINT_H
#define _HARDWARE_BREAKPOINT_H
#include "breakpoint.h"
#define MAX_HARDWARE_BREAKPOINT 4
#define REG_DR0_BIT 1
#define REG_DR1_BIT 4
#define REG_DR2_BIT 16
#define REG_DR3_BIT 64
class HardwareBreakpoint : public Breakpoint
{
public:
typedef enum
{
REG_INVALID = -1,
REG_DR0 = 0,
REG_DR1 = 1,
REG_DR2 = 2,
REG_DR3 = 3
} Register;
typedef enum
{
CODE,
READWRITE,
WRITE,
} Type;
typedef enum
{
SIZE_1,
SIZE_2,
SIZE_4,
SIZE_8,
} Size;
typedef struct
{
void *pAddress;
bool bBusy;
Type nType;
Size nSize;
Register nRegister;
} Info;
public:
HardwareBreakpoint(HANDLE hThread);
virtual ~HardwareBreakpoint(void);
/**
* Sets a hardware breakpoint. If no register is free or an error occured
* REG_INVALID is returned, otherwise the hardware register for the given breakpoint.
*/
HardwareBreakpoint::Register set(void *pAddress, Type nType, Size nSize);
void remove(void *pAddress);
void remove(Register nRegister);
inline Info const *getInfo(Register nRegister) const { return &mBreakpoint[nRegister]; }
private:
typedef Breakpoint super;
private:
Info mBreakpoint[MAX_HARDWARE_BREAKPOINT];
size_t mRegBit[MAX_HARDWARE_BREAKPOINT];
size_t mRegOffset[MAX_HARDWARE_BREAKPOINT];
};
#endif // _HARDWARE_BREAKPOINT_H
void SetBits(DWORD_PTR &dw, size_t lowBit, size_t bits, size_t newValue)
{
DWORD_PTR mask = (1 << bits) - 1;
dw = (dw & ~(mask << lowBit)) | (newValue << lowBit);
}
HardwareBreakpoint::HardwareBreakpoint(HANDLE hThread)
: super(hThread)
{
mRegBit[REG_DR0] = REG_DR0_BIT;
mRegBit[REG_DR1] = REG_DR1_BIT;
mRegBit[REG_DR2] = REG_DR2_BIT;
mRegBit[REG_DR3] = REG_DR3_BIT;
CONTEXT ct;
mRegOffset[REG_DR0] = reinterpret_cast<size_t>(&ct.Dr0) - reinterpret_cast<size_t>(&ct);
mRegOffset[REG_DR1] = reinterpret_cast<size_t>(&ct.Dr1) - reinterpret_cast<size_t>(&ct);
mRegOffset[REG_DR2] = reinterpret_cast<size_t>(&ct.Dr2) - reinterpret_cast<size_t>(&ct);
mRegOffset[REG_DR3] = reinterpret_cast<size_t>(&ct.Dr3) - reinterpret_cast<size_t>(&ct);
memset(&mBreakpoint[0], 0, sizeof(mBreakpoint));
for(int i = 0; i < MAX_HARDWARE_BREAKPOINT; i++)
mBreakpoint[i].nRegister = (Register)i;
}
HardwareBreakpoint::Register HardwareBreakpoint::set(void *pAddress, Type nType, Size nSize)
{
CONTEXT ct = {0};
super::setAddress(pAddress);
ct.ContextFlags = CONTEXT_DEBUG_REGISTERS;
if(!GetThreadContext(getThread(), &ct))
return HardwareBreakpoint::REG_INVALID;
size_t iReg = 0;
for(int i = 0; i < MAX_HARDWARE_BREAKPOINT; i++)
{
if (ct.Dr7 & mRegBit[i])
mBreakpoint[i].bBusy = true;
else
mBreakpoint[i].bBusy = false;
}
Info *reg = NULL;
// Address already used?
for(int i = 0; i < MAX_HARDWARE_BREAKPOINT; i++)
{
if(mBreakpoint[i].pAddress == pAddress)
{
iReg = i;
reg = &mBreakpoint[i];
break;
}
}
if(reg == NULL)
{
for(int i = 0; i < MAX_HARDWARE_BREAKPOINT; i++)
{
if(!mBreakpoint[i].bBusy)
{
iReg = i;
reg = &mBreakpoint[i];
break;
}
}
}
// No free register available
if(!reg)
return HardwareBreakpoint::REG_INVALID;
*(void **)(((char *)&ct)+mRegOffset[iReg]) = pAddress;
reg->bBusy = true;
ct.Dr6 = 0;
int st = 0;
if (nType == CODE)
st = 0;
if (nType == READWRITE)
st = 3;
if (nType == WRITE)
st = 1;
int le = 0;
if (nSize == SIZE_1)
le = 0;
else if (nSize == SIZE_2)
le = 1;
else if (nSize == SIZE_4)
le = 3;
else if (nSize == SIZE_8)
le = 2;
SetBits(ct.Dr7, 16 + iReg*4, 2, st);
SetBits(ct.Dr7, 18 + iReg*4, 2, le);
SetBits(ct.Dr7, iReg*2, 1, 1);
ct.ContextFlags = CONTEXT_DEBUG_REGISTERS;
if(!SetThreadContext(getThread(), &ct))
return REG_INVALID;
return reg->nRegister;
}
I'm setting the breakpoint in the main debugger loop whenever a new thread is created CREATE_THREAD_DEBUG_EVENT but looking at the sourcecode of GDB it seems not to be done there, so maybe that is to early?
So I finally found the answer to this problem.
In the debug event loop, I'm monitoring the events that windows sends me. One of those events is CREATE_THREAD_DEBUG_EVENT which I used to set the hardware breakpoint whenever a new thread was created.
The problem is, that the notification of this event comes before the thread got actually started. So Windows is setting the context for the first time AFTER this event is sent, which of course overwrites any context data that I have set before.
The solution I implemented now is, when a CREATE_THREAD_DEBUG_EVENT comes I put a software breakpoint at the start adress of the thread, so that the first instruction is my breakpoint. When I receive the breakpoint event, I restore the original code and install the hardware breakpoint, which now fires fine.
If there is a better solution, I'm all ears. :)
I've been trying to write a class that derives from FramedSource in Live555 that will allow me to stream live data from my D3D9 application to an MP4 or similar.
What I do each frame is grab the backbuffer into system memory as a texture, then convert it from RGB -> YUV420P, then encode it using x264, then ideally pass the NAL packets on to Live555. I made a class called H264FramedSource that derived from FramedSource basically by copying the DeviceSource file. Instead of the input being an input file, I've made it a NAL packet which I update each frame.
I'm quite new to codecs and streaming, so I could be doing everything completely wrong. In each doGetNextFrame() should I be grabbing the NAL packet and doing something like
memcpy(fTo, nal->p_payload, nal->i_payload)
I assume that the payload is my frame data in bytes? If anybody has an example of a class they derived from FramedSource that might at least be close to what I'm trying to do I would love to see it, this is all new to me and a little tricky to figure out what's happening. Live555's documentation is pretty much the code itself which doesn't exactly make it easy for me to figure out.
Ok, I finally got some time to spend on this and got it working! I'm sure there are others who will be begging to know how to do it so here it is.
You will need your own FramedSource to take each frame, encode, and prepare it for streaming, I will provide some of the source code for this soon.
Essentially throw your FramedSource into the H264VideoStreamDiscreteFramer, then throw this into the H264RTPSink. Something like this
scheduler = BasicTaskScheduler::createNew();
env = BasicUsageEnvironment::createNew(*scheduler);
framedSource = H264FramedSource::createNew(*env, 0,0);
h264VideoStreamDiscreteFramer
= H264VideoStreamDiscreteFramer::createNew(*env, framedSource);
// initialise the RTP Sink stuff here, look at
// testH264VideoStreamer.cpp to find out how
videoSink->startPlaying(*h264VideoStreamDiscreteFramer, NULL, videoSink);
env->taskScheduler().doEventLoop();
Now in your main render loop, throw over your backbuffer which you've saved to system memory to your FramedSource so it can be encoded etc. For more info on how to setup the encoding stuff check out this answer How does one encode a series of images into H264 using the x264 C API?
My implementation is very much in a hacky state and is yet to be optimised at all, my d3d application runs at around 15fps due to the encoding, ouch, so I will have to look into this. But for all intents and purposes this StackOverflow question is answered because I was mostly after how to stream it. I hope this helps other people.
As for my FramedSource it looks a little something like this
concurrent_queue<x264_nal_t> m_queue;
SwsContext* convertCtx;
x264_param_t param;
x264_t* encoder;
x264_picture_t pic_in, pic_out;
EventTriggerId H264FramedSource::eventTriggerId = 0;
unsigned H264FramedSource::FrameSize = 0;
unsigned H264FramedSource::referenceCount = 0;
int W = 720;
int H = 960;
H264FramedSource* H264FramedSource::createNew(UsageEnvironment& env,
unsigned preferredFrameSize,
unsigned playTimePerFrame)
{
return new H264FramedSource(env, preferredFrameSize, playTimePerFrame);
}
H264FramedSource::H264FramedSource(UsageEnvironment& env,
unsigned preferredFrameSize,
unsigned playTimePerFrame)
: FramedSource(env),
fPreferredFrameSize(fMaxSize),
fPlayTimePerFrame(playTimePerFrame),
fLastPlayTime(0),
fCurIndex(0)
{
if (referenceCount == 0)
{
}
++referenceCount;
x264_param_default_preset(¶m, "veryfast", "zerolatency");
param.i_threads = 1;
param.i_width = 720;
param.i_height = 960;
param.i_fps_num = 60;
param.i_fps_den = 1;
// Intra refres:
param.i_keyint_max = 60;
param.b_intra_refresh = 1;
//Rate control:
param.rc.i_rc_method = X264_RC_CRF;
param.rc.f_rf_constant = 25;
param.rc.f_rf_constant_max = 35;
param.i_sps_id = 7;
//For streaming:
param.b_repeat_headers = 1;
param.b_annexb = 1;
x264_param_apply_profile(¶m, "baseline");
encoder = x264_encoder_open(¶m);
pic_in.i_type = X264_TYPE_AUTO;
pic_in.i_qpplus1 = 0;
pic_in.img.i_csp = X264_CSP_I420;
pic_in.img.i_plane = 3;
x264_picture_alloc(&pic_in, X264_CSP_I420, 720, 920);
convertCtx = sws_getContext(720, 960, PIX_FMT_RGB24, 720, 760, PIX_FMT_YUV420P, SWS_FAST_BILINEAR, NULL, NULL, NULL);
if (eventTriggerId == 0)
{
eventTriggerId = envir().taskScheduler().createEventTrigger(deliverFrame0);
}
}
H264FramedSource::~H264FramedSource()
{
--referenceCount;
if (referenceCount == 0)
{
// Reclaim our 'event trigger'
envir().taskScheduler().deleteEventTrigger(eventTriggerId);
eventTriggerId = 0;
}
}
void H264FramedSource::AddToBuffer(uint8_t* buf, int surfaceSizeInBytes)
{
uint8_t* surfaceData = (new uint8_t[surfaceSizeInBytes]);
memcpy(surfaceData, buf, surfaceSizeInBytes);
int srcstride = W*3;
sws_scale(convertCtx, &surfaceData, &srcstride,0, H, pic_in.img.plane, pic_in.img.i_stride);
x264_nal_t* nals = NULL;
int i_nals = 0;
int frame_size = -1;
frame_size = x264_encoder_encode(encoder, &nals, &i_nals, &pic_in, &pic_out);
static bool finished = false;
if (frame_size >= 0)
{
static bool alreadydone = false;
if(!alreadydone)
{
x264_encoder_headers(encoder, &nals, &i_nals);
alreadydone = true;
}
for(int i = 0; i < i_nals; ++i)
{
m_queue.push(nals[i]);
}
}
delete [] surfaceData;
surfaceData = NULL;
envir().taskScheduler().triggerEvent(eventTriggerId, this);
}
void H264FramedSource::doGetNextFrame()
{
deliverFrame();
}
void H264FramedSource::deliverFrame0(void* clientData)
{
((H264FramedSource*)clientData)->deliverFrame();
}
void H264FramedSource::deliverFrame()
{
x264_nal_t nalToDeliver;
if (fPlayTimePerFrame > 0 && fPreferredFrameSize > 0) {
if (fPresentationTime.tv_sec == 0 && fPresentationTime.tv_usec == 0) {
// This is the first frame, so use the current time:
gettimeofday(&fPresentationTime, NULL);
} else {
// Increment by the play time of the previous data:
unsigned uSeconds = fPresentationTime.tv_usec + fLastPlayTime;
fPresentationTime.tv_sec += uSeconds/1000000;
fPresentationTime.tv_usec = uSeconds%1000000;
}
// Remember the play time of this data:
fLastPlayTime = (fPlayTimePerFrame*fFrameSize)/fPreferredFrameSize;
fDurationInMicroseconds = fLastPlayTime;
} else {
// We don't know a specific play time duration for this data,
// so just record the current time as being the 'presentation time':
gettimeofday(&fPresentationTime, NULL);
}
if(!m_queue.empty())
{
m_queue.wait_and_pop(nalToDeliver);
uint8_t* newFrameDataStart = (uint8_t*)0xD15EA5E;
newFrameDataStart = (uint8_t*)(nalToDeliver.p_payload);
unsigned newFrameSize = nalToDeliver.i_payload;
// Deliver the data here:
if (newFrameSize > fMaxSize) {
fFrameSize = fMaxSize;
fNumTruncatedBytes = newFrameSize - fMaxSize;
}
else {
fFrameSize = newFrameSize;
}
memcpy(fTo, nalToDeliver.p_payload, nalToDeliver.i_payload);
FramedSource::afterGetting(this);
}
}
Oh and for those who want to know what my concurrent queue is, here it is, and it works brilliantly http://www.justsoftwaresolutions.co.uk/threading/implementing-a-thread-safe-queue-using-condition-variables.html
Enjoy and good luck!
The deliverFrame method lacks the following check at its start:
if (!isCurrentlyAwaitingData()) return;
see DeviceSource.cpp in LIVE
I write an application, which plays a sound getting from Hardware (like a ring buffer filled with a sinus wave with certain frequency). Everything works fine, and I can playback the created sound correctly except a periodical clicking (maybe at the end of buffer?) and noise.
I initialize and run the Buffer:
void Audiooutput::InitializeAudioParameters()
{
Audio_DataWritten = 0;
Audio_fragments = 4;
Audio_channels = 2;
Audio_BufferSize = 256;
Audio_Samplerate = 8000;
Audio_ResamplingFactor = 1;
Audio_Framesize = 2;
// (SND_PCM_FORMAT_S16_LE / 8);
Audio_frames = Audio_BufferSize / Audio_Framesize * Audio_fragments;
snd_pcm_uframes_t size;
err = snd_pcm_hw_params_any(pcmPlaybackHandle, hw_params);
err = snd_pcm_hw_params_set_rate_resample(pcmPlaybackHandle, hw_params, 1);
// qDebug()<<a1.sprintf(" % d \t snd_pcm_hw_params_set_rate: %s",Audio_Samplerate,snd_strerror(err));
err =
snd_pcm_hw_params_set_format(pcmPlaybackHandle, hw_params,
SND_PCM_FORMAT_S16_LE);
err =
snd_pcm_hw_params_set_channels(pcmPlaybackHandle, hw_params,
Audio_channels);
err = snd_pcm_hw_params_set_rate_near(pcmPlaybackHandle, hw_params, &Audio_Samplerate, 0);
// qDebug()<<a1.sprintf(" % d \t snd_pcm_hw_params_set_rate: %s",Audio_Samplerate,snd_strerror(err));
if ((err =
snd_pcm_hw_params_set_periods_near(pcmPlaybackHandle, hw_params,
&Audio_fragments, 0)) < 0) {
qDebug() << a1.sprintf("Error setting # fragments to %d: %s\n",
Audio_fragments, snd_strerror(err));
} else
qDebug() << a1.sprintf("setting # fragments to %d: %s\n",
Audio_fragments, snd_strerror(err));
err = snd_pcm_hw_params_get_buffer_size(hw_params, &size);
if ((err =
snd_pcm_hw_params_set_buffer_size_near(pcmPlaybackHandle,
hw_params,
&Audio_frames)) < 0) {
qDebug() << a1.
sprintf("Error setting buffer_size %d frames: %s",
Audio_frames, snd_strerror(err));
} else
qDebug() << a1.sprintf("setting Buffersize to %d --> %d: %s\n",
Audio_BufferSize, Audio_frames,
snd_strerror(err));
Audio_BufferSize = Audio_frames;
if ((err = snd_pcm_hw_params(pcmPlaybackHandle, hw_params)) < 0) {
qDebug() << a1.sprintf("Error setting HW params: %s",
snd_strerror(err));
}
Q_ASSERT(err >= 0);
}
void Audiooutput::ProduceAudioOutput(int n, int mmodes, int totalMModeGates,
short *sinusValue, short *cosinusValue)
{
for (int audioSample = 0; audioSample < n;
audioSample += Audio_ResamplingFactor) {
currentposition =
(int)(m_Audio.generalPos % (Audio_BufferSize / 2));
if (currentposition == 0) {
QueueAudioBuffer();
m_Audio.currentPos = 0;
}
m_Audio.generalPos++;
AudioData[currentposition * 2] =
(short)(sinusValue[audioSample]);
AudioData[currentposition * 2 + 1] =
(short)(cosinusValue[audioSample]);
}
}
void Audiooutput::QueueAudioBuffer()
{
snd_pcm_prepare(pcmPlaybackHandle);
Audio_DataWritten +=
snd_pcm_writei(pcmPlaybackHandle, AudioData, Audio_BufferSize);
}
Changing the audiobuffer size or fragments changes also the clicking period.
Can anyone help me with this issue ?
I checked also the first and Last Values. Thy are always difference.
OS: Ubuntu 11
more detail.
the count of received data is dynamically, and changes depend of different parameters. But I play always a certain part e.g. 128 values or 256 or 512....
// I get the Audiodata from a hardware (in a Timerloop)
audiobuffersize = 256;
short *AudioData = new short[256];
int generalAudioSample = 0;
void CollectDataFromHw()
{
...
int n = 0;
n = GetData(buf1,buf2);//buf1 = new short[MAX_SHRT]
if(n > 0)
FillAudioBuffer(n,buf1,buf2)
...
}
-------------------------------------------
void FillAudioBuffer(int n, short*buf1, short*buf2)
{
for(int audioSample = 0;audioSample < n; audioSample++){
iCurrentAudioSample = (int)(generalAudioSample % (audiobuffersize/2));
if(iCurrentAudioSample == 0) {
snd_pcm_writei(pcmPlaybackHandle,AudioData,audiobuffersize );
memset(AudioData,0x00,audiobuffersize*sizeof(short));
}
generalAudioSample++;
AudioData[iCurrentAudioSample * 2] = (short)(buf1[audioSample];
AudioData[iCurrentAudioSample * 2 +1] = (short)(buf2[audioSample];
}
}
I changed the audiobuffersize also. If I set it to a bigger size, I have some Echo additional to clicks.
any Idea ?
//-----------------------
the Problem is
snd_pcm_prepare(pcmPlaybackHandle);
every call of this function produce a click in sound !
Can't test the source code, but I think that the high-frequency clicks you hear are discontinuities in the sound wave. You have to assure that looping period (or, buffer size) is multiple of wave period.
Check if first and last value of buffer are almost the same (+/- 1, for example). Their distance determines the amplitude of the unwanted click.
solved
buffer has been played several times before it was filled with the data.
stupid error in the code.missing a parantez --> audio_buffersize/2 <--
and therefore the result was very often if(iCurrentAudioSample == 0) true !!!!!
iCurrentAudioSample = (int)(generalAudioSample % (audio_buffersize/2));
if(iCurrentAudioSample == 0)
{
writetoaudioStream(audiobuffer);
}