I want the VLC to run video and audio at different speeds, and synchronize them at a given point (as an input).
I have two ways to that:
Build two new modules which control respectively the audio and video separately, so I don't need to "go throw" all source code. It seems to be a hard solution, because we would have to write and re-write big pieces of code that already appear in VLC. The question is how to build these modules and what do we need to control to do it.
Enter the code and add pieces of code where VLC passes the audio and video frames to the buffers - I searched for this point a lot of time and I didn't find it. If someone know where this point it will help me a lot.
Related
For the quick and concise: I want to find out how to work with .ogg files in visual c++ to put it in a format the PC can play and have tools to control how it's played and be able to play multiple at once, but all of it limited to a 2d scale. Preferably with no or very little cost. Also I've heard about streaming audio but I'm not so sure what the details on that are. The rest is an explanation of my though process on the whole thing, which is likely flawed in other areas this short hand doesn't convey but getting the short hand down will make that apparent eventually.
So I'm working on a game which will be wholly in the 2D space, for purposes of self improvement I want to try and build the engine myself and work things the best I can. For audio I'm thinking ahead of how to program it while I finish with the visual side of things, and have gotten an idea of things I want to be able to do with the sound but I'm inexperienced and probably have a flawed sense of the basic process. So I'll list what I functions I'll need done and my current mindset of how to approach it. After this I'll clarify my issues with .ogg. So the functions I know I'll need at current are:
Ability to play audio files
Ability to change volume of audio files mid playback
Ability to play them starting a specific times (this along with the above to allow for me to fade a background song out and play a different version of the same background music at where the last one dropped off)
Ability to play them in a loop
Ability to play multiple audio files at the same time (sound effects with background music.)
Ability to pan sounds to one ear or the other when using headphones and such.
Now for my mindset of how to approach it:
When loading in decode the audio files and load them into objects I can reference later on that same screen, scene, or event till I no longer need them and can delete them.
have events which will call audio effects which will play them or change things. IE: player gets hit, sound plays then ends.
I think I'll need buffers or channels to play them like 1 channel for background music, another for a secondary background music that fades into the current as I fade out and remove the current. Then I can free up the first channel for another fading of tracks. Then a few more to make space for sound effects. Like say 10 that stay empty but allow for me to play a sound in one, then delete it and then go back and play some other effect in that same channel later. However if I need to play multiple sound effects at once I can play up to 10 at the same time. However if I need them I'm not sure if I'll be able to clear them up or each channel has its own song tied to it instead of me being able to pass the decoded sound to what it has set to play.
Then every frame progresses the sound being played, I'm not exactly sure if I need or will do this. I intend to try and work online play into my game using rollback netcode. From my experience sounds don't freeze or progress based on connection and just run on. As in game freezes from connection waiting for opponent, the background music will play like normal. If it freezes on a hit, the hit sound should just play out once and end so desync there should be fine but I am kind of curious on how and if there is a good reason for it. I also heard a term dropped, streaming, I'm trying to understand how to go about it but I struggled to get a good grasp on it.
That's about it for how I think I should go about it, if I had to come up with a sort psuedo code probably would be:
LoadScene(..., file backgroundsounda, file backgroundsoundb, file soundeffecthit){
.
.
.
Backsadeco = new decodedsound(backgroundsounda)
.
.
.
LoadScene.setBackSoundA(backsadeco)
.
.
.
LoadScene.makeBuffers(2,10)
//makeBuffers from a Buffer class which makes two for the background and 10 effect buffers that the scene can reference later for those purposes. The buffers use a switch statement to denote how to play the sound based on number given later. EX:1 plays sound once and then empties buffer. Buffer also manages which effect or background buffer to use by checking which ones aren't playing sound.
}
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
class collision
strikeHit(...,scene){
.
.
.
scene.useEffectBuffer(scene.getSoundEffectHit(),1)
}
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
(the pseudo code above for handling the scene stuff could also be flawed cause I haven't gotten to making scene system yet with load screens and such, first I want to get what I need done and existing in one scene first then work on handling the transitions. Though if you have advice on that I'm all ears too.)
Aside from any potential flaws in my plan when I can play sound, working with ogg which I planned to use since its' a smaller file size and planning to have 3 versions of every song which can go upwards of 2 minutes each would be wise, I'm not sure how or what to do. I've heard of OpenAL but I'm confused on the license stuff and since I plan to hopefully sell the game one day and I want to reduce costs I'm trying to avoid fees where I can. Secondly I hear it has quite a few stuff for 3d and I'm not so sure since all my actions are intended for 2d and at most I'll pan sounds, if it's worth it or if I can manage something more streamlined and focused for my needs. Realistically all I should need is a way to format it to be played by the PC and then tools to handle how that is played. Also understanding of how to keep the PC playing it as the rest of the game updates and changes logically and visually on 60fps.
Edit based on first comment: I intend to keep it usable by multiple systems but limited to Windows, focusing on Windows 10 currently. I am working In visual studio and so far have just used things available there for my needs so I think the only framework would be .NET. As for video playing, at current I don't have a need for that since I plan to work more so with 2d images and animation, and currently can't find a purpose for it. As for licenses I'm not well versed on that topic but I'm trying to stay free for commercial use if possible.
I write a MFC project . I use IXAudio2 to play wav file.
my code is like this :
pSourceVoice->SubmitSourceBuffer( &buffer );
hr = pSourceVoice->Start( 0 );
but in this way I only can play one sound by a time. It must wait this .wav file play over. I can play the second file. How can I play the file I need and not wait for the first one is over, likes mixing sounds?
How can I achieve this function?
You create one IXAudio2SourceVoice for each 'playing, active' sound you want at a time. Typically games will create a few hundred source voices to manage--beyond that the sound mix is typically too muddy to hear anyhow.
Game audio engines have a mix of 'one-shot' voices that just play-on-demand until they complete--at which point that frees up that source voice for use by another one-shot sound--, 'ambients' or 'music' voices that are manipulated as they play, and 3D audio positioned sounds which have to be updated every rendering frame to track the 3D rendering object they are emitted from.
If you submit the same audio data to two distinct IXAudio2SourceVoice instances, then you'll hear it being played twice, mixed together in real-time. Typically you won't start them at precisely the same instant because the result is just a louder version of the one sound. Normally they are started at different times so you get overlapped playback.
See XAudio2 Programming Guide, GitHub, and DirectX Tool Kit for Audio.
I'm trying to write an application that records and saves the screen in C++ on the windows platform. I'm not sure where to start with this. I assume I need some sort of API, (FFMPEG, maybe OpenGL?). Could someone point me in the right direction?
You could start by looking at Windows remote desktop protocol, maybe some programming libraries are provided for that.
I know of a product that intercepts calls into the Windows GDI dll and uses that to store the screen drawing activities.
A far more simpler approach would be to do screenshots as often as possible and somehow minimize redundant data (parts of the screen that didn't change between frames).
If the desired output of your app is a video file (like mpeg) you are probably better off just grabbing frames and feeding them into a video encoder. I don't know how fast the encoders are these days. Ffmpeg would be a good place to start.
If the encoder turns out not fast enough, you can try storing the frames and encoding the video file afterwards. Consecutive frames should have many matching pixels, so you could use that to reduce the amount of data stored.
I am looking for a fast way to load in a video file and to create images from them at certain intervals ( every second, every minute, every hour, etc.).
I tried using DirectShow, but it just ran too slow for me to start the video file and move to a certain location to get data and to save it out to an image. Even if I disabled the reference clock. Tried OpenCV, but it has trouble opening the AVI file unless I know the exact codec information. So if I know a way to get the codec information out from OpenCV I may give it another shot. I tried to use FFMPEG, but I don't have as much control over it as well as I would wish.
Any advice would be greatly appreciated. This is being developed on a Windows box since it has to be hosted on a Windows box.
MPEG-4 format is not an intra-coded format, so you can't just jump to a random frame and decode it on its own, as most frames only encode the differences from one or more other frames. I suspect your decoding is slow because when you land on a frame for which several other dependent frames to be decoded first.
One way to improve performance would be to determine which frames are keyframes (or sometimes also called 'sync' points) and limit your decoding to those frames, since these can be decoded on their own.
I'm not very familiar with DirectShow capabilities, but I would expect it has some API to expose sync points.
Also, I should mention that the QuickTime SDK on Windows is possibly another good option that you have for decoding frames from movies. You should first test that your AVI movies are played correctly in the QuickTime Player. And the QT SDK does expose sync points, see the section Finding Interesting Times in the QT SDK documentation.
ffmpeg's libavformat might work for ya...
What is the best/easiest way to display a video (with sound!) in an application using XAudio2 and Direct3D9/10?
At the very least it needs to be able to stream potentially larger videos, and take care of the fact that the windows aspect ratio may differ from the videos (eg by adding letter boxes), although ideally Id like the ability to embed the video into a 3D scene.
I could of course work out a way to load each frame into a texture, discarding/reusing the textures once rendered, and playing the audio separately through XAudio2, however as well as writing a loader for at least one format, ive also got to deal with stuff like synchronising the video and audio components, so hopefully there is an eaier solution available or even a ready made free one with a suitable lisence (commercial distribution in binary form, dynamic linking is fine in the case of say LGPL).
In Windows SDK, there is a DirectShow example for rendering video to texture. It handles audio output too.
But there are limitations and I can't honestly call it easy.
Have you looked at Bink video? Its what lots of games use for video playback. Works great and you don't have to code all that video stuff yourself from scratch.