I have tried to use [transcirbe_steaming_infinite.py] module with multiple mics. The first one is equipped on my pc(mac book pro) and the other one is external one (Jabra EVOLVE 20). Through Audio MIDI setup I made an aggregate device option (Jabra for channel #1, mac for #2).
To use these mics I modified the codes like ResumableMicrophoneStream._num_channels as 2 and added two extra lines after RecognitionConfig audio_channel_count=2 and enable_seperate_recognition_per_channel=True. And the language in ja-JP
When I tried to use these codes at least work (they are able to recognize each channels) but the problem is that in a certain case, responses comes too late.
The case is when I switch the mic from one of each to the other one. For example, when I try to use mic on channel #1(Jabra) right after using the mic on channel #2, I cannot get the response in time but about 15000ms later.
When I checked the mics on Audio MIDI setup those two's sample rate was different(16kHz, 44.1kHz per each), so I thought up with a possibility it has affected on the library processing audio input streams like PyAudio and finally it has caused late request and response as well. It will be dummiest hypothesis XD.
So I want to know, as the title this problem(late response) can be fixed with good mics setup or just there is another good problem solving way for this case.
A common cause of latency is the API not detecting the end of the audio and therefore it will continue to listen and process audio until either the stream is closed directly, or the stream's limit length has been exceeded. You can discard it using single_utterance which indicates whether the request should automatically end after speech is no longer detected. Also if you are using noise filtering, that should be removed so that Cloud sees the raw audio and can properly detect isFinal.
If the latency issues only occurs when mics are changed and you are following the best practices, you can reach the STT team through the public issue tracker
Currently using the lib's from FFPMEG to stream some MPEG2 TS (h264 encoded) video. The streaming is done via UDP multicast.
The issue I am having currently is two main things. There is a long initial connection time / getting the video to show (the stream also contains metadata, and that stream is detected by my media tool immediately).
Once the video gets going things are fine but it is always delayed by that initial connection time.
I am trying to get as near to LIVE streaming as possible.
Currently using the av_dict_set(&dict, "tune", "zerolatency", 0) and "profile" -> "baseline" options.
GOP size = 12;
At first I thought the issue was an i frame issue, but the initial delay is there if gopsize is 12 or default 250. Sometimes the video will connect quickly, but it is immediately dropped, the delay occurs, then it starts back up and is good from that point on.
According to documentation the zero latency option should be sending many i frames, to limit initial syncing delays.
I am starting to think its a buffering type issue, as when I close the application and leave the media player up, it then fast forwards through the delay till it hits basically where the file stops streaming.
So while I don't completely understand what was wrong, I at least fixed the problem I was having.
The issue came from using the av_write_interleaved_frame() vs. the regular av_write_frame()(this one works for live streaming), when writing out the video frames. Ill have to dig into the differences a bit more to fully understand it, but its funny sometimes how you figure out the problem you are having on a total whim after bashing your face for a few days.
I can get pretty good live ish video streaming with the tune "zerolatency" option set.
Question: Is it possible to update 100+ objects in the Flash Player over Socket Connections? More details and my own try's below!
Details
For my internship I got the time to create a multiplayer physics game. I worked for a steady three months on it. My internship is coming to an end and I couldn't finish my game.
My problem is that its hard to send multiple packets each time-step to the server and back. The packets I send are position updates of the objects and mouse of other clients.
I will try to explain the network/game flow.
Client connects to the server using the binary Socket class in AS3
Server ask for verification and client sends name and thumbnail.
Server waits until 4 clients are connected (Some matchmaking etc)
Server picks 4 clients and makes them run on a separate Thread(Combined as a Team)
Client sends his performance score to the server range 1-100.
Server makes the best client the host machine for the physics and the other 3 slaves
Host game sets up the level and makes around 1-100 shapes in the level(primary shapes and complex shapes like bridges, motors, springs)
Every time-step the host gets all updated property's of the shapes and sends them to the clients (x, y, rotation, sleep)
The client applys all the shape property's to the correct shapes
I tried different time-steps and noticed that until a time-step of 1/15 second the client(slave) won't notice any lagging in the game. I also tried to pick a lower time-step and tween the movement of the shapes but that did give some strange movement on the client(slave) side.
I will give an example of a single object update packet.
<O|t=s:u|x=201|y=202|f=automaticoo</O
<O|t=m:p|x=100|y=345|f=automaticoo</O
I noticed that the Flash Player can stack a lot of packets in the buffer before sending. For example if I send a lot of packets at once it stacks them up and send them together to the server. With faster time-steps you don't get more updates on the client(slave) side but more updates in the same packet row.
Tries
Use the new RTMFP(udp & p2p) protocol for updates. (little bit better in performance but less in reliability)
Code my entire socket server in c++ instead of Air(with the ServerSocket) (better in performance but noticed the lagging part is not the server but the Flash Player)
Use the ByteArray compress method and the AMF serialized format (performance about the same except the c++ server can't unserialize the messages)
Do you guys think it is possible in the Flash Player too handle so many update request each time-step.
Discoveries
There is a stick arena game that is multiplayer in ActionScript 3.0. They used a lot of tricking and even then I get a ping of about 300ms and it only updates the players constantly (4 players in a lobby).
Sorry for the long post.
I believe your problem comes down to type of game and data.
This again is broken up into:
server speed (calculations need in CPU + RAM requirements for world/player data)
connection speed (bandwidth on server)
data size (how much info is needed and how often)
player interaction form (event or FPS)
distance from client to server (ping)
MMORG
Eg. World Of Warcraft, is to my knowledge on the none-PVP worlds using a "client is a viewport" and "client sends keystrokes", "server validates and performs, and tells client what happens" on a CLIENT-SERVER base.
This gives the game a lot of acceptable latency as you only need to transfer commands from client and then results to the client. The rest is drawn on the client.
Its very event driven and from you click an icon or press a key, its okay that your "spell" needs some time to fire on the server. Secondly there is no player collision needed. This lets the server process less data too and keeps the requirements to the server CPU smaller.
Counter-Strike / Battlefield etc.
FPS, fast paced action, with quick response needs to get information about every detail all the time. This makes a higher demand on precision. Collision is a must for both player and weapons.
This sort of game usually doesnt handle more than 32 players on a single map, as they all need to be able to share their positions, bullets, explosions etc. very fast and all this data has to go through the server-validation which again is a bottleneck for any type of online game.
Network latency
In a perfect world this would be 0 ms, but as we all know. All the hardware from client to the server and back takes time. Both going through the network stacks and through the internet connection (switch, router, modem, fiber centrals etc) so the way many modern real time games fixes this is by "prediction". The let the server look at your direction and speed. Then they try to predict (much like the GPS do in a tunnel too) that you were last seen moving forward with a speed of +4 so given timeframe you have moved (timeframes x 4) - but what if you had slowed down or speeded up? then they either instantly "hyperjump" you from A to B in a split second and this you feel like a lagging game or they easy up to the real position so your "hero" slides a little faster or slower into the right possition.
This technique is explained many places on the net, so no need for details in here, but it takes time and tweaks to get a good performance from this - but it works and saves a lot of headaches for the programmers.
What network data is needed?
I read your question and thought: that could be compressed quite a lot. Secondly, I have made a Flash socket chat with pure ByteStream and that worked awesome. It was hard to get running for a start, but once I got it up and running it was fast.
The flash client/player itself isnt the biggest networking client, so expect a lot of lost speed there too. I would go for 10-15 fps for the networking part and then use a more RAW approach for the data sent back and forth.
Lastly, try to keep the data as simple as possible.
Eg. use COMMANDS/SHORTCUTS for certain data/events.
Like a server data bytestring could be: 0x99, 0x45,0x75,0x14,0x04,0x06
Where 0x99 means : BIG EXPLOSION at the following COORDS: (0x45,0x75)
Then 0x14 means: PLAYER 0x14 (player 20 in decimal) has moved to (0x04, 0x06)
So the staring opcode tells the networking protocol handler in your client and server what to expect next. (Its how a CPU knows how to read memory btw.)
For my chat I had commands for each type of data parsed. One for login, one for broadcasting, for for telling the name of a user etc. So once the client made a login, the client received a command + a packed of online users. This was only transfed once to the client. After that each attached client received a "new user online" command too with the name of the new user. Each client maintained its own list with current users and ID's so that I only needed to tell which client number say the text. This kept the traffic at a minimum. Same would go for coordinates or commands of what to do. "Player #20 goes north" etc. could be be 0x14, 0x41, 0xf0 (0x41 could be MOVE, 0xf0 could be NORTH, 0xf1 EAST etc.)
This physical distance to the game
This one you cant change, but you can put in some restrictions or make the servers run in multiple locations worldwide, depending on what type of game you wanna make. Amazon EC2 is a great platform for such projects as they have data-centers all over the world and then you can benchmark the users network against these and then redirect the users to the nearest datacenter where you are running a server.
Hacking/cheating
Also remember, if something gets popular and you start earning money on it, sooner or later SOMEONE will try to break the protocols or break down the accounts to gain access to servers, informations or cheat to get further items/points in the games. You could also be attacked by DDOS where they bomb your network with wrong data just to crash everything and render the game unusable.
Dont mind it so much for a start, just remember that once you go online, you NEVER know who in the world or where in the world they are. I'm not trying to make you paranoid, but there are sick people who will try to earn money by cheating others.
So think this into your structures, dont show data in network packages that isnt needed. Dont believe data from client always is correct. Validate data on server-side.
This also takes time if you have 100 active players at the same time.
But once you do it, you can sleep much better if it gets to be a big success for you, which I really hope.
That was my thoughts from experience. Hope some of it might be usefull eventhough I didnt quite answer if 100 players are possible.
Infact I would say: YES 100 players is possible, but it depends if they all move at the same time and there is collission testing involved and if you will accept lag or not.
Question: Is it possible to update 100+ objects in the Flash Player over Socket Connections?
Phosphor 2 seems to pull it off.
Maybe the best option was do the physics on server AND on each client, with synchronization (server object positions are overwriting client's). This way all clients get equal lags. Until discrepancy is low (as it should be) corrections will not be noticeable. If you use Box2D, you have both AS3 and C++ version ready. But this is totally different architecture, worth 3 month to implement by itself. What lag do you get on empty/simple arena? In limited time, simplification may be your only option.
I have a c++ object that accepts sound requests and plays them with ALSA. There is thread that processes the sound requests. Some sounds are periodic and are rescheduled after the wav file contents have been written to the ALSA library. Is there a way I can find out when all the data has been played? The function snd_pcm_writei is a blocking write function, but it does not necessarily mean that the file has been played.
One option that I am considering is to call snd_pcm_drain after playing each sound file, then call snd_pcm_prepare when I play the next file. Would this be an good solution? Or is this inefficient?
Update: The "drain solution" seems to work, but is not very efficient. The calls takes a while to return (maybe it cleans up some resources) and adds latency to the program. The latency is seen best when I play many small files consecutively. A few seconds of silence can be heard between each file; this is snd_pcm_drain executing.
Might not be correct (i've done very little work in this area), but from looking at the ALSA docs here: http://www.alsa-project.org/alsa-doc/alsa-lib/pcm.html
It looks like snd_pcm_status_t holds the status information that should give you an indication of whether the stream is currently processing data or not.
I receive audio packets from net (4 packets per second, 250ms each) and video - 15fps. Everything goes with my own timestamps. How should I sync them? I've seen the source code of one of our developers but he did syncing VIDEO according to audio. I.e. audio is always played immediately and video can be dropped or buffered. I don't think it is correct because audio can overrun video for a second or two - in that case we will not have actual video frames at all.
I'd like to know some basics in sync stuff. What should be buffered? Should audio and video in sync mode be played in separate thread(s)? Any clues would be regardful!
Thanks a lot!
I needed in smth like that - http://www.freepatentsonline.com/7680153.html
Pretty difficult to understand but I think this patent explains the basics of sync.