I am trying to build a Virtual Assistant in UE4. I need to somehow send my response from DialogFlow to UE4 for use in the Oculus Lipsync Plugin.
Basically, I have 3 media options for the response:
-16-Bit Linear PCM
-mp3
-Ogg Opus
I have a TCP Server and Client connection set up between a Python Script and UE4, so I can send data to and from easily.
I have my sequencing correct so one script waits for the full Byte Array to be sent via the socket etc.
Basically, I want to send the responses to each Query the user sends to DialogFlow INTO UE4 Via my TCP Socket, and be able to play and access that Audio within UE4.
I need to somehow stream the response from DialogFlow into UE4 as it gives me the responses.
Is what I'm trying to do even possible? I'm just trying to stream in Audio for use within UE4 and I am really struggling to get it working. Very annoying as this is the last piece of the puzzle I need to finish....
Please let me know if you have any advice or help you can offer!
Related
I am writing a server and client code where i want to identify on server side official client software sending data.
The protocol is text based, so anyone can write it's own client - and i want to keep it, but want to group the devices by "Official" / "Non Official" based on the 'fingerprint' of my official client application.
Question:
How can i filter out my clients from other clones sending?
I am thinking to send encrypted message from my client to SERVER using the same key, so once decoded if it's valid, then i know its my app code sending - but before i go this way, any other better ideas ?
I don’t know if I can say “I’m sorry for ask” but I spent more than a week looking for a solution without success. I have a Jetson Nano and with OpenCV I get and process an image at 4fps, I need to send this video to a web server to allow the client connected to the server get the video. Everything need to be written in C++.
Because a need a low latency I did test with GStreamer and WebRTC without success. I don’t have any web server ready, so I can use any implementation.
Anyone know where I can find some example implementation with this schema?
You can use mediasoup to send data to the server to then send the stream with rtp to another endpoint like gstreamer or ffmpeg.
Here is a recording project where data is sent from the browser -> server -> gstreamer -> file.
Mediasoup is written in c++ and has a wrapper for js.
I had similar problem and used such example from GStreamer WebRTC official repo. It's written in Python for Janus Gateway video rooms but I think it can be easily rewritten in C++ as you need.
In the code for OpenCV, I used V4L2Loopback as a virtual output device to be used as input for GStreamer WebRTC example.
I hope such approach may help you.
I think no need to send it to a Web Server. In Gstreamer examples [https://github.com/GStreamer/gst-examples]. The SendOnly example sends a video to a Web Client Using WebRTC. You can modify it to send an OpenCV mat.
I manged to run WebRTC peerconnection example, but it is not running on the browser.
I'm trying to find a way to stream both video and audio from browser to my native program.
Is there any way?
It can be done. WebRTC is designed to work in a peer-to-peer manner between two WebRTC agents (typically a Web Browser). Your native program needs to become the second peer.
If you need to rely on open source components a good starting point is:
OpenSSL for the DTLS key exchange.
libsrtp to encrypt the RTP packets.
ffmpeg to decode the PCM audio from the browser (libvpx if you need to do video).
You'll also need to handle the ICE negotiation which requires processing STUN messages. Also extract the media payloads from the RTP packets. All these steps are also after you've determined a signalling method to exchange the SDP offer and answer between you app and the browser.
As you've probably realised starting from scratch it's a major task. There are probably some commercial libraries that will do the job and save you a lot of pain.
If that doesn't scare you and you do still want to make an attempt using open source components this example "may" help. The sample is doing the reverse of what you've asked and is sending a video stream to Chrome rather than receiving an audio stream. The useful aspect is the connection negotiation. The sample program is able to get RTP packets flowing which is often the main problem.
The example is also using Windows Media Foundation which is Windows specific. It also has lots of shortcuts particularly with the RTP and STUN packet processing.
Forgive me for the following, as my understanding of winsock is not quite yet complete (I have only started looking in to it over the past two days). For now, my question is brief and more of a "can I do this the way I want to do it?" ordeal. On to the inquiry itself:
A member of my lab has written a nice winsock server that communicates with a client to transmit instructions to various instruments connected via serial. That part is already written and thankfully I do not have to touch; it works universally for all serial devices. However, the existing clients are all written in MatLab, and I'm trying to rewrite them in C/C++.
I've managed to successfully connect to the server and establish a socket. I can receive the data it sends and display it on the client. Great!
The problem I'm having thus far - after all of the initial data (containing instructions) is sent to the client, I cannot do anything. It will display the received data as text, but does not break the simple do while loop I have and ask the user for an input to transmit.
I'm wondering if it possible to set up a receive-then-transmit loop between the server and the client, such that I can send a command and receive the response with one socket in the client until I am done with talking to the server?
I will post my code tomorrow morning once I'm back at my lab computer!
Thank you for your time.
I am trying build an C++ application, which will try to connect Shoutcast/Icecast server, get an audio file, then stream the audio file into audio device. There is winamp application in windows, which already does the same. But I would like the same kind of application in C++. I have read the http://www.smackfu.com/stuff/programming/shoutcast.html link. But not sure how I should start. I would need specific help on this: a) How do I connect to shotcast server? b) get the audio file url? c) How should I start streaming? I have built one socket program which can connect to server, can send/recv binary data. Hope this can help here. But I am unable to start on above three points? I have read stackoverflow urls on this:
Stream getUserMedia to an Icecast server?
Audio streaming using C++ tutorial and sample code
http://forums.radiotoolbox.com/viewtopic.php?t=74
But could not help much.
Can you please help me on this?
SHOUTcast/Icecast are essentially HTTP. You should be searching on how to make an HTTP request.
There is a small difference for SHOUTcast servers though... the status line they return is non-standard:
ICY 200 OK
A normal status line would be:
HTTP/1.1 200 OK
Once you learn how to make an HTTP request, you will be streaming. Later on if you want to get the metadata, you can follow the SmackFu link you posted.