I try to create an audio streaming server in java. There are several protocols to stream media like RTP, and I'm a little confused with all protocols.
What are the differences between RTP and Shoutcast? Do they use TCP, UDP or HTTP? Does anyone have a clear explanation on this point?
SHOUTcast and Icecast use a client protocol very similar to HTTP. (In fact, Icecast is compliant with HTTP as spec'ed in RFC2616, and most HTTP clients work with SHOUTcast without modification.) A request comes in for a stream, and they return the stream audio data in the same way as an HTTP response, along with some extra metadata.
GET /radioreddit/main_mp3_128k HTTP/1.1
HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Access-Control-Allow-Headers: X-Requested-With
Server: AudioPump Server/0.8.1 (http://audiopump.co)
Content-Type: audio/mpeg
Cache-Control: no-cache
Pragma: no-cache
Expires: Sat, 15 Aug 2009 22:00:00 GMT
Connection: close
icy-genre: Indie,Rock,Talk
icy-name: Radio Reddit - Main
icy-pub: 1
icy-url: http://radioreddit.com
Date: Tue, 05 Aug 2014 13:40:55 GMT
In this example, the response is purely HTTP. If this were a SHOUTcast server, instead of seeing HTTP/1.1 200 OK in the status line, you would see ICY 200 OK. The headers that start with icy- are those that describe the station. Sometimes there are more, sometimes they don't exist at all. A client would be able to play MP3 data from this station as-is.
Now, sometimes the client will request metadata to be sent in the stream. This allows the player to tell you what is playing. A client does this by sending a icy-metadata: 1 header. The server will respond with icy-metaint: 8192, which means that every 8192 bytes there will be a chunk of metadata. You can read more about the format of that metadata on a previous answer.
I should also note that this form of streaming is often called HTTP Progressive Streaming. To the client, it's no different than playing a media file as it is being downloaded... except that the file is infinite in size.
Now, RTP is a protocol used in conjunction with RTSP. RTP is the actual media data where RTSP is used for control. These protocols are much more complicated, as they are meant for true streaming. That is, if the client can't handle the bandwidth, they can drop down to a lower bitrate. If the client needs to control the remote, that can be done as well. This complexity comes at a price. The servers are complicated to implement. There isn't great client compatibility.
A couple years back when I started to create my own streaming server, I had asked myself the same question. Do I implement a real streaming protocol that has not-so-great client support and will take a long time to figure out, or do I implement a protocol that everything can play, and is easy to build for. I went the HTTP route.
These days, you should also consider HLS which is nothing more than chunking your source stream into pieces at several bitrates and serving it over HTTP. If the client wants to change bitrates due to lack of bandwidth, it can just start requesting the low bitrate chunks. HLS also doesn't have great client support, but it is getting better. I suspect it will surpass all others for media delivery on websites in time.
The comparison between RTP (a protocol) and Shoutcast (an application) is not applicable.
RTP is the Internet standard for transporting multimedia over IP.
It is an application layer protocol, so RTP can be transported over TCP or UDP, depending on the nature of the communication. Also, it needs to be implemented by an application before it can be of any use, much like HTTP has to be implemented by a browser so that a user can access web pages. When implemented by an application, it is used for specifying the media format, numerous other characteristics and basic control information (via RTSP).
Shoutcast is a streaming application created by Nullsoft (the Winamp authors). It uses its own protocols for media transportation and can run over either TCP or UDP.
There are numerous other streaming protocols, so a Google search on the subject is, as usually, a good idea. Wikipedia also hosts a comparison of streaming media systems.
Related
Okay, so a little context:
I have an app running on an embedded system that sends a few different requests over HTTP (using libcurl in C++) at the following intervals:
5 minutes
15 minutes
1 hour
24 hours
My goal: Reduce data consumption (runs over cellular)
We have both client and server side TLS authentication, so the handshake is costly. The idea is that we use persistent connections (at least for the shorter interval files) to avoid doing the handshake every time.
Unfortunately, after much tinkering I've figured out that the server is closing the connection before the intervals pass. Maybe this is something we can extend? I'll have to talk to the server side guys.
I was under the impression that was the reason the "TCP keep-alive" packets existed, but supposedly those "check the connection" not "keep it open" like the name suggests.
My idea is this:
Have my app send a packet (as small as possible) every 2 minutes or so (however long the timeout is) to "nudge" the connection into staying open.
My questions are:
Does that make any sense?
I don't suppose there is an easy way to do this in libcurl is there?
If so, how small could we get the request?
Is there an even easier way to do it? My only issue here is that all the connection stuff "lives" in libcurl.
Thanks!
It would be easier to give a more precise answer if you gave a little more detail on your application architecture. For example, is it a RESTful API? Is the use of HTTP absolutely mandatory? If so, what HTTP server are you using (nginx, apache, ...)? Could you consider websockets as an alternative to plain HTTP?
If you are at liberty to use something other than regular HTTP or HTTPs - and to use something other than libcurl on the client side - you would have more options.
If, on the other hand, if you are constrained to both
use HTTP (rather than a raw TCP connection or websockets), and
use libcurl
then I think your task is a good bit more difficult - but maybe still possible.
One of your first challenges is that the typical timeouts for a HTTP connection are quite low (as low as a few seconds for Apache 2). If you can configure the server you can increase this.
I was under the impression that was the reason the "TCP keep-alive" packets existed, but supposedly those "check the connection" not "keep it open" like the name suggests.
Your terminology is ambiguous here. Are you referring to TCP keep-alive packets or persistent HTTP connections? These don't necessarily have anything to do with each other. The former is an optional mechanism in TCP (which is disabled by default). The latter is an application-layer concept which is specific to HTTP - and may be used regardless of whether keep-alive packets are being used at the transport layer.
My only issue here is that all the connection stuff "lives" in libcurl.
The problem with using libcurl is that it first and foremost a transfer library. I don't think it is tailored for long-running, persistent TCP connections. Nonetheless, according to Daniel Stenberg (the author of libcurl), the library will automatically try to reuse existing connections where possible - as long as you re-use the same easy handle.
If so, how small could we get the request?
Assuming you are using a 'ping' endpoint on your server - which accepts no data and returns a 204 (success but no content) response, then the overhead - in the application layer - would be the size of the HTTP request headers + the size of the HTTP response headers. Maybe you could get it down to 200-300 bytes, or thereabouts.
Alternatives to (plain) HTTP
If you are using a RESTful API, this paradigm sort of goes against the idea of a persistent TCP connection - although I can not think of any reason why it would not work.
You might consider websockets as an alternative, but - again - libcurl is not ideal for this. Although I know very little about websockets, I believe they would offer some advantages.
Compared to plain HTTP, websockets offer:
significantly less overhead than HTTP per message;
the connection is automatically persistent: there is no need to send extra 'keep alive' messages to keep it open;
Compared to a raw TCP connection, the benefits of websockets are that:
you don't have to open a custom port on your server;
it automatically handles the TLS/SSL stuff for you.
(Someone who knows more about websockets is welcome to correct me on some of the above points - particularly regarding TLS/SSL and keep alive messages.)
Alternatives to libcurl
An alternative to libcurl which might be useful here is the Mongoose networking library. It would provide you with a few different alternatives:
use a plain TCP connection (and a custom application layer protocol),
use a TCP connection and handle the HTTP requests yourself manually,
use websockets - which it has very good support for (both as server and client).
Mongoose allows you to enable SSL for all of these options also.
Requirement:
Client supports 2MBPS bitrate and it will be sending HTTPS CURL request to server
Server supports 1000kbps bitrate.So it has to send the data at the speed of 1000kbps to the client..
But In my case,Server doesn't validating the bitrate which i've set.Instead its sending the data at rapid speed to the client.Now my requirement is to control the bandwidth on server side and send the data at the specified rate of 1000kbps.
Be able to fetch the bitrate from code level: I need some clue about how we need to handle the bitrate at socket level for HTTPS.
Could someone please guide me on achieving this?
Note:
For HTTP, it works properly because bit-rate is handled at socket level whereas for HTTPS, they use ssl_write library call to transfer the data. In that call, they are not dealing with bitrate.
If someone guide me about how to deal with this, it will be more helpful for me.
I send binary data from client (Debian 6.0.3) to server (Windows Server 2003). To bypass most firewalls I use HTTPS POST. Client and server are implemented using Boost.Asio and OpenSSL. First I implemented the simplest possible version and it worked fine.
HTTP Header:
POST / HTTP/1.1
User-Agent: my custom client v.1
[binary data]
([binary data] is not base64 encoded if this matters)
Then, on another client machine it failed (connected to the same server machine). The behavior is not stable. Connection always is established fine (port 443). Most time I pass SSL handshake fine but server receive no data (almost no data, sometimes a packet or two are actually received). Sometimes I receive SSL handshake error "short read". Sometimes I receive invalid data.
Client connects to server, handshakes, sends HTTP POST header and then infinitely sends binary data until something wrong hapenned. For test I use custom generated SSL certificate.
Server code:
namespace ssl = boost::asio::ssl;
ssl::context context(io_service, ssl::context::sslv23);
context.set_options(ssl::context::default_workarounds | ssl::context::no_sslv2);
context.use_certificate_chain_file("server.pem");
context.use_private_key_file("server.pem", boost::asio::ssl::context::pem);
ssl::stream<tcp::socket> socket(io_service, context);
// standard connection accepting
socket.async_handshake(ssl::stream_base::server, ...);
...
boost::asio::async_read_until(socket, POST_header, "\r\n\r\n", ...);
...
Client code:
ssl::context context(io_service, ssl::context::sslv23);
context.load_verify_file("server.crt");
socket.reset(new ssl::stream<tcp::socket>(io_service, context));
socket->set_verify_mode(ssl::verify_none);
// standard connection
socket.async_handshake(ssl::stream_base::client, ...);
...
(error handling is omitted along with not relevant code)
As you can see, it's the simplest possible SSL connection. What is wrong? Can the reason be a firewall?
I tried simple TCP w/o SSL over the same 443 port, this works fine.
EDIT:
Tried adding "Content-Type: application/octet-stream", doesn't help.
EDIT 2:
Usually I receive HTTP POST header fine. Then I send data chunks as chunk-size(4 bytes)chunk(chunk-size bytes).... Server receives chunk-size fine, but then nothing. Client doesn't notify server problems (no errors) and continue to send data. Sometimes server can receive chunk or two, sometimes it receives invalid chunk-size, but most time just nothing.
EDIT 3:
Compared captured traffic on client and server, didn't find any differences.
Solution
I was misled from the start with this problem. Narrowed it down to surprising details:
Sending over SSL socket fails if I use Boost.Asio multi-buffers in Boost v.1.48 (the most recent one at this moment). Example:
// data to send, protocol is [packet size: 4 bytes][packet: packet_size bytes]
std::vector<char> packet = ...;
uint32_t packet_size = packet.size();
// prepare buffers
boost::array<boost::asio::const_buffer, 2> bufs = {{boost::asio::buffer(&packet_size, sizeof(packet_size)), boost::asio::buffer(packet)}};
// send multi buffers by single call
boost::asio::async_write(socket, bufs, ...);
Sending separately packet_size and packet in this example works around the problem. I'm far from calling any suspicious behavior as a bug, especially if it's related with Boost libraries. But this one really looks like a bug. Tried on Boost v.1.47 - works fine. Tried with usual TCP socket (not SSL one) - works fine. The same on both Linux and Windows.
I'm going to find any reports about this problem in Asio mailing list and will report it if nothing found.
If you don't have to operate in front of web server, you don't have
to use HTTPS protocol. From the firewall point of view HTTPS looks like yet
another SSL connection and it has no idea what going through. So if the
only thing you need is just to pass the data - not to actual web server, use
just SSL connection over 443 port.
So just troubleshoot your SSL connection the problem has nothing to do with HTTP.
If you want to use HTTP web server and not custom client:
Two points:
You need to specify Content-Length.
If you are using HTTP/1.1 you need to specify Host header.
The simplest would be
POST /url HTTP/1.0
User-Agent: my custom client v.1
Content-Type: application/octet-stream
Content-Length: NNN
Actual Content
Or for HTTP/1.1
POST /url HTTP/1.1
Host: www.example.com
User-Agent: my custom client v.1
Content-Type: application/octet-stream
Content-Length: NNN
Actual Content
Note: you can't send infinite data. HTTP protocol requires fixed content-lenght
and most of the time web servers would load the data first before passing it to the
backend.
So you will have to transfer data by chunks.
I was misled from the start with this problem. Narrowed it down to surprising details:
Sending over SSL socket fails if I use Boost.Asio multi-buffers in Boost v.1.48 (the most recent one at this moment). Example:
// data to send, protocol is [packet size: 4 bytes][packet: packet_size bytes]
std::vector<char> packet = ...;
uint32_t packet_size = packet.size();
// prepare buffers
boost::array<boost::asio::const_buffer, 2> bufs = {{boost::asio::buffer(&packet_size, sizeof(packet_size)), boost::asio::buffer(packet)}};
// send multi buffers by single call
boost::asio::async_write(socket, bufs, ...);
Sending separately packet_size and packet in this example works around the problem. I'm far from calling any suspicious behavior as a bug, especially if it's related with Boost libraries. But this one really looks like a bug. Tried on Boost v.1.47 - works fine. Tried with usual TCP socket (not SSL one) - works fine. The same on both Linux and Windows.
I'm going to find any reports about this problem in Asio mailing list and will report it if nothing found.
(EDIT: I had originally deleted this because I had realised it wasn't using HTTP really. Following a comment where you think you might have a MITM proxy and should use proper HTTP, I'm undeleting/editing.)
POST / HTTP/1.1
User-Agent: my custom client v.1
[binary data]
Whether it's binary data or not, in plain HTTP or with SSL/TLS, you'll need a Content-Length header or to use chunked transfer encoding. This this section of the HTTP spec. A Content-Type header would be useful too.
Chunked transfer encoding is for when you don't necessarily know the length of the stream. (You always need some form of delimiters when sending data, if only to detect reliably when it ends.)
This being said, you should be able to find out whether you're behind a MITM proxy that looks into the application layer on top of SSL/TLS if you get a certificate that's not your servers. If you do still get a successful handshake with your won server cert, there isn't such a proxy. Even an HTTP proxy would use CONNECT and relay everything, without altering the SSL/TLS connection (and thus without altering your original pseudo-HTTP on top).
So I've been playing around with some simple HTML forwarding with c++. Haven't accomplished much and I have some questions on the backbone.
First: Do I need to use any special libraries other than socket libraries to simply forward HTML data and connections?
Second: When a client connects to an HTML server, is the TCP connection kept open? Or is it closed once data is sent?
Third: When I forward data, from a client to the server, the packet includes the destination address. I should technically be able to read this address and connect to the server via port 80, keep it open, and send and receive on that newly opened port right? Is there anything I have to do? Any time constraints? If I directly forward every single packet directly between the client and server the website should show up correctly on the client, correct?
I would prefer to keep any external libs to a minimum. But if necessary I can expand the program to include any required libraries.
So far I've gotten data to and from both parties, however the website does not function.
[platform] :: windows.primary && posix_compliant.secondary
First: No you do not need other special libraries but not using any that are available would to some extent be reinventing the wheel.
Second: No, HTTP is a connectionless protocol.
Third: An HTTP session begins with a request header, which in your case sounds like a POST. A POST may take more than one package, during which time the connection remains open. The server may well time you out.
You might look at libCURL even if you do not intend using it. (The source for that is in C, and is rather monolithic but it is commonly used).
After doing quite a bit of research, the greatest help I've had in my endeavors has been this website.
This one also helped quite a bit.
LibCURL is certainly the way to go. It's kind of dated, and everything is in C, but it's much easier than redoing everything..
quote from second site:
Like most network protocols, HTTP uses the client-server model: An HTTP client opens a connection and sends a request message to an HTTP server; the server then returns a response message, usually containing the resource that was requested. After delivering the response, the server closes the connection (making HTTP a stateless protocol, i.e. not maintaining any connection information between transactions).
I'm not real hip on exactly what role(s) today's proxy servers can play and I'm learning so go easy on me :-) I have a client/server system I have written using a homegrown protocol and need to enhance the client side to negotiate its way out of a proxy environment.
I have an existing client and server system written in C and C++ for the speed and a small amount of MFC in the client to handle the user interface. I have written both the server and client side of the system on Windows (the people I work for are mainly web developers using Windows everything - not a choice) sticking to Berkeley Sockets as it were via wsock32 for efficiency. The clients connect to the server through a nonstandard port (even though using port 80 is an option to get out of some environments but the protocol that goes over it isn't HTTP). The TCP connection(s) stay open for the duration of the clients participation in real time conferences.
Our customer base is expanding to all kinds of networked environments. I have been able to solve a lot of problems by adding the ability to connect securely over port 443 and using secure sockets which allows the protocol to pass through a lot environments since the internal packets can't be sniffed. But more and more of our customers are behind a proxy server environment and my direct connections don't make it through. My old school understanding of proxy servers is that they act as a proxy for external HTML content over HTTP, possibly locally caching popular material for faster local access, and also allowing their IT staff to blacklist certain destination sites. Customer are complaining that my software doesn't recognize and easily navigate its way through their proxy environments but I'm finding it difficult to decide what my "best fit" solution should be. My software doesn't tear down the connection after each client request, and on top of that packets can come from either side at any time, basically your typical custom client/server system for a specific niche.
My first reaction is "why can't they just add my server's addresses to their white list" but if there is a programmatic way I can get through without requiring their IT staff to help it is politically better and arguably a better solution anyway. Plus maybe I'm still not understanding the role and purpose of what proxy servers and environments have grown to be these days.
My first attempt at a solution was to use WinInet with its various proxy capabilities to establish a connection over port 80 to my non-standard protocol server (which knows enough to recognize and answer a simple HTTP-looking GET request and answer it with a simple HTTP response page to get around some environments that employ initial packet sniffing (DPI)). I retrieved the actual SOCKET handle behind WinInet's HINTERNET request object and had hoped to use that in place of my software's existing SOCKET connection and hopefully not need to change much more on the client side. It initially seemed to be my solution but on further inspection it seems that the OS gets first-chance at the received data on this socket since when I get notified of events via the standard select(...) statement on the socket and query the size of the data available via ioctlsocket the call succeeds but returns 0 bytes available, the reads don't work and it goes downhill from there.
Can someone tell me of a client-side library (commercial is fine) will let me get past these proxy server environments with as little user and IT staff help as possible? From what I read it has grown past SOCKS and I figure someone has to have solved this problem before me.
Thanks for reading my long-winded question,
Ripred
If your software can make an SSL connection on port 443, then you are 99% of the way there.
Typically HTTP proxies are set up to proxy SSL-on-443 (for the purposes of HTTPS). You just need to teach your software to use the HTTP proxy. Check the HTTP RFCs for the full details, but the Cliffs Notes version is:
Connect to the HTTP proxy on the proxy port;
Send to the proxy:
.
CONNECT your.real.server:443 HTTP/1.1\r\n
Host: your.real.server:443\r\n
User-Agent: YourSoftware/1.234\r\n
\r\n
Then parse the proxy response, which will start with a HTTP status code, followed by HTTP headers, followed by a blank line. You'll then be talking with your destination (if the status code indicated success, anyway), and can start talking SSL.
In many corporate environments you'll have to authenticate with the proxy - this is almost always HTTP Basic Authentication, which is pretty easy - again, see the RFCs.