When I enabled debug logs in istio I saw something like the following:
[C186][S86652966017378712] router decoding headers:
After searching a little bit I understood that this is the connection id followed by the stream id: [connection_id][stream_id] message
What is the difference between a connection and a stream?
Does the connection refer to a TCP connection and stream to an HTTP request?
Related
Here is my code: http://pastebin.com/znrwaT0n
When you press enter it should send a usual HTTP GET request to www.google.com.
As you can see it also successfully receives the HTML-code.
But for some reason Wireshark and other network monitors don't detect that it was sent as HTTP request, they just mark it as TCP (while other HTTP requets get marked as "HTTP") and the data doesn't seem to be readable to humans.
Could you explain me why this happens, what the side-effects are and how I could avoid it?
May be the port is not '80'. If so, set the Wireshark to decode the Transport layer of capture as HTTP (both on the receiving and sending side)
I have a client implementation and a server implementation. I want the client to send data about itself along with its connection request to the server. So for example, I could send an additional packet with the username. I was wondering if this is at all possible using connect() before a stream has actually been established. I would then accept only based on that information.
Thanks
Don't mix layers. The TCP connect is on the network layer, establishing a communication channel between the machines.
Instead, define your protocol (the rules for how data is transmitted in your TCP stream) to contain a handshake first. E.g. SMTP uses a header and handshake to first set up the connection before any actual emails are sent.
I send binary data from client (Debian 6.0.3) to server (Windows Server 2003). To bypass most firewalls I use HTTPS POST. Client and server are implemented using Boost.Asio and OpenSSL. First I implemented the simplest possible version and it worked fine.
HTTP Header:
POST / HTTP/1.1
User-Agent: my custom client v.1
[binary data]
([binary data] is not base64 encoded if this matters)
Then, on another client machine it failed (connected to the same server machine). The behavior is not stable. Connection always is established fine (port 443). Most time I pass SSL handshake fine but server receive no data (almost no data, sometimes a packet or two are actually received). Sometimes I receive SSL handshake error "short read". Sometimes I receive invalid data.
Client connects to server, handshakes, sends HTTP POST header and then infinitely sends binary data until something wrong hapenned. For test I use custom generated SSL certificate.
Server code:
namespace ssl = boost::asio::ssl;
ssl::context context(io_service, ssl::context::sslv23);
context.set_options(ssl::context::default_workarounds | ssl::context::no_sslv2);
context.use_certificate_chain_file("server.pem");
context.use_private_key_file("server.pem", boost::asio::ssl::context::pem);
ssl::stream<tcp::socket> socket(io_service, context);
// standard connection accepting
socket.async_handshake(ssl::stream_base::server, ...);
...
boost::asio::async_read_until(socket, POST_header, "\r\n\r\n", ...);
...
Client code:
ssl::context context(io_service, ssl::context::sslv23);
context.load_verify_file("server.crt");
socket.reset(new ssl::stream<tcp::socket>(io_service, context));
socket->set_verify_mode(ssl::verify_none);
// standard connection
socket.async_handshake(ssl::stream_base::client, ...);
...
(error handling is omitted along with not relevant code)
As you can see, it's the simplest possible SSL connection. What is wrong? Can the reason be a firewall?
I tried simple TCP w/o SSL over the same 443 port, this works fine.
EDIT:
Tried adding "Content-Type: application/octet-stream", doesn't help.
EDIT 2:
Usually I receive HTTP POST header fine. Then I send data chunks as chunk-size(4 bytes)chunk(chunk-size bytes).... Server receives chunk-size fine, but then nothing. Client doesn't notify server problems (no errors) and continue to send data. Sometimes server can receive chunk or two, sometimes it receives invalid chunk-size, but most time just nothing.
EDIT 3:
Compared captured traffic on client and server, didn't find any differences.
Solution
I was misled from the start with this problem. Narrowed it down to surprising details:
Sending over SSL socket fails if I use Boost.Asio multi-buffers in Boost v.1.48 (the most recent one at this moment). Example:
// data to send, protocol is [packet size: 4 bytes][packet: packet_size bytes]
std::vector<char> packet = ...;
uint32_t packet_size = packet.size();
// prepare buffers
boost::array<boost::asio::const_buffer, 2> bufs = {{boost::asio::buffer(&packet_size, sizeof(packet_size)), boost::asio::buffer(packet)}};
// send multi buffers by single call
boost::asio::async_write(socket, bufs, ...);
Sending separately packet_size and packet in this example works around the problem. I'm far from calling any suspicious behavior as a bug, especially if it's related with Boost libraries. But this one really looks like a bug. Tried on Boost v.1.47 - works fine. Tried with usual TCP socket (not SSL one) - works fine. The same on both Linux and Windows.
I'm going to find any reports about this problem in Asio mailing list and will report it if nothing found.
If you don't have to operate in front of web server, you don't have
to use HTTPS protocol. From the firewall point of view HTTPS looks like yet
another SSL connection and it has no idea what going through. So if the
only thing you need is just to pass the data - not to actual web server, use
just SSL connection over 443 port.
So just troubleshoot your SSL connection the problem has nothing to do with HTTP.
If you want to use HTTP web server and not custom client:
Two points:
You need to specify Content-Length.
If you are using HTTP/1.1 you need to specify Host header.
The simplest would be
POST /url HTTP/1.0
User-Agent: my custom client v.1
Content-Type: application/octet-stream
Content-Length: NNN
Actual Content
Or for HTTP/1.1
POST /url HTTP/1.1
Host: www.example.com
User-Agent: my custom client v.1
Content-Type: application/octet-stream
Content-Length: NNN
Actual Content
Note: you can't send infinite data. HTTP protocol requires fixed content-lenght
and most of the time web servers would load the data first before passing it to the
backend.
So you will have to transfer data by chunks.
I was misled from the start with this problem. Narrowed it down to surprising details:
Sending over SSL socket fails if I use Boost.Asio multi-buffers in Boost v.1.48 (the most recent one at this moment). Example:
// data to send, protocol is [packet size: 4 bytes][packet: packet_size bytes]
std::vector<char> packet = ...;
uint32_t packet_size = packet.size();
// prepare buffers
boost::array<boost::asio::const_buffer, 2> bufs = {{boost::asio::buffer(&packet_size, sizeof(packet_size)), boost::asio::buffer(packet)}};
// send multi buffers by single call
boost::asio::async_write(socket, bufs, ...);
Sending separately packet_size and packet in this example works around the problem. I'm far from calling any suspicious behavior as a bug, especially if it's related with Boost libraries. But this one really looks like a bug. Tried on Boost v.1.47 - works fine. Tried with usual TCP socket (not SSL one) - works fine. The same on both Linux and Windows.
I'm going to find any reports about this problem in Asio mailing list and will report it if nothing found.
(EDIT: I had originally deleted this because I had realised it wasn't using HTTP really. Following a comment where you think you might have a MITM proxy and should use proper HTTP, I'm undeleting/editing.)
POST / HTTP/1.1
User-Agent: my custom client v.1
[binary data]
Whether it's binary data or not, in plain HTTP or with SSL/TLS, you'll need a Content-Length header or to use chunked transfer encoding. This this section of the HTTP spec. A Content-Type header would be useful too.
Chunked transfer encoding is for when you don't necessarily know the length of the stream. (You always need some form of delimiters when sending data, if only to detect reliably when it ends.)
This being said, you should be able to find out whether you're behind a MITM proxy that looks into the application layer on top of SSL/TLS if you get a certificate that's not your servers. If you do still get a successful handshake with your won server cert, there isn't such a proxy. Even an HTTP proxy would use CONNECT and relay everything, without altering the SSL/TLS connection (and thus without altering your original pseudo-HTTP on top).
Quite new to boost and asio, need help:
connect to proxy
asio::ip::tcp::socket socket_;
send CONNECT host: ssl server to the proxy
receive response 200
asio::ssl::context ctx(io_service, asio::ssl::context::sslv23);
sslsocket_(socket_,context)
try handshake
sslsocket_.async_handshake(asio::ssl::stream_base::client, boost::bind(&client::handle_handshake, this, asio::placeholders::error));
and get asio.ssl error
Wireshark: host sends FIN after 1st message of handshake
Direct async connection to ssl server works fine, same through Proxifier
You'll probably need to examine the OpenSSL error stack in step 5, which isn't terribly easy since Boost.Asio doesn't map it into the normal system_error codes that it uses. There was a post by Guy Sirton recently on the asio mailing list describing how to modify the source to accomplish this.
Forgot to mention that was running the application in VMplayer with NAT network; bridged removed the problem with initial handshake but sometimes after reconnect saw the same message;
using the method in Sam's link got to "unexpected record" during handshake - google said on this error about renegotiations but that was not the case; digging into sources of OpenSSL: the new connection used the same BIO for reading and recevied application data from previous connection
Don't know how to reset the SSL structure with asio, so made dynamic socket allocation with new|delete
If the client wants to watch a stream that is on my RTSP server, it first tries to setup a stream through the UDP protocol. How can I tell it that my server only supports RTP/AVP/TCP and that it should switch transports?
I want to terminate the UDP support on my server, but all the clients first try to SETUP the session over UDP, and later they do so over TCP... and I want to switch them to TCP as soon as possible in RTSP protocol.
How can I do that?
As far as I know, there is no control at server side for transport type preference. Server should be made generic it should support RTP over UDP, RTP over TCP, RTP over RTSP and RTP over RTSP over HTTP(S). And its clients choice which transport to choose. Transport field is first sent in SETUP request
1) UDP
C->A: SETUP rtsp://audio.example.com/twister/audio.en RTSP/1.0
CSeq: 1
Transport: RTP/AVP/UDP;unicast;client_port=3056-3057
2) TCP
C->A: SETUP rtsp://audio.example.com/twister/audio.en RTSP/1.0
CSeq: 1
Transport: RTP/AVP/TCP;unicast;client_port=3056-3057
3) RTP over RTSP and RTP over RTSP over HTTP(S)
S->C: RTSP/1.0 200 OK
CSeq: 2
Date: 05 Jun 1997 18:57:18 GMT
Transport: RTP/AVP/TCP;interleaved=0-1
As we can see "Transport type" request is sent by client side.
If you want to support TCP only server you can send "400 Bad Request" or "461 Unsupported transport" in response to SETUP request as suggested by you or another way is to send 200 OK but do not transmit any RTP packets. Client will timeout and get to know that it is behind proxy and it will send SETUP request again with RTP/AVP/TCP parameter (Not an ideal case).
To expand on the answer for android,
For Android clients, they will always attempt to establish a UDP connection first.
For both OpenCore and StageFright I can confirm that if I return "461 Unsupported Transport" from my server in response to the first SETUP request for UDP transfer, both of these clients will then IMMEDIATELY attempt to establish a TCP based connection over the RTSP port.
All other responses are detailed here: http://www.ietf.org/rfc/rfc2326.txt
OK one way is to send "400 Bad Request" as the response to the client's SETUP request... and it automatically switches to TCP protocol. This is for RealOne and QuickTime.
But I am not sure that it will work on all other players since this is a hack.
Any other ideas? =|
If you used ffmpeg, you can force switch rtsp transport layer protocol.
av_dict_set(&format_opts, "rtsp_transport", "tcp", 0);
err = avformat_open_input(&ic, is->filename, is->iformat, &format_opts);
What client connects to your server? Some clients can be triggered through the URI method in the URL. For example, you could specify rtspt://myhost/path.sdp.
If you have control over client/servers you could use the Require header on clients and Unsupported on servers to indicate that UDP isn't supported; but most clients I've seen don't use this.
You can try to pass "transport" header in a response to Describe request, and state there that your server only supports RTP/AVP/TCP transport, and the client should know that UDP is unsupported.