Problem
- I am working on a Streaming server & created a nonblocking socket using:
flag=fcntl(m_fd,F_GETFL);
flag|=O_NONBLOCK;
fcntl(m_fd,F_SETFL,flag);
Server then sends the Media file contents using code:
bool SendData(const char *pData,long nSize)
{
int fd=m_pSock->get_fd();
fd_set write_flag;
while(1)
{
FD_ZERO(&write_flag);
FD_SET(fd,&write_flag);
struct timeval tout;
tout.tv_sec=0;
tout.tv_usec=500000;
int res=select(fd+1,0,&write_flag,0,&tout);
if(-1==res)
{
print("select() failure\n");
return false;
}
if(1==res)
{
unsigned long sndLen=0;
if(!m_pSock->send(pData,nSize,&sndLen))
{
print(socket send() failure\n");
return false;
}
nSize-=sndLen;
if(!nSize)
return true; //everything is sent
}
}
}
Using above code, I am streaming a say 200sec audio file, which I expect that Server should stream it in 2-3secs using full n/w available bandwidth(Throttle off), but the problem is that Server is taking 199~200secs to stream full contents.
While debugging, I commented the
m_pSock->send()
section & tried to dump the file locally. It takes 1~2secs to dump the file.
Questions
- If I am using a NonBlocking TCP socket, why does send() taking so much time?
Since the data is always available, select() will return immediately (as we have seen while dumping the file). Does that mean send() is affected by the recv() on the client side?
Any inputs on this would be helpul. Client behavior is not in our scope.
Your client is probably doing some buffering to avoid network jitter, but it is likely still playing the audio file in real time. So, the file transfer rate is matched to the rate that the client is consuming the data. Since it is a 200 second audio file, it will take about 200 seconds to complete the transfer.
Because TCP output and input buffers are propably much smaller than the audio file, reading speed of the receiving application can slow down the sending speed.
When both the TCP output buffer of sender and the input buffer of receiver are both full, TCP stack of the sender is not able to receive any data from the sender. So sending will be blocked, until there is space.
If the receiver reads the TCP stream same speed as data is needed for playing. Then the transfer takes about 200 seconds. Or little bit less.
This can be avoided by using application layer buffering in the receiving end.
The problem could be that if the client side is using blocking TCP, plus is processing all the data on a single thread with no no buffer/queue etc right through to the "player" of the file, then your side being non-blocking will only speed things until you reach the point where the TCP/IP protocol stack buffers, NIC buffers etc are full. Then you will ultimately still only be able to send data as fast as the client side is consuming it. Remember TCP is a reliable, point-to-point protocol.
Where does your client code come from in your testing? Is it some sort of simple test client someone has written?
Related
first of all a little background on my situation:
- Qt/C++ UI desktop application
- embedded device (Stm32l4xx family) +ATWINC1500 wifi module
I'm developing the gui application in order to send commands and files to the emdedded device via sockets.
For simple commands I've done all successfully, but for sending files (text files in GCODE format) I am stuck with some issues.
The embedded device has already a socket management(not written by me, so I have not the possibility to modify the way sockets are managed, coming from third party company), and the reception of that type of files is managed in a way that the API waits for every single line of the file being sent, and then wrotes it into a reserved portion of the flash.
My problem is that when I send file from qt Application(by reading each line and and calling write() on the line, in reality my socket sends an entire chunk of the file, like 50 lines, resulting in my device not managing the file reception.
My sending code is this:
void sendGCODE(const QString fileName)
{
QFile *file = new QFile(fileName,this);
bool result = true;
if (file->open(QIODevice::ReadOnly))
{
while (!file->atEnd())
{
QByteArray bytes(file->readLine());
result = communicationSocket->write(bytes);
communicationSocket->flush();
if(result)
{
console->append("-> GCODE line sent:"+ QString(bytes));
}
else
{
console->append("-> Error sending GCODE line!");
}
}
file->close();
}
}
Have anyone of you guys any hints on what I am doing wrong?
I've already searched and someone suggests on other topic that for this purpose it should be better to use UDP instead of TCP sockets, but unfortunately I cannot touch the embedded-device-side code.
thank you all!
EDIT
After suggestions from comments, I've sniffed tcp packets and the packets are sent correctly(i.e. each packet contains a single line). BUT... at the receiver(device), I understood that there is something regarding memory which is not well managed. an example:
sender sends the line "G1 X470.492 Y599.623 F1000" ; receiver receives correctly the string "G1 X470.492 Y599.623 F1000"
next, if the line length is less than the previous sent, i.e. sending "G1 Z5", the receiver receives: "G1 Z5\n\n.492 Y599.623 F1000", so it is clear that the buffer used to store the data packet is not re-initialized from previous packet content, and the new part overwrites the previous values where the remaining part is from the previous packet
I'm trying to figure out how I could reset that part of memory.
This is all wrong. TCP is not a message-oriented protocol. There is no way to ensure that the TCP packets contain any particular amount of data. The receiver code on the device mustn't expect that either - you perhaps misunderstood the receiver's code, or are otherwise doing something wrong (or the vendor is). What the receiver must do is wait for a packet, add the packet's data to a buffer, then extract and process as many complete lines as it can, then move the remaining data to the beginning of the buffer. And repeat that on every packet.
Thus you're looking for the wrong problem at the wrong place, unless your device never ever had a chance of working. If that device works OK with other software, then your "packetized" TCP assumption doesn't hold any water.
Here's how to proceed:
If the device is commercially available and has been tested to work, then you're looking in the wrong place.
If the device is a new product and still in development, then someone somewhere did something particularly stupid and you either need to fix that stupidity, or have the vendor fix it, or hire a consultant to fix it. But just to be completely clear: that's not how TCP works, and you cannot just accept that "it's how it is".
I am trying to read high-frequency data from a non-blocking, no-delay, tcp client socket. I do my reading in a spin-loop that basically looks like this:
while(true)
{
r = recv(socket,some_buffer,20000);
if (r > 0)
{
cout << calculate_delay() << endl;
}
}
This thread is running on a dedicated core alone, while the whole OS (centos in my case) runs on core 0. So nothing can interfere with my reading loop. I also use a kernel-bypass networboard (solarflare) with openonload driver to bypass the kernel completely when reading my network data. The data comes from a server that is cross-connected with me in a constant 500usec latency line. My problem is that 0.1% of the time the calculate_delay() returns 10,15,100ms delays which is absurd and huge. Why is this happening what could be the cause of it?
The server sends 250bytes packets, sometimes up to 5-10 messages in 1 millisecond.And I also noticed it's possible i receive buffered data reading 500,1000,2000... bytes at once on my receiving socket.
Why is this buffering even happening if i read in a spin-loop without delays?
thank you
I'm having a strange behaviour with the recv() function.
My C++ (MFC) application with WinSock implements a simple HTTP client (non-blocking socket) for accessing HTML pages on a web server. Some of these pages are taking a few seconds for loading. On Windows 7 this is not a problem, because recv() also returns partial data. But on Windows XP the recv() function always returns SOCKET_ERROR and the error code is WSAEWOULDBLOCK. Only when the connection is finished the data is returned in one access.
Does anyone know this problem? How can I force Windows XP to also receive partial data?
I setted the buffer size (SO_RCVBUF) to 1000 Bytes. On Windows 7 this is also reflected to the TCP Window Size - on XP not.
The real problem which I have with this issue is, that I don't know how to check if the connection is still alive or not. How can I check if a connection is still alive? Or how can I specify a timeout (max time between two received packets from the server)?
By default, a socket operates in blocking mode, so the only way you can get a WSAEWOULDBLOCK error at all is if you explicitly put the socket into non-blocking mode instead. Doing so, you agree to handle WSAEWOULDBLOCK (otherwise, don't use non-blocking mode).
WSAEWOULDBLOCK is not a real error, it is just an indication that the operation you attempted to perform cannot be completed at that moment because it would block the calling thread. You need to detect this "error" and simply retry the same operation again at a later time, preferably after a socket state change is detected.
For recv(), WSAEWOULDBLOCK simply means there is no data available on the socket to be read at that moment. In non-blocking mode, you should be using select() (or WSAEventSelect(), or WSAAsyncSelect(), or Overlapped I/O, or an I/O Completion Port) to detect inbound data before you then read it.
That being said, you are implementing an HTTP client, so you must follow the HTTP protocol properly, regardless of the socket I/O mode you are using, regardless of your socket buffer sizes. You must follow the pseudo code logic I outlined in this answer on another question:
You must follow the rules outlined in RFC 2616. Namely:
Read until the "\r\n\r\n" sequence is encountered. Do not read any more bytes past that yet.
Analyze the received headers, per the rules in RFC 2616 Section 4.4. They tell you the actual format of the remaining response data.
Read the data per the format discovered in #2.
Check the received headers for the presence of a Connection: close header if the response is using HTTP 1.1, or the lack of a Connection: keep-alive header if the response is using HTTP 0.9 or 1.0. If detected, close your end of the socket connection because the server is closing its end. Otherwise, keep the connection open and re-use it for subsequent requests (unless you are done using the connection, in which case do close it).
Process the received data as needed.
In short, you need to do something more like this instead (pseudo code):
string headers[];
byte data[];
string statusLine = read a CRLF-delimited line;
int statusCode = extract from status line;
string responseVersion = extract from status line;
do
{
string header = read a CRLF-delimited line;
if (header == "") break;
add header to headers list;
}
while (true);
if ( !((statusCode in [1xx, 204, 304]) || (request was "HEAD")) )
{
if (headers["Transfer-Encoding"] ends with "chunked")
{
do
{
string chunk = read a CRLF delimited line;
int chunkSize = extract from chunk line;
if (chunkSize == 0) break;
read exactly chunkSize number of bytes into data storage;
read and discard until a CRLF has been read;
}
while (true);
do
{
string header = read a CRLF-delimited line;
if (header == "") break;
add header to headers list;
}
while (true);
}
else if (headers["Content-Length"] is present)
{
read exactly Content-Length number of bytes into data storage;
}
else if (headers["Content-Type"] == "multipart/byteranges")
{
string boundary = extract from Content-Type header;
read into data storage until terminating boundary has been read;
}
else
{
read bytes into data storage until disconnected;
}
}
if (!disconnected)
{
if (responseVersion == "HTTP/1.1")
{
if (headers["Connection"] == "close")
close connection;
}
else
{
if (headers["Connection"] != "keep-alive")
close connection;
}
}
check statusCode for errors;
process data contents, per info in headers list;
As you can see, HTTP requires reading CRLF-delimited lines of text, or fixed lengths of raw bytes. To do that, you must call recv() in a loop until you encounter the terminating CRLF, or have received the expected number of bytes, whichever the case may be. Whether you use a synchronous loop that just ignores WSAEWOULDBLOCK errors while looping, or you use a state machine driven by asynchronous events/callbacks, that is up to you to decide. That doesn't change how you must process the HTTP protocol.
This applies to all versions of Windows (even all platforms that use BSD-style socket APIs). What you are encountering is not a Windows bug at all. It is an underlying flaw in your understanding of how to use socket I/O correctly and effectively.
As for checking if the connection is alive, recv() will return 0 if the server closed the connection gracefully, or will report an error otherwise (usually WSAECONNABORTED or WSAECONNRESET, though there can be others). But an abnormal disconnect may take a long time to detect, so you should implement timeouts in your code instead. In synchronous mode, you can use setsockopt(SO_RCVTIMEO). In non-blocking mode, you can use select(). In asynchronous (overlapped) mode, you can use WaitForSingleObject() on whatever event/object you use to drive your state machine.
You can't expect recv to give you any data on a non-blocking socket. If there's no data available it returns WOULDBLOCK. You just need to call recv again (normally after select notifies you some data is available). Whether you get data on the first (or any) call is going to depend on how fast the server is sending it.
When the socket is closed you'll get a different error from recv, like WSAECONNRESET or WSAENOTCONN. select will also notify you when the socket is closed.
It's very strange.
Today I have changed my software to use blocking sockets. But it still doesn't work on Windows XP. Windows 7 is no problem.
So I thought: Let's try another PC. On this PC (also Windows XP) it does work. Now I tried a 3rd PC with Windows XP and here it also works.
I still don't know what the problem is but I think there must be a bug with the PC.
I'm using boost::asio::write() to write data from a buffer to a com-Port. It's a serial port with a baud rate 115200 which means (as far as my understanding goes) that I can write effectively 11520 byte/s or 11,52KB/s data to the socket.
Now I'm having a quite big chunk of data (10015 bytes) which i want to write. I think that this should take little less than a second to really write on the port. But boost::asio::write() returns already 300 microseconds after the call with the transferred bytes 10015. I think this is impossible with that baud rate?
So my question is what is it actually doing? Really writing it to the port, or just some other kind of buffer maybe, which later writes it to the port.
I'd like the write() to only return after all the bytes have really been written to the port.
EDIT with code example:
The problem is that i always run into the timeout for the future/promise because it takes alone more than 100ms to send the message, but I think the timer should only start after the last byte is sent. Because write() is supposed to block?
void serial::write(std::vector<uint8_t> message) {
//create new promise for the request
promise = new boost::promise<deque<uint8_t>>;
boost::unique_future<deque<uint8_t>> future = promise->get_future();
// --- Write message to serial port --- //
boost::asio::write(serial_,boost::asio::buffer(message));
//wait for data or timeout
if (future.wait_for(boost::chrono::milliseconds(100))==boost::future_status::timeout) {
cout << "ACK timeout!" << endl;
//delete pointer and set it to 0
delete promise;
promise=nullptr;
}
//delete pointer and set it to 0 after getting a message
delete promise;
promise=nullptr;
}
How can I achieve this?
Thanks!
In short, boost::asio::write() blocks until all data has been written to the stream; it does not block until all data has been transmitted. To wait until data has been transmitted, consider using tcdrain().
Each serial port has both a receive and transmit buffer within kernel space. This allows the kernel to buffer received data if a process cannot immediately read it from the serial port, and allows data written to a serial port to be buffered if the device cannot immediately transmit it. To block until the data has been transmitted, one could use tcdrain(serial_.native_handle()).
These kernel buffers allow for the write and read rates to exceed that of the transmit and receive rates. However, while the application may write data at a faster rate than the serial port can transmit, the kernel will transmit at the appropriate rates.
I have a program that sends a set of TCP SYN packets to a host (using raw sockets) and uses libpcap (with a filter) to obtain the responses. I'm trying to implement this in an asynchronous I/O framework, but it seems that libpcap is missing some of the responses (namely the first packets of a series when it takes less than 100 microseconds between the TCP SYN and the response). The pcap handle is setup like this:
pcap_t* pcap = pcap_open_live(NULL, -1, false, -1, errorBuffer);
pcap_setnonblock(pcap, true, errorBuffer);
Then I add a filter (contained on the filterExpression string):
struct bpf_program filter;
pcap_compile(pcap, &filter, filterExpression.c_str(), false, 0);
pcap_setfilter(pcap, &filter);
pcap_freecode(&filter);
And on a loop, after sending each packet, I use select to know if I can read from libpcap:
int pcapFd = pcap_get_selectable_fd(pcap);
fd_set fdRead;
FD_ZERO(&fdRead);
FD_SET(pcapFd, &fdRead);
select(pcapFd + 1, &fdRead, NULL, NULL, &selectTimeout);
And read it:
if (FD_ISSET(pcapFd, &fdRead)) {
struct pcap_pkthdr* pktHeader;
const u_char* pktData;
if (pcap_next_ex(pcap, &pktHeader, &pktData) > 0) {
// Process received response.
}
else {
// Nothing to receive (or error).
}
}
As I said before, some of the packets are missed (falling into the "nothing to receive" else). I know these packets are there, because I can capture them on a synchronous fashion (using tcpdump or a thread running pcap_loop). Am I missing some detail here? Or is this an issue with libpcap?
If the FD for the pcap_t is reported as readable by select() (or poll() or whatever call/mechanism you're using), there is no guarantee that this means that only one packet can be read without blocking.
If you use pcap_next_ex(), you will read only one packet; if there's more than one packet available to be read, then, if you do another select(), it should immediately return, reporting the FD as being readable again, in which case you'll presumably call pcap_next_ex() again, and so on. This means at least one system call per packet (the select()), and possibly more calls, depending on what version of what OS you're doing and what version of libpcap you have.
If, instead, you were to call pcap_dispatch(), with a packet-count argument of -1, that call will return all the packets that can be obtained with a single read operation and process all of them, so, on most platforms, you may get multiple packets with one or two system calls if there are multiple packets available (which, with high network traffic, as you might get if you're testing your program with a SYN flood, is likely to be the case).
In addition, on Linux systems that support memory-mapped packet capture (I think all 2.6 and later kernels do, and most if not all 2.4 kernels do), and with newer versions of libpcap, pcap_next_ex() has to make a copy of the packet to avoid having the kernel change the packet out from under the code processing the packet and to avoid "locking up" a slot in the ring buffer for an indefinite period of time, so there's an extra copy involved.
This seem to be an issue with libpcap using memory mapping under Linux. Please see my other question for details.