I'm facing a particular issue that regards serial communication under win32.
I'm communicating with a device can only accept frames when it is not already communicating. So I must find a valid frame and then inmediatelly send my request.
I developed a class named Serial that handles basic operations on serial port (open, close, read, write) and then a Thread calls inside a loop read and write functions.
Thread loop
//Device is an object of class Serial
while( device->isOpen() && !terminate )
{
unsigned int readed = 0;
unsigned long error = ERROR_SUCCESS;
unsigned char* data = device->read( &readed, &error );
if( error==ERROR_SUCCESS )
{
//If data received, deliver to upper level
if( readed>0 )
{
QByteArray output( (const char*)data, (signed int)readed );
emit dataArrived( output, readed );
}
}
else
{
//unrelated stuff
}
//Here I manage the writting issue
//Only when nothing is received, and Upper layer wants to send a frame
//(Upper layer only will mark as something to send when it detects a valid frame)
if( readed==0 )
{
out_lock.lock();
//If something to send...
if( something_to_send > 0 )
{
if( device->write( output_buffer, output_size, &error ) )
{ //things...
}
}
}
}
The Thread basically keeps reading, and when nothing is received, sees if somebody has signaled to send a frame (this means that a valid frame is just received).
When this happens, it writes the frame through serial port.
Here comes my problem.
Inside the Serial::read() function:
I use the overlapped way of reading:
::ClearCommError( handle, &dwErrors, &stat);
if( stat.cbInQue )
{
//If there's something to read, read it, please note the bytes to read parameter, here 1.
bool ok = ::ReadFile( handle, buffer_in, 1, &bytes_read, &ov_reader );
if( !ok )
{
DWORD _error = ::GetLastError();
if( _error == ERROR_IO_PENDING )
{
DWORD result = ::WaitForMultipleObjects( 2, waiters, FALSE,INFINITE );
switch( result )
{ //Eventshutdown
case WAIT_OBJECT_0: /*code omitted*/break;
case WAIT_OBJECT_0+1: ok = ::GetOverlappedResult( handle, &ov_reader, &bytes_read, true );
//check ok value omitted
break;
}
}
}
}
if( bytes_read>0 )
{
*size = bytes_read;
}
Here starts my problem.
When device sends me small frames (around 30 bytes) everything works fine, but when larger frames are sent, the code is not able to find any free time between frames causing the thread to never be able send any frame because readed is never 0.
If I increase the number of bytes to read inside the read() function, lose the ability to detect when the device "listens":
bool ok = ::ReadFile(handle, buffer_in, 50, &bytes_read, &ov_reader );
This happens because my app can receive the end of a frame together with the start of the next one. This behaviour is very common.
In the other hand, if I change the INFINITE argument by a valid timeout in the WaitForMultipleObjects function, I lose data.
So my question basically is... what I'm doing wrong? Why when reading 1 byte each time I don't find any free time to send my own frames?
Thank you
I'm not sure if this will help or not, but since you already have a good idea of how many bytes are in the serial device's input queue (stat.cbInQue) maybe it would help to read in that many bytes instead of just 1 byte or an arbitrary number of bytes (like 50):
bool ok = ::ReadFile( handle, buffer_in, stat.cbInQue, &bytes_read, &ov_reader );
Of course, you'd need to make sure that the buffer_in had the capacity for that number of bytes, so there might be some other logic you'd have to add to make sure there's no buffer overruns.
Also, because the serial driver and ReadFile() APIs depend heavily on buffering for handling received characters, you might be able to get more precise indications of when characters have been received (and not received) using the
WaitCommEvent() and SetCommMask() APIs.
How big are the "larger frames?" When you call ReadFile one byte at a time, it will obviously take a long time to work through the whole frame, probably longer than it takes the send the frame itself due to call-overhead.
Some alternatives:
Does the device send frames whenever it feels like it? If you have the opportunity to design both ends of the protocol, can you switch to a command/response style of communication?
Can you, from the start of the packet, predict the number of characters in the rest of the packet? If so, you could build a state machine into your read function. You could poll one byte at a time, then when you detect the start of the packet read most of the rest of the packet on one call, then switch back to a byte at a time.
Can you use DSR/CTS to control the timing?
In general, it's really hard to read whole packets from within a serial-port read function. The usual procedure is the read a bunch of characters and pass them up to a higher level for protocol parsing. It sounds like you have to have tighter timing control than that method allows though. Good luck...
Related
I am trying to write reverse proxy with nonblocking socket and epoll. That seems ok at first, but when I tried to open a big jpg file, I got stuck.
When I try to write into client sometimes It may not writable and how can I handle proper way.
Additional Notes:
this->getFd() = ProxyFd
this->clientHandler->getFd = clientFd
I am using EPOLLET flag both proxy and client
if( (flag & EPOLLIN) ){
char buffer[1025] = {'\0'};
int readSize;
while( (readSize = read(this->getFd(),buffer,1024)) > 0){
this->headerParse(buffer);
this->readSize += readSize;
int check = 0;
do{
check = write(this->clientHandler->getFd(),buffer,readSize);
}while(check < 0);
}
if(this->headerEnd == 1 && this->readSize >= this->headerLenght ){
close(this->clientHandler->getFd());
close(this->getFd());
delete this->clientHandler;
delete this;
}
}
Thanks for taking time to read.
Assuming your headerParse() method doesn't change buffer in a size-extending way (you'd need to update readsize, at least, not to mention the buffer full scenario), it seems like your write() path is broken.
if the socket you're writing to is also in nonblocking mode, it's perfectly legal for write() to return -1 (and set errno to EGAIN or EWOULDBLOCK or whatever your platform has) before you wrote all data.
In that case, you must store the remaining data (the remainder of buffer minus what was written if one or more calls to write() succeeded), program epoll to notify the clientHandler->getFd() descriptor for writeability, if not already, and when you get subsequent "write ready" event, you write the data you stored. On this case, the write() can again be unable to flush all your data, so you must cycle until all data is sent.
I attempted to modify Teunis van Beelen's Rs232 library, from Polling to event driven and non-overlapped to suit my project. RS232 Library
I expect to receive blocks of data (roughly 100 to 200 chars) every 200ms.
Problem I am having is the received data is very inconsistent, it is cut off at random points, and incomplete.
I would like ReadFile() to return only after reading one whole block of data.( or something to that effect)
I feel like the problem is with the time out settings, because by altering the figures I get different results, but I just cant get it right, my best result so far has been set all time out values to 0 and let ReadFile() expect 150 bytes, this way ReadFile() dose not return unless it reads 150 chars, but this just go out of sync after few transmissions, as I have no idea how much data to expect.
these are the main changes to the polling function in Teunis's code , besides time out settings, all other settings are unchanged:
//Using the EV_RXCHAR flag will notify the thread that a byte arrived at the port
DWORD dwError = 0;
//use SetCommMask and WaitCommEvent to see if byte has arrived at the port
//SetCommMask sets the desired events that cause a notification.
if(!SetCommMask(Cport[comport_number],EV_RXCHAR)){
printf("SetCommMask Error");
dwError = GetLastError();
// Error setting com mask
return FALSE;
}
//WaitCommEvent function detects the occurrence of the events.
DWORD dwCommEvent;
for( ; ; )
{
//wait for event to happen
if (WaitCommEvent(Cport[comport_number],&dwCommEvent,NULL))
{
if(ReadFile(Cport[comport_number], buf, 1, (LPDWORD)((void *)&n), NULL)){
//Byte has been read, buf is processed in main
}
else{
//error occoured in ReadFile call
dwError = GetLastError();
break;
}
else{
//error in WaitCommEvent
break;
}
break; //break after read file
}
attempt 2 as suggested by MSDN article on serial com using Do While to cycle through every character in the buffer, this method did not yield any good results either.
DWORD dwError = 0;
/*
Using the EV_RXCHAR flag will notify the thread that a byte arrived at the port
*/
//use SetCommMask and WaitCommEvent to see if byte has arrived at the port
//SetCommMask sets the desired events that cause a notification.
if(!SetCommMask(Cport[comport_number],EV_RXCHAR)){
printf("SetCommMask Error");
dwError = GetLastError();
// Error setting com mask
return FALSE;
}
//WaitCommEvent function detects the occurrence of the events.
DWORD dwCommEvent;
for( ; ; )
{
//wait for event to happen
if (WaitCommEvent(Cport[comport_number],&dwCommEvent,NULL))
{
//Do while loop will cycle ReadFile until bytes-read reach 0,
do{
if(ReadFile(Cport[comport_number], buf, size, (LPDWORD)((void *)&n), NULL)){
//Byte has been read, buf is processed in main
}
else{
//error occoured in ReadFile call
dwError = GetLastError();
break;
}
}while(n);
}
else{
//error in WaitCommEvent
break;
}
break; //break after read file
}
I am wondering if rewriting the code in overlapped mode will improve things, but I dont see the advantages as I have no need for multi threading. any suggestions would be great!
Thank you.
ReadFile has no way to detect what a "block of data" is. You should not expect it to understand your data or the timing of that data. The only fix for this issue is for you to process whatever it gives you, using your own knowledge of the data to divide it up into "blocks" for further processing. If you get a partial block keep it, and append to it with the next read.
There is no need to call WaitCommEvent for data. ReadFile will wait for data. But give it a suitably sized buffer and ask for a lot more than one byte at a time. It's extremely inefficient to call it for only one byte. Select the requested count and the timeouts so that ReadFile will return within an acceptable time, whether there is data or not.
I'm writing a distributed system in c++ using TCP/IP and sockets.
For each of my messages, I need to receive the first 5 bytes to know the full length of the incoming message.
What's the best way to do this?
recv() only 5 bytes, then recv() again. if I choose this, would it be safe to assume I'll get 0 or 5 bytes in the recv (aka not write a loop to keep trying)?
use MSG_PEEK
recv() some larger buffer size, then read the first 5 bytes and allocate the final buffer then.
You don't need to know anything. TCP is a stream protocol, and at any given moment you can get as little as one byte, or as much as multiple megabytes of data. The correct and only way to use a TCP socket is to read in a loop.
char buf[4096]; // or whatever
std::deque<char> data;
for (int res ; ; )
{
res = recv(fd, buf, sizeof buf, MSG_DONTWAIT);
if (res == -1)
{
if (errno == EAGAIN || errno == EWOULDBLOCK)
{
break; // done reading
}
else
{
// error, break, die
}
}
if (res == 0)
{
// socket closed, finalise, break
}
else
{
data.insert(data.end(), buf, buf + res);
}
}
The only purpose of the loop is to transfer data from the socket buffer to your application. Your application must then decide independently if there's enough data in the queue to attempt extraction of some sort of higher-level application message.
For example, in your case you would check if the queue's size is at least 5, then inspect the first five bytes, and then check if the queue holds a complete application message. If no, you abort, and if yes, you extract the entire message and pop if off from the front of the queue.
Use a state machine with two states:
First State.
Receive bytes as they arrive into a buffer. When there are 5 or more bytes perform your check on those first 5 and possibly process the rest of the buffer. Switch to the second state.
Second State.
Receive and process bytes as they arrive to the end of the message.
to answer your question specifically:
it's not safe to assume you'll get 0 or 5. it is possible to get 1-4 as well. loop until you get 5 or an error as others have suggested.
i wouldn't bother with PEEK, most of the time you'll block (assuming blocking calls) or get 5 so skip the extra call into the stack.
this is fine too but adds complexity for little gain.
I'm having a problem with unix local sockets. While reading a message that's longer than my temp buffer size, the request takes too long (maybe indefinitely).
Added after some tests:
there is still problem with freeze at ::recv. when I send (1023*8) bytes or less to the UNIX socket - all ok, but when sended more than (1023*9) - i get freeze on recv command.
maybe its FreeBSD default UNIX socket limit or C++ default socket settings? Who know?
i made some additational tests and I am 100% sure that its "freeze" on the last 9th itteration when executing ::recv command, when trying to read message >= (1023*9) bytes long. (first 8th itterationg going well.)
What I'm doing:
The idea is to read in a do/while loop from a socket with
::recv (current_socket, buf, 1024, 0);
and check buf for a SPECIAL SYMBOL. If not found:
merge content of buffer to stringxxx += buf;
bzero temp buf
continue the ::recv loop
How do I fix the issue with the request taking too long in the while loop?
Is there a better way to clear the buffer? Currently, it's:
char buf [1025];
bzero(buf, 1025);
But I know bzero is deprecated in the new c++ standard.
EDIT:
*"Why need to clean the buffer*
I see questions at comments with this question. Without buffer cleanup on the next(last) itteration of reading to the buffer, it will contain the "tail" of first part of the message.
Example:
// message at the socket is "AAAAAACDE"
char buf [6];
::recv (current_socket, buf, 6, 0); // read 6 symbols, buf = "AAAAAA"
// no cleanup, read the last part of the message with recv
::recv (current_socket, buf, 6, 0);
// read 6 symbols, but buffer contain only 3 not readed before symbols, therefore
// buf now contain "CDEAAA" (not correct, we waiting for CDE only)
When your recv() enters an infinite loop, this probably means that it's not making any progress whatsoever on the iterations (i.e., you're always getting a short read of zero size immediately, so your loop never exits, because you're not getting any data). For stream sockets, a recv() of zero size means that the remote end has disconnected (it's something like read()ing from a file when the input is positioned at EOF also gets you zero bytes), or at least that it has shut down the sending channel (that's for TCP specifically).
Check whether your PHP script is actually sending the amount of data you claim it sends.
To add a small (non-sensical) example for properly using recv() in a loop:
char buf[1024];
std::string data;
while( data.size() < 10000 ) { // what you wish to receive
::ssize_t rcvd = ::recv(fd, buf, sizeof(buf), 0);
if( rcvd < 0 ) {
std::cout << "Failed to receive\n"; // Receive failed - something broke, see errno.
std::abort();
} else if( !rcvd ) {
break; // No data to receive, remote end closed connection, so quit.
} else {
data.append(buf, rcvd); // Received into buffer, attach to data buffer.
}
}
if( data.size() < 10000 ) {
std::cout << "Short receive, sender broken\n";
std::abort();
}
// Do something with the buffer data.
Instead of bzero, you can just use
memset(buf, 0, 1025);
These are 2 separate issues. The long time is probably some infinite loop due to a bug in your code and has nothing to do with the way you clear your buffer. As a matter of fact you shouldn't need to clear the buffer; receive returns the number of bytes read, so you can scan the buffer for your SPECIAL_SYMBOL up to that point.
If you paste the code maybe I can help. more.
Just to clarify: bzero is not deprecated in C++ 11. Rather, it's never been part of any C or C++ standard. C started out with memset 20+ years ago. For C++, you might consider using std::fill_n instead (or just using std::vector, which can zero-fill automatically). Then again, I'm not sure there's a good reason to zero-fill the buffer in this case at all.
I am working on network programming using epoll and I have this code...
int read = read(socket, buf, bufsize);
I have a huge buffer size and I assumed it will receive everything clients sent.
However, I started facing problems like packet segmentation.
One example is that if a client sent 500 bytes but it somehow got into two 250 bytes packets then there is no way to handle this situation.
I looked up online and found this code
int handle_read(client *cli, struct epoll_event *ev) {
size_t len = 4096;
char *p;
ssize_t received;
cli->state = 1;
if (cli->buffer != NULL) {
//free(cli->buffer);
//printf("Buff not null %s\n", cli->buffer);
}
//allocate space for data
cli->buffer = (char*)malloc( (size_t)(sizeof(char) * 4096) );
p = cli->buffer;
do { //read until loop conditions
received = recv(ev->data.fd, p, len, 0);
if (received < 0 && errno != EAGAIN && errno != EWOULDBLOCK) {
//if error, remove from epoll and close socket
printf("Handle error!!!\nClient disconnected!\n");
epoll_ctl(epollfd, EPOLL_CTL_DEL, ev->data.fd, ev);
close(ev->data.fd);
}
p = &cli->buffer[received];
} while (received >= len && errno != EAGAIN && errno != EWOULDBLOCK);
return received;
}
Do you guys think it handles all the exceptions might happen while receiving? Also could you please provide me tutorials or examples that handles socket exceptions? Sample codes online don't cover details.. Thanks in advance
recv can return any of three things, and your code needs to handle each one correctly:
1) Positive number. This means it read some bytes.
2) Negative number. This means an "error" occurred.
3) Zero. This means the other end of the connection performed a successful shutdown() (or close()) on the socket. (In general, a return of 0 from read() or recv() means EOF.)
The "error" case further breaks down into "EAGAIN or EWOULDBLOCK" and "everything else". The first two just means it is a non-blocking socket and there was no data to give you at this time. You probably want to go back and call poll() (or select() or epoll()) again to avoid busy waiting...
"Everything else" means a real error. You need to handle those too; see the POSIX spec for recv() for a complete list.
Given all this, I would say your sample code is bad for several reasons. It does not handle 0 (closed connection) properly. It does not handle any errors. It does a busy-loop when the recv() returns EAGAIN/EWOULDBLOCK.
Oh, and it uses sizeof(char), which is a sure sign it was written by somebody who is not familiar with the C or C++ programming languages.
You can't know "How many datas client sent" in normaly. you should use scalable data format(that have data length in the header) or separator for data tokens. For example, you may add \xff between data and next data. Or, you should use fixed data format.