Serial port not reading complete transmission - c++

I attempted to modify Teunis van Beelen's Rs232 library, from Polling to event driven and non-overlapped to suit my project. RS232 Library
I expect to receive blocks of data (roughly 100 to 200 chars) every 200ms.
Problem I am having is the received data is very inconsistent, it is cut off at random points, and incomplete.
I would like ReadFile() to return only after reading one whole block of data.( or something to that effect)
I feel like the problem is with the time out settings, because by altering the figures I get different results, but I just cant get it right, my best result so far has been set all time out values to 0 and let ReadFile() expect 150 bytes, this way ReadFile() dose not return unless it reads 150 chars, but this just go out of sync after few transmissions, as I have no idea how much data to expect.
these are the main changes to the polling function in Teunis's code , besides time out settings, all other settings are unchanged:
//Using the EV_RXCHAR flag will notify the thread that a byte arrived at the port
DWORD dwError = 0;
//use SetCommMask and WaitCommEvent to see if byte has arrived at the port
//SetCommMask sets the desired events that cause a notification.
if(!SetCommMask(Cport[comport_number],EV_RXCHAR)){
printf("SetCommMask Error");
dwError = GetLastError();
// Error setting com mask
return FALSE;
}
//WaitCommEvent function detects the occurrence of the events.
DWORD dwCommEvent;
for( ; ; )
{
//wait for event to happen
if (WaitCommEvent(Cport[comport_number],&dwCommEvent,NULL))
{
if(ReadFile(Cport[comport_number], buf, 1, (LPDWORD)((void *)&n), NULL)){
//Byte has been read, buf is processed in main
}
else{
//error occoured in ReadFile call
dwError = GetLastError();
break;
}
else{
//error in WaitCommEvent
break;
}
break; //break after read file
}
attempt 2 as suggested by MSDN article on serial com using Do While to cycle through every character in the buffer, this method did not yield any good results either.
DWORD dwError = 0;
/*
Using the EV_RXCHAR flag will notify the thread that a byte arrived at the port
*/
//use SetCommMask and WaitCommEvent to see if byte has arrived at the port
//SetCommMask sets the desired events that cause a notification.
if(!SetCommMask(Cport[comport_number],EV_RXCHAR)){
printf("SetCommMask Error");
dwError = GetLastError();
// Error setting com mask
return FALSE;
}
//WaitCommEvent function detects the occurrence of the events.
DWORD dwCommEvent;
for( ; ; )
{
//wait for event to happen
if (WaitCommEvent(Cport[comport_number],&dwCommEvent,NULL))
{
//Do while loop will cycle ReadFile until bytes-read reach 0,
do{
if(ReadFile(Cport[comport_number], buf, size, (LPDWORD)((void *)&n), NULL)){
//Byte has been read, buf is processed in main
}
else{
//error occoured in ReadFile call
dwError = GetLastError();
break;
}
}while(n);
}
else{
//error in WaitCommEvent
break;
}
break; //break after read file
}
I am wondering if rewriting the code in overlapped mode will improve things, but I dont see the advantages as I have no need for multi threading. any suggestions would be great!
Thank you.

ReadFile has no way to detect what a "block of data" is. You should not expect it to understand your data or the timing of that data. The only fix for this issue is for you to process whatever it gives you, using your own knowledge of the data to divide it up into "blocks" for further processing. If you get a partial block keep it, and append to it with the next read.
There is no need to call WaitCommEvent for data. ReadFile will wait for data. But give it a suitably sized buffer and ask for a lot more than one byte at a time. It's extremely inefficient to call it for only one byte. Select the requested count and the timeouts so that ReadFile will return within an acceptable time, whether there is data or not.

Related

Epoll reverse proxy stuck while writing client

I am trying to write reverse proxy with nonblocking socket and epoll. That seems ok at first, but when I tried to open a big jpg file, I got stuck.
When I try to write into client sometimes It may not writable and how can I handle proper way.
Additional Notes:
this->getFd() = ProxyFd
this->clientHandler->getFd = clientFd
I am using EPOLLET flag both proxy and client
if( (flag & EPOLLIN) ){
char buffer[1025] = {'\0'};
int readSize;
while( (readSize = read(this->getFd(),buffer,1024)) > 0){
this->headerParse(buffer);
this->readSize += readSize;
int check = 0;
do{
check = write(this->clientHandler->getFd(),buffer,readSize);
}while(check < 0);
}
if(this->headerEnd == 1 && this->readSize >= this->headerLenght ){
close(this->clientHandler->getFd());
close(this->getFd());
delete this->clientHandler;
delete this;
}
}
Thanks for taking time to read.
Assuming your headerParse() method doesn't change buffer in a size-extending way (you'd need to update readsize, at least, not to mention the buffer full scenario), it seems like your write() path is broken.
if the socket you're writing to is also in nonblocking mode, it's perfectly legal for write() to return -1 (and set errno to EGAIN or EWOULDBLOCK or whatever your platform has) before you wrote all data.
In that case, you must store the remaining data (the remainder of buffer minus what was written if one or more calls to write() succeeded), program epoll to notify the clientHandler->getFd() descriptor for writeability, if not already, and when you get subsequent "write ready" event, you write the data you stored. On this case, the write() can again be unable to flush all your data, so you must cycle until all data is sent.

Microsoft Windows API Serial ReadFile Producing Unexpected Output

I am currently trying to write a program that will read Bluetooth output from an Arduino HC-05 module on a Serial Communications Port.
http://cdn.makezine.com/uploads/2014/03/hc_hc-05-user-instructions-bluetooth.pdf
When I open a Putty terminal and tell it to listen to COM4, I am able to see the output that the program running on the Arduino is printing.
However, when I run the following program to try to process incoming data on the serial port programatically, I get the output shown.
#include <Windows.h>
#include <string>
#include <atltrace.h>
#include <iostream>
int main(int argc, char** argv[]) {
HANDLE hComm = CreateFile(
L"COM4",
GENERIC_READ | GENERIC_WRITE,
0,
0,
OPEN_EXISTING,
NULL,
0
);
if (hComm == INVALID_HANDLE_VALUE) {
std::cout << "Error opening COM4" << std::endl;
return 1;
}
DWORD dwRead;
BOOL fWaitingOnRead = false;
OVERLAPPED osReader = { 0 };
char message[100];
osReader.hEvent = CreateEvent(NULL, TRUE, FALSE, NULL);
if (osReader.hEvent == NULL) {
std::cout << "Error creating overlapping event" << std::endl;
return 2;
}
while (1) {
if (!fWaitingOnRead) {
if (!ReadFile(
hComm,
&message,
sizeof(message),
&dwRead,
NULL
)) {
if (GetLastError() != ERROR_IO_PENDING) {
std::cout << "Communications error" << std::endl;
return 3;
}
}
else {
message[100] = '\0';
std::cout << message << std::endl;
}
}
}
return 0;
}
I have made changes to the handle and the ReadFile function call so that it will be making the calls synchronously in an infinite loop. However, Visual Studio pops up a warning saying that the program has stopped working then asks to debug or close program. My assumption is that it must be stalling somewhere or failing to execute some WindowsAPI function somewhere up the stack.
Any help, pointers, greatly appreciated.
At least IMO, using overlapped I/O for this job is pretty severe overkill. You could make it work, but it would take a lot of extra effort on your part, and probably accomplish very little.
The big thing with using comm ports under Windows is to set the timeouts to at least halfway meaningful values. When I first did this, I started by setting all of the values to 1, with the expectation that this would sort of work, but probably consume excessive CPU time, so I'd want to experiment with higher values to retain fast enough response, while reducing CPU usage.
So, I wrote some code that just set all the values in the COMMTIMEOUTS structure to 1, and setup the comm port to send/read data.
I've never gotten around to experimenting with longer timeouts to try to reduce CPU usage, because even on the machine I was using when I first wrote this (probably a Pentium II, or thereabouts), it was functional, and consumed too little CPU time to care about--I couldn't really see the difference between the machine completely idle, and this transferring data. There might be circumstances that would justify more work, but at least for any need I've had, it seems to be adequate as it is.
That's because message has the wrong type.
To contain a string, it should be an array of characters, not an array of pointers to characters.
Additionally, to treat it as a string, you need to set the array element after the last character to '\0'. ReadFile will put the number of characters it reads into dwRead.
Also, it appears that you are not using overlapped I/O correctly. This simple program has no need for overlapped I/O - remove it. (As pointed out by #EJP, you are checking for ERROR_IO_PENDING incorrectly. Remove that too.)
See comments below, in your program:
if (!fWaitingOnRead) {
if (!ReadFile( // here you make a non-blocking read.
hComm,
message,
sizeof(*message),
&dwRead,
&osReader
)) {
// Windows reports you should wait for input.
//
if (GetLastError() != ERROR_IO_PENDING) {
std::cout << "Communications error" << std::endl;
return 3;
}
else { // <-- remove this.
// insert call to GetOverlappedcResult here.
std::cout << message << std::endl;
}
}
}
return 0; // instead of waiting for input, you exit.
}
After you call ReadFile() you have to insert a call for GetOverlappedResult(hComm, &osReader, &dwBytesRceived, TRUE) to wait for the read operation to complete and have some bytes in your buffer.
You will also need to have a loop in your program if you don't want to exit prematurely.
If you do not want to do overlapped i/o (which is a wise decision) , do not pass an OVERLAPPED pointer to ReadFile. ReadFile will block until it has some data to give you. You will then obviously not need to call GetOverlappedresult()
For the serial port, you also need to fill in a DCB structure. https://msdn.microsoft.com/en-us/library/windows/desktop/aa363214(v=vs.85).aspx
You can use BuildCommDCB()to initialize it. There is a link to it in the MS doc, CallGetCommState(hComm, &dcb) to initialize the serial port hardware. The serial port needs to know which baud rate etc. you need for your app.

recv the first few bytes from a socket to determine buffer size

I'm writing a distributed system in c++ using TCP/IP and sockets.
For each of my messages, I need to receive the first 5 bytes to know the full length of the incoming message.
What's the best way to do this?
recv() only 5 bytes, then recv() again. if I choose this, would it be safe to assume I'll get 0 or 5 bytes in the recv (aka not write a loop to keep trying)?
use MSG_PEEK
recv() some larger buffer size, then read the first 5 bytes and allocate the final buffer then.
You don't need to know anything. TCP is a stream protocol, and at any given moment you can get as little as one byte, or as much as multiple megabytes of data. The correct and only way to use a TCP socket is to read in a loop.
char buf[4096]; // or whatever
std::deque<char> data;
for (int res ; ; )
{
res = recv(fd, buf, sizeof buf, MSG_DONTWAIT);
if (res == -1)
{
if (errno == EAGAIN || errno == EWOULDBLOCK)
{
break; // done reading
}
else
{
// error, break, die
}
}
if (res == 0)
{
// socket closed, finalise, break
}
else
{
data.insert(data.end(), buf, buf + res);
}
}
The only purpose of the loop is to transfer data from the socket buffer to your application. Your application must then decide independently if there's enough data in the queue to attempt extraction of some sort of higher-level application message.
For example, in your case you would check if the queue's size is at least 5, then inspect the first five bytes, and then check if the queue holds a complete application message. If no, you abort, and if yes, you extract the entire message and pop if off from the front of the queue.
Use a state machine with two states:
First State.
Receive bytes as they arrive into a buffer. When there are 5 or more bytes perform your check on those first 5 and possibly process the rest of the buffer. Switch to the second state.
Second State.
Receive and process bytes as they arrive to the end of the message.
to answer your question specifically:
it's not safe to assume you'll get 0 or 5. it is possible to get 1-4 as well. loop until you get 5 or an error as others have suggested.
i wouldn't bother with PEEK, most of the time you'll block (assuming blocking calls) or get 5 so skip the extra call into the stack.
this is fine too but adds complexity for little gain.

read handle problem

I am working on network programming using epoll and I have this code...
int read = read(socket, buf, bufsize);
I have a huge buffer size and I assumed it will receive everything clients sent.
However, I started facing problems like packet segmentation.
One example is that if a client sent 500 bytes but it somehow got into two 250 bytes packets then there is no way to handle this situation.
I looked up online and found this code
int handle_read(client *cli, struct epoll_event *ev) {
size_t len = 4096;
char *p;
ssize_t received;
cli->state = 1;
if (cli->buffer != NULL) {
//free(cli->buffer);
//printf("Buff not null %s\n", cli->buffer);
}
//allocate space for data
cli->buffer = (char*)malloc( (size_t)(sizeof(char) * 4096) );
p = cli->buffer;
do { //read until loop conditions
received = recv(ev->data.fd, p, len, 0);
if (received < 0 && errno != EAGAIN && errno != EWOULDBLOCK) {
//if error, remove from epoll and close socket
printf("Handle error!!!\nClient disconnected!\n");
epoll_ctl(epollfd, EPOLL_CTL_DEL, ev->data.fd, ev);
close(ev->data.fd);
}
p = &cli->buffer[received];
} while (received >= len && errno != EAGAIN && errno != EWOULDBLOCK);
return received;
}
Do you guys think it handles all the exceptions might happen while receiving? Also could you please provide me tutorials or examples that handles socket exceptions? Sample codes online don't cover details.. Thanks in advance
recv can return any of three things, and your code needs to handle each one correctly:
1) Positive number. This means it read some bytes.
2) Negative number. This means an "error" occurred.
3) Zero. This means the other end of the connection performed a successful shutdown() (or close()) on the socket. (In general, a return of 0 from read() or recv() means EOF.)
The "error" case further breaks down into "EAGAIN or EWOULDBLOCK" and "everything else". The first two just means it is a non-blocking socket and there was no data to give you at this time. You probably want to go back and call poll() (or select() or epoll()) again to avoid busy waiting...
"Everything else" means a real error. You need to handle those too; see the POSIX spec for recv() for a complete list.
Given all this, I would say your sample code is bad for several reasons. It does not handle 0 (closed connection) properly. It does not handle any errors. It does a busy-loop when the recv() returns EAGAIN/EWOULDBLOCK.
Oh, and it uses sizeof(char), which is a sure sign it was written by somebody who is not familiar with the C or C++ programming languages.
You can't know "How many datas client sent" in normaly. you should use scalable data format(that have data length in the header) or separator for data tokens. For example, you may add \xff between data and next data. Or, you should use fixed data format.

Adjust parameters of serial port reading

I'm facing a particular issue that regards serial communication under win32.
I'm communicating with a device can only accept frames when it is not already communicating. So I must find a valid frame and then inmediatelly send my request.
I developed a class named Serial that handles basic operations on serial port (open, close, read, write) and then a Thread calls inside a loop read and write functions.
Thread loop
//Device is an object of class Serial
while( device->isOpen() && !terminate )
{
unsigned int readed = 0;
unsigned long error = ERROR_SUCCESS;
unsigned char* data = device->read( &readed, &error );
if( error==ERROR_SUCCESS )
{
//If data received, deliver to upper level
if( readed>0 )
{
QByteArray output( (const char*)data, (signed int)readed );
emit dataArrived( output, readed );
}
}
else
{
//unrelated stuff
}
//Here I manage the writting issue
//Only when nothing is received, and Upper layer wants to send a frame
//(Upper layer only will mark as something to send when it detects a valid frame)
if( readed==0 )
{
out_lock.lock();
//If something to send...
if( something_to_send > 0 )
{
if( device->write( output_buffer, output_size, &error ) )
{ //things...
}
}
}
}
The Thread basically keeps reading, and when nothing is received, sees if somebody has signaled to send a frame (this means that a valid frame is just received).
When this happens, it writes the frame through serial port.
Here comes my problem.
Inside the Serial::read() function:
I use the overlapped way of reading:
::ClearCommError( handle, &dwErrors, &stat);
if( stat.cbInQue )
{
//If there's something to read, read it, please note the bytes to read parameter, here 1.
bool ok = ::ReadFile( handle, buffer_in, 1, &bytes_read, &ov_reader );
if( !ok )
{
DWORD _error = ::GetLastError();
if( _error == ERROR_IO_PENDING )
{
DWORD result = ::WaitForMultipleObjects( 2, waiters, FALSE,INFINITE );
switch( result )
{ //Eventshutdown
case WAIT_OBJECT_0: /*code omitted*/break;
case WAIT_OBJECT_0+1: ok = ::GetOverlappedResult( handle, &ov_reader, &bytes_read, true );
//check ok value omitted
break;
}
}
}
}
if( bytes_read>0 )
{
*size = bytes_read;
}
Here starts my problem.
When device sends me small frames (around 30 bytes) everything works fine, but when larger frames are sent, the code is not able to find any free time between frames causing the thread to never be able send any frame because readed is never 0.
If I increase the number of bytes to read inside the read() function, lose the ability to detect when the device "listens":
bool ok = ::ReadFile(handle, buffer_in, 50, &bytes_read, &ov_reader );
This happens because my app can receive the end of a frame together with the start of the next one. This behaviour is very common.
In the other hand, if I change the INFINITE argument by a valid timeout in the WaitForMultipleObjects function, I lose data.
So my question basically is... what I'm doing wrong? Why when reading 1 byte each time I don't find any free time to send my own frames?
Thank you
I'm not sure if this will help or not, but since you already have a good idea of how many bytes are in the serial device's input queue (stat.cbInQue) maybe it would help to read in that many bytes instead of just 1 byte or an arbitrary number of bytes (like 50):
bool ok = ::ReadFile( handle, buffer_in, stat.cbInQue, &bytes_read, &ov_reader );
Of course, you'd need to make sure that the buffer_in had the capacity for that number of bytes, so there might be some other logic you'd have to add to make sure there's no buffer overruns.
Also, because the serial driver and ReadFile() APIs depend heavily on buffering for handling received characters, you might be able to get more precise indications of when characters have been received (and not received) using the
WaitCommEvent() and SetCommMask() APIs.
How big are the "larger frames?" When you call ReadFile one byte at a time, it will obviously take a long time to work through the whole frame, probably longer than it takes the send the frame itself due to call-overhead.
Some alternatives:
Does the device send frames whenever it feels like it? If you have the opportunity to design both ends of the protocol, can you switch to a command/response style of communication?
Can you, from the start of the packet, predict the number of characters in the rest of the packet? If so, you could build a state machine into your read function. You could poll one byte at a time, then when you detect the start of the packet read most of the rest of the packet on one call, then switch back to a byte at a time.
Can you use DSR/CTS to control the timing?
In general, it's really hard to read whole packets from within a serial-port read function. The usual procedure is the read a bunch of characters and pass them up to a higher level for protocol parsing. It sounds like you have to have tighter timing control than that method allows though. Good luck...