Serial communication read not working at all time - c++

I am writing a c++ application for half duplex communication to download data from a device. Following is the class i am using for serial communication.
class CSerialCommHelper
{
HANDLE m_pPortHandle; //Handle to the COM port
HANDLE m_hReadThread; //Handle to the Read thread
HANDLE m_hPortMutex; //Handle to Port Mutex
std::wstring m_strPortName; //Portname
COMMTIMEOUTS m_CommTimeouts; //Communication Timeout Structure
_DCB dcb; //Device Control Block
DWORD m_dwThreadID; //Thread ID
string m_strBuffer;
public:
CSerialCommHelper();
HRESULT Open();
HRESULT ConfigPort();
static void * ReadThread(void *);
HRESULT Write(const unsigned char *,DWORD);
string GetFrameFromBuffer();
HRESULT Close();
~CSerialCommHelper(void);
};
ReadThread and Write function is as follows :
void * CSerialCommHelper::ReadThread(void * pObj)
{
CSerialCommHelper *pCSerialCommHelper =(CSerialCommHelper *)pObj;
DWORD dwBytesTransferred =0;
DWORD byte=0;;
while (pCSerialCommHelper->m_pPortHandle != INVALID_HANDLE_VALUE)
{
pCSerialCommHelper->m_strBuffer.clear();
pCSerialCommHelper->m_usBufSize=0;
WaitForSingleObject(pCSerialCommHelper->m_hPortMutex,INFINITE);
do
{
dwBytesTransferred = 0;
ReadFile (pCSerialCommHelper->m_pPortHandle,&byte,1,&dwBytesTransferred,NULL);
if (dwBytesTransferred == 1)
{
pCSerialCommHelper->m_strBuffer.push_back((char)byte);
pCSerialCommHelper->m_usBufSize++;
continue;
}
}
while ((dwBytesTransferred == 1) && (pCSerialCommHelper->m_pPortHandle != INVALID_HANDLE_VALUE));
ReleaseMutex(pCSerialCommHelper->m_hPortMutex);
Sleep(2);
}
ExitThread(0);
return 0;
}
Write function waits for readthread to release mutex and writes to data to port .
GetFrameFromBuffer will be called from application which uses the SerialCommhelper
and it returns the m_strBuffer string .
My problem is whenever i am trying to download huge amount of data.
I am losing some data frames .
I am getting response from device in between .0468 to .1716 secs.
After analysing different error scenarios i came to know that is not problem with time as other frames are getting downloaded at the same time interval.
Function which is calling getframebuffer is continuosly calling it until is gets a filled string.

It seems like these two statements should not be in your outer while loop:
pCSerialCommHelper->m_strBuffer.clear();
pCSerialCommHelper->m_usBufSize=0;
Your inner while loop reads bytes as long as they're immediately available, and the outer loop does a Sleep(2) the moment the inner loop doesn't give you a byte.
If you're waiting until an entire packet is available, it seems like you should keep looping until you get all the bytes, without clearing partway through the process.
I don't really know the ReadFile API, but I'm guessing that ReadFile might return 0 if there's no bytes immediately available, or at least available by whatever timeout you specified when opening the serial device.

ReleaseMutex(pCSerialCommHelper->m_hPortMutex);
Sleep(2);
That Sleep() call is hiding the real problem. It is never correct in threaded code, always a band-aid for a timing bug.
You certainly seem to have one, that m_hPortMutex spells doom as well. If you do in fact have multiple threads trying to read from the serial port then they are going to start fighting over that mutex. The outcome will be very poor, each thread will get a handful of bytes from the port. But clearly you want to read a frame of data. There is zero hope that you can glue the handfuls of bytes that each thread gets back together into a frame, you've lost their sequence. So sleeping for a while seemed like a workaround, it inject delays that can give you a better shot at reading a frame. Usually, not always. You also wrote it in the wrong place.
This code is just broken. Delete the Sleep(). Do not exit the loop until you've read the entire frame.

Related

Rearrangement of packets using pthreads

So this is the first time I'm actually asking a question in here, although I have been using this site for ages!
My problem is a bit tricky. I'm trying to develop a client server application for sending large files, using UDP with my own error checking and flow control. Now, I've developed a fully-functioning server and client. Client requests for a specific file, server starts sending. The file is read in parts into a buffer to avoid having to read small bits of the file every time a packet is send, thus saving processing time. Packets consist of 1400 bytes of actual data + a header of 28 bytes (sequence numbers, ack numbers, checksum etc..).
So I had the basics down, a simple stop-and-wait protocol. Send packet and receive ack, before sending next packet.
To be able to implement a smarter flow control algorithm, for starters with just some windowing, I have to run the sending-part and receiving-ack part in two different threads. Now here's where I got into problems. This is my first time working with threads, so please bear with me.
My problem is that the file written from the packets on the client side is corrupt. Well, when testing with a small jpg file, the file is only corrupt 50% of the times, when testing with a MP4 file, it's always corrupt! So I guess maybe the thread somehow rearranges the order in which the packets are send? I use sequence numbers, so the problem must occur before assigning the sequence number to the packets...
I know for sure that the part where I split up the file is correct, and also where I reassemble it on the client side, since I have tested this before trying to implement the threading. It should also be noted that I copied the exact sending-part of the code into the sending-thread, and this also worked perfectly before putting it into a thread.. This is also why I'm just posting the threading-part of my code, since this is clearly what is creating the problem (and since the entire code of the project would take up a loooot of space)
My sending thread code:
pthread_mutex_t mutex = PTHREAD_MUTEX_INITIALIZER;
pthread_cond_t condition_var = PTHREAD_COND_INITIALIZER;
static void *send_thread(void *){
if (file.is_open()) {
while(!file.reachedEnd()){
pthread_mutex_lock(& mutex);
if(seq <= upperwindow) {
int blocksize = file.getNextBlocksize();
senddata = new unsigned char[blocksize + 28];
Packet to_send;
to_send.data = new char[blocksize];
to_send.sequenceNumber = seq;
to_send.ackNumber = 0;
to_send.type = 55; // DATA
file.readBlock(*to_send.data);
createPacket(senddata, to_send, blocksize + 28);
if (server.sendToClient(reinterpret_cast<char*>(senddata), blocksize + 28) == -1)
perror("sending failed");
incrementSequenceNumber(seq);
/* free memory */
delete [] to_send.data;
delete [] senddata;
}
pthread_mutex_unlock(& mutex);
}
pthread_exit(NULL);
} else {
perror("file opening failed!");
pthread_exit(NULL);
}
}
My receiving ack thread code:
static void *wait_for_ack_thread(void *){
while(!file.reachedEnd()){
Packet ack;
if (server.receiveFromClient(reinterpret_cast<char*>(receivedata), 28) == -1) {
perror("error receiving ack");
} else {
getPacket(receivedata, ack, 28);
pthread_mutex_lock(& mutex);
incrementSequenceNumber(upperwindow);
pthread_mutex_unlock(& mutex)
}
}
pthread_exit(NULL);
}
All comments are very much appreciated! :)
EDIT:
Added code of the readBlock function:
void readBlock(char & in){
memcpy(& in, buffer + block_position, blocksize);
block_position = block_position + blocksize;
if(block_position == buffersize){
buf_position ++;
if(buf_position == buf_reads){
buffersize = filesize % buffersize;
}
fillBuffer();
block_position = 0;
}
if(blocksize < MAX_DATA_SIZE){
reached_end = true;
return;
}
if((buffersize - block_position) < MAX_DATA_SIZE){
blocksize = buffersize % blocksize;
}
}
Create an array that represents the status of the communication.
0 means unsent, or sent and receiver reported error. 1 means sending. 2 means sent, and ack gotten.
Allocate this array, and guard access to it with a mutex.
The sending thread keeps two pointers into the array -- "has been sent up to" and "should sent next". These are owned by the sending thread.
The ack thread simply gets ack packets, locks the array, and does the transition on the state.
The sending thread locks the array, checks if it can advance the "has been sent up to" pointer (or if it should resend old stuff). If it notices an error, it reduces the "should be sent next" pointer to point at it.
It then sees if it should send stuff next. If it should, it marks the node as "being sent", unlocks the array, and sends it.
If the sending thread did no work, and found nothing to do, it goes to sleep on a timeout, and possibly a "kick awake" by the ack thread.
Now, note that the client can get the packets sent by this in the wrong order, unless you limit it to having 1 packet in transit.
The connection status array does not have to be a literal array, but it is easier if you start with that and optimize later.
On the receiving end, you have to pay attention to the sequence number, as the packets can get there out of sequence. To test this, write a server that sends the packets in the wrong order on purpose, and ensure that the client manages to stitch it together properly.

Concurrent server using pthread API

I am writing a simple client-server application using pthread-s API, which in pseudo code
looks something like this:
static volatile sig_atomic_t g_running = 1;
static volatile sig_atomic_t g_threads = 0;
static pthread_mutex_t g_threads_mutex;
static void signalHandler(int signal)
{
g_running = 0;
}
static void *threadServe(void *params)
{
/* Increment the number of currently running threads. */
pthread_mutex_lock(&g_threads_mutex);
g_threads++;
pthread_mutex_unlock(&g_threads_mutex);
/* handle client's request */
/* decrement the number of running threads */
pthread_mutex_lock(&g_threads_mutex);
g_threads--;
pthread_mutex_unlock(&g_threads_mutex);
}
int main(int argc, char *argv[])
{
/* do all the initialisation
(set up signal handlers, listening socket, ... ) */
/* run the server loop */
while (g_running)
{
int comm_sock = accept(listen_socket, NULL, 0);
pthread_create(&thread_id, NULL, &threadServe, comm_sock) ;
pthread_detach(thread_id);
}
/* wait for all threads that are yet busy processing client requests */
while (1)
{
std::cerr << "Waiting for all threads to finish" << std::endl;;
pthread_mutex_lock(&g_threads_mutex);
if (g_threads <= 0)
{
pthread_mutex_unlock(&g_threads_mutex);
break;
}
pthread_mutex_unlock(&g_threads_mutex);
}
/* clean up */
}
So the server is running in an infinite loop until a signal (SIGINT or SIGTERM) is received. The purpose of the second while loop is to let all the threads (that were processing client requests at the time a signal was received) to have a chance to finish the work they already started.
However I don't like this design very much, because that second while loop is basically a busy loop wasting cpu resources.
I tried to search on Google for some good examples on threaded concurrent server, but I had no luck. An idea that came to my mind was to use pthread_cond_wait() istead of that loop, but I am not sure if this does not bring further problems.
So the question is, how to improve my design, or point me to a nice simple example that deals with similar problem as mine.
EDIT:
I was considering pthread_join(), but I din't know how to join with worker thread,
while the main server loop (with accept() call in it) would be still running.
If I called pthread_join() somewhere after pthread_create()
(instead of pthread_detach()), then the while loop would be blocked until the worker
thread is done and the whole threading would not make sense.
I could use pthread_join() if I spawned all the threads at program start,
but then I would have them around for the entire life of my server,
which I thought might be a little inefficient.
Also after reading man page I understood, that pthread_detach() is exactly
suitable for this purpose.
The busy loop slurping CPU can easily be altered by having a usleep(10000); or something like that outside your mutex lock.
It would be more light-weight if you use a std::atomic<int> g_threads; - that way, you could get rid of the mutex altogether.
If you have an array of (active) thread_id's, you could just use a loop of
for(i = 0; i < num_active_threads; i++)
pthread_join(arr[i]);

How to reduce cpu usage during data transfer on TCP ports realtime

I have a socket program which acts like both client and server.
It initiates connection on an input port and reads data from it. On a real time scenario it reads data on input port and sends the data (record by record ) on to the output port.
The problem here is that while sending data to the output port CPU usage increases to 50% while is not permissible.
while(1)
{
if(IsInputDataAvail==1)//check if data is available on input port
{
//condition to avoid duplications while sending
if( LastRecordSent < LastRecordRecvd )
{
record_time temprt;
list<record_time> BufferList;
list<record_time>::iterator j;
list<record_time>::iterator i;
// Storing into a temp list
for(i=L.begin(); i != L.end(); ++i)
{
if((i->recordId > LastRecordSent) && (i->recordId <= LastRecordRecvd))
{
temprt.listrec = i->listrec;
temprt.recordId = i->recordId;
temprt.timestamp = i->timestamp;
BufferList.push_back(temprt);
}
}
//Sending to output port
for(j=BufferList.begin(); j != BufferList.end(); ++j)
{
LastRecordSent = j->recordId;
std::string newlistrecord = j->listrec;
newlistrecord.append("\n");
char* newrecord= new char [newlistrecord.size()+1];
strcpy (newrecord, newlistrecord.c_str());
if ( s.OutputClientAvail() == 1) //check if output client is available
{
int ret = s.SendBytes(newrecord,strlen(newrecord));
if ( ret < 0)
{
log1.AddLogFormatFatal("Nice Send Thread : Nice Client Disconnected");
--connected;
return;
}
}
else
{
log1.AddLogFormatFatal("Nice Send Thread : Nice Client Timedout..connection closed");
--connected; //if output client not available disconnect after a timeout
return;
}
}
}
}
// Sleep(100); if we include sleep here CPU usage is less..but to send data real time I need to remove this sleep.
If I remove Sleep()...CPU usage goes very high while sending data to out put port.
}//End of while loop
Any possible ways to maintain real time data transfer and reduce CPU usage..please suggest.
There are two potential CPU sinks in the listed code. First, the outer loop:
while (1)
{
if (IsInputDataAvail == 1)
{
// Not run most of the time
}
// Sleep(100);
}
Given that the Sleep call significantly reduces your CPU usage, this spin-loop is the most likely culprit. It looks like IsInputDataAvail is a variable set by another thread (though it could be a preprocessor macro), which would mean that almost all of that CPU is being used to run this one comparison instruction and a couple of jumps.
The way to reclaim that wasted power is to block until input is available. Your reading thread probably does so already, so you just need some sort of semaphore to communicate between the two, with a system call to block the output thread. Where available, the ideal option would be sem_wait() in the output thread, right at the top of your loop, and sem_post() in the input thread, where it currently sets IsInputDataAvail. If that's not possible, the self-pipe trick might work in its place.
The second potential CPU sink is in s.SendBytes(). If a positive result indicates that the record was fully sent, then that method must be using a loop. It probably uses a blocking call to write the record; if it doesn't, then it could be rewritten to do so.
Alternatively, you could rewrite half the application to use select(), poll(), or a similar method to merge reading and writing into the same thread, but that's far too much work if your program is already mostly complete.
if(IsInputDataAvail==1)//check if data is available on input port
Get rid of that. Just read from the input port. It will block until data is available. This is where most of your CPU time is going. However there are other problems:
std::string newlistrecord = j->listrec;
Here you are copying data.
newlistrecord.append("\n");
char* newrecord= new char [newlistrecord.size()+1];
strcpy (newrecord, newlistrecord.c_str());
Here you are copying the same data again. You are also dynamically allocating memory, and you are also leaking it.
if ( s.OutputClientAvail() == 1) //check if output client is available
I don't know what this does but you should delete it. The following send is the time to check for errors. Don't try to guess the future.
int ret = s.SendBytes(newrecord,strlen(newrecord));
Here you are recomputing the length of the string which you probably already knew back at the time you set j->listrec. It would be much more efficient to just call s.sendBytes() directly with j->listrec and then again with "\n" than to do all this. TCP will coalesce the data anyway.

After sending a lot, my send() call causes my program to stall completely. How is this possible?

So basically I'm making an MMO server in C++ that runs on linux. It works fine at first, but after maybe 40 seconds with 50 clients it will completely pause. When I debug it I find that basically the last frame its on before it stops responding is syscall() at which point it disappears into the kernel. Once it disappears into the kernel it never even returns a value... it's completely baffling.
The 50 clients are each sending 23 bytes every 250 milliseconds. These 23 bytes are then broadcasted to all the other 49 clients. This process begins to slow down and then eventually comes to a complete halt where the kernel never returns from a syscall for the send() command. What are some possible reasons here? This is truly driving me nuts!
One option I found is Nagles algorithm which forces delays. I've tried toggling it but it still happens however.
Edit: The program is stuck here. Specifically, in the send, which in turn calls syscall()
bool EpollManager::s_send(int curFD, unsigned char buf[], int bufLen, int flag)
// Meant to counteract partial sends
{
int sendRetVal = 0;
int bytesSent = 0;
while(bytesSent != bufLen)
{
print_buffer(buf, bufLen);
sendRetVal = send(curFD, buf + bytesSent, bufLen - bytesSent, flag);
cout << sendRetVal << " ";
if(sendRetVal == -1)
{
perror("Sending failed");
return false;
}
else
bytesSent += sendRetVal;
}
return true;
}
Also this is the method which calls the s_send.
void EpollManager::broadcast(unsigned char msg[], int bytesRead, int sender)
{
for(iMap = connections.begin(); iMap != connections.end(); iMap++)
{
if(sender != iMap->first)
{
if(s_send(iMap->first, msg, bytesRead, 0)) // MSG_NOSIGNAL
{
if(debug)
{
print_buffer(msg, bytesRead);
cout << "sent on file descriptor " << iMap->first << '\n';
}
}
}
}
if(connections.find(sender) != connections.end())
connections[sender]->reset_batch();
}
And to clarify connections is an instance of boost's unordered_map. The data that the program chokes on is not unique in any way either. It has been broadcast successfully to other file descriptors, but chokes on a, at least seemingly, random one.
TCP congestion control, i.e. Nagle's algorithm, along side a full buffer (SO_SNDBUF socket option) will cause the send() and similar operations to block.
The lazy way around this is to implement separate threads for each socket but this does not scale too far. On Linux you should use non-blocking sockets with poll() or similar, with Windows you would investigate IO completion ports. Look at middleware libraries to simplify this, libevent is a popular cross platform example with recent inclusion of Windows IOCP support, alternatively Boost:ASIO for C++.
A useful article to read on IO scalability would be The C10K problem.
Note you really do not want to disable Nagle's on Internet traffic, even on a LAN you might see major problems without some form of congestion feedback.
The kernel keeps a finite buffer for sending data. If the receiver isn't receiving, that buffer will fill up and the sender will block. Could that be the problem?

Speeding up non-blocking Unix Sockets (C++)

I've implemented a simple socket wrapper class. It includes a non-blocking function:
void Socket::set_non_blocking(const bool b) {
mNonBlocking = b; // class member for reference elsewhere
int opts = fcntl(m_sock, F_GETFL);
if(opts < 0) return;
if(b)
opts |= O_NONBLOCK;
else
opts &= ~O_NONBLOCK;
fcntl(m_sock, F_SETFL, opts);
}
The class also contains a simple receive function:
int Socket::recv(std::string& s) const {
char buffer[MAXRECV + 1];
s = "";
memset(buffer,0,MAXRECV+1);
int status = ::recv(m_sock, buffer, MAXRECV,0);
if(status == -1) {
if(!mNonBlocking)
std::cout << "Socket, error receiving data\n";
return 0;
} else if (status == 0) {
return 0;
} else {
s = buffer;
return status;
}
}
In practice, there seems to be a ~15ms delay when Socket::recv() is called. Is this delay avoidable? I've seen some non-blocking examples that use select(), but don't understand how that might help.
It depends on how you using sockets. If you have multiple sockets and you loop over all of them checking for data that may account for the delay.
With non-blocking recv you are depending on data being there. If your application need to use more than one socket you will have to constantly pool each socket in turns to find out if any of them have data available.
This is bad for system resources because it means your application is constantly running even when there is nothing to do.
You can avoid that with select. You basically set up your sockets, add them to group and select on the group. When anything happens on any of the selected sockets select returns specifying what happened and on which socket.
For some code about how to use select look at beej's guide to network programming
select will let you a specify a timeout, and can test if the socket is ready to be read from. So you can use something smaller than 15ms. Incidentally you need to be careful with that code you have, if the data on the wire can contain embedded NULs s won't contain all the read data. You should use something like s.assign(buffer, status);.
In addition to stefanB, I see that you are zeroing out your buffer every time. Why bother? recv returns how many bytes were actually read. Just zero out the one byte after ( buffer[status+1]=NULL )
How big is your MAXRECV? It might just be that you incur a page fault on the stack growth. Others already mentioned that zeroing out the receive buffer is completely unnecessary. You also take memory allocation and copy hit when you create a std::string out of received character data.