socket: multithreading doesn't work when client reads messages - c++

I have a server, which can accept two socket connections. It creates a thread for each socket so that messages can be sent parallel.
Now I'm trying to code my client.
I create a class named SocketThread as a thread of socket. Here is the main code:
void SocketThread::ReadData()
{
int n = 0;
while (!finished)
{
while ((n = read(sockfd, recvBuff, sizeof(Data))) > 0)
{
std::cout<<std::this_thread::get_id()<<std::endl;
std::this_thread::sleep_for(std::chrono::milliseconds(2000));
}
}
}
std::thread SocketThread::run()
{
return std::thread([=] { ReadData(); });
}
in the function main:
SocketThread s0("127.0.0.1", 10000);
SocketThread s1("127.0.0.1", 10000);
std::thread td0{sts[0].run()};
std::thread td1{sts[1].run()};
td0.join(); // stop here
td1.join();
// something else
When I execute the program, it will block at td0.join();, meaning that I can get the id of the thread td0 on the console and I can NEVER get the other thread.
However, when I remove (n = read(sockfd, recvBuff, sizeof(Data))) > 0, which means that now the client is just a simple thread, that it won't receive anything, things gonna be fine ---- I can get two ids of the two threads.
Why?
EDIT
It seems that I used join incorrectly.
What I need is that main doesn't execute //something else until the two threads get 1000 characters together.
What should I do?

You did not use join() incorrectly. If you want main() to block until both threads end, your code is correct : td0.join() will block until thread td0 ends, and the same for td1.
Now, if you want your threads to end after receiving sizeof(Data) bytes, your function void SocketThread::ReadData() should rather look like this :
void SocketThread::ReadData()
{
int n, total = 0;
while (!finished)
{
while ((n = read(sockfd, &recvBuff[total], sizeof(Data) - total)) > 0)
{
total += n;
}
if (n == -1)
{
// manage error here
}
if (n == 0)
{
std::cout << "client shut the socket down; got " << total << " bytes over " << sizeof(Data) << std::endl;
finished = true;
}
}
}
For a short explanation : there is no guarantee that you can get all data sent by client in a single read() operation, so you need to call read() and cumulate data into the buffer until you get a return value of 0 (meaning the client shut down the socket). read(sockfd, &recvBuff[total], sizeof(Data) - total) ensures that the incomming data is properly appended at the right position in the buffer.

Related

SocketCAN with C++ on Raspberry Pi: messages lost when read is delayed

In a C++ application running on a Raspberry Pi, I am using a loop in a thread to continuously wait for SocketCAN messages and process them. The messages come in at around 1kHz, as verified using candump.
After waiting for poll() to return and reading the data, I read the timestamp using ioctl() with SIOCGSTAMP. I then compare the timestamp with the previous one, and this is where it gets weird:
Most of the time, the difference is around 1ms, which is expected. But sometimes (probably when the data processing takes longer than usual or gets interrupted by the scheduler) it is much bigger, up to a few hundred milliseconds. In those instances, the messages that should have come in in the meantime (visible in candump) are lost.
How is that possible? If there is a delay somewhere, the incoming messages get buffered? Why do they get lost?
This is the slightly simplified code:
while(!done)
{
struct pollfd fd = {.fd = canSocket, .events = POLLIN};
int pollRet = poll(&fd, 1, 20); // 20ms timeout
if(pollRet < 0)
{
std::cerr << "Error polling canSocket" << errno << std::endl;
done = true;
return;
}
if(pollRet == 0) // timeout, never happens as expected
{
std::cout << "canSocket poll timeout" << std::endl;
if(done) break;
continue;
}
struct canfd_frame frame;
int size = sizeof(frame);
int readLength = read(canSocket, &frame, size);
if(readLength < 0) throw std::runtime_error("CAN read failed");
else if(readLength < size) throw std::runtime_error("CAN read incomplete");
struct timeval timestamp;
ioctl(canSocket, SIOCGSTAMP, &timestamp);
uint64_t timestamp_us = (uint64_t)timestamp.tv_sec * 1e6 + (uint64_t)timestamp.tv_usec;
static uint64_t timestamp_us_last = 0;
if((timestamp_us - timestamp_us_last) > 20000)
{
std::cout << "timestamp difference large: " << (timestamp_us - timestamp_us_last) << std::endl; // this sometime happens, why?
}
timestamp_us_last = timestamp_us;
// data processing
}

While loop breaks but the code after it is never executed

Here is the method where the while loop resides. I am just connecting to a server , sending an HTTP Request and reading the response. When I debug, I still cannot see why this while loop is not passed.
void HttpSocket::Get(string address)
{
std::string response, host, httpRequest;
uint32_t ipAddress;
ParseRequest(address, host, httpRequest);
ResolveHostAddress(host, ipAddress);
HttpSocket::Connect(ipAddress);
strcpy(bufferToSend, httpRequest.c_str());
n = write(sockfd,bufferToSend,strlen(bufferToSend));
if (n < 0) { throw IO_Exception("Cannot send request."); }
memset(bufferToSend, 0, 500);
memset(bufferToReceive, 0, 200);
n = read(sockfd,bufferToReceive,200);
if (n <= 0){
throw IO_Exception("Cannot read response.");
}
else
{
response += bufferToReceive;
while(n != 0)
{
n = 0;
memset(bufferToReceive, 0, 200);
n = read(sockfd,bufferToReceive,200);
response += bufferToReceive;
cout << "still in the loop" << n << endl;
}
cout << "Response: " << response << endl;
}
}
By the way n is volatile int thus, I don't think compiler optimization causes it. FYI, Everything is fine and working until the last while loop.
::read() is a synchronous function. When there is nothing left on your socket to be read, the call to n = read(sockfd,bufferToReceive,200); just hangs blocks waiting new information.
To solve your problem, you should set the socket as non-blocking and read it with ::recv() which would return -E_WOULDBLOCK if there is no data available:
#include <fcntl.h>
flags = ::fcntl(fd, F_GETFL, 0);
flags |= O_NONBLOCK;
::fcntl(fd, F_SETFL, flags);
An alternative would be to check for available data before reading the file descriptor:
#include <sys/ioctl.h>
int count;
::ioctl(fd, FIONREAD, &count);

Do I need to close fds when reading and writing to the pipe among threads created using pthread_create?

There is a client server application I am working on. Below is the code from client side.
pipe_input, pipe_output are shared variables.
int fds[2];
if (pipe(fds)) {
printf("pipe creation failed");
} else {
pipe_input = fds[0];
pipe_output = fds[1];
reader_thread_created = true;
r = pthread_create(&reader_thread_id,0,reader_thread,this);
}
void* reader_thread(void *input)
{
unsigned char id;
int n;
while (1) {
n = read(pipe_input , &id, 1);
if (1 == n) {
//process
}if ((n < 0) ) {
printf("ERROR: read from pipe failed");
break;
}
}
printf("reader thread stop");
return 0;
}
There is a writer thread also which writes data on event change from server.
void notify_client_on_event_change(char id)
{
int n;
n= write(pipe_output, &id, 1);
printf("message written to pipe done ");
}
My question is do I need to close the write end in reader thread and read end in case of writer thread. In the destructor, I am waiting for reader thread to exit but sometimes it doesn't exit from reader thread.
[...] do i need to close the write end in reader thread and read end in case of writer thread[?]
As those fds "are shared", closing them in one thread would close them for all threads. That is not what you want, I suspect.

Winsock RIO: RIOReceive returns immediately with no bytesTransferred

I'm having problems with getting winsock RIO working.
It seems that every time I post a RIOReceive it returns immediately with 0 bytes transferred, and my peer can't get a message through.
After posting a RIOReceive, I wait on the RIODequeCompletion, which deques immediately with numResults = 1, but when I inspect the bytesTransferred of the RIORESULT struct, it's 0. This tells me that I'm not setting this thing up properly, but I can't find docs or examples that tell me what else I should be doing.
The internet seems to have very little on RIO. I've looked through MSDN, Len Holgate with TheServerFramework, this site, and two GitHub RIO servers.
RIOEchoServer and RIOServer_sm9 are on GitHub, but I can't post more than two links (this is my first question on this site).
This code is just to get things proven. It's currently not set to use the sendCQ, doesn't handle errors well, etc...
Here's the prep work:
void serverRequestThread() {
//init buffers
//register big buffers
recvBufferMain = rio.RIORegisterBuffer(pRecvBufferMain, bufferSize);
sendBufferMain = rio.RIORegisterBuffer(pSendBufferMain, bufferSize);
if (recvBufferMain == RIO_INVALID_BUFFERID) {
cout << "RIO_INVALID_BUFFERID" << endl;
}
if (sendBufferMain == RIO_INVALID_BUFFERID) {
cout << "RIO_INVALID_BUFFERID" << endl;
}
//create recv buffer slice
recvBuffer1.BufferId = recvBufferMain;
recvBuffer1.Offset = 0;
recvBuffer1.Length = 10000;
//create send buffer slice
sendBuffer1.BufferId = sendBufferMain;
sendBuffer1.Offset = 0;
sendBuffer1.Length = 10000;
//completion queue
recvCQ = rio.RIOCreateCompletionQueue(CQsize, NULL);
sendCQ = rio.RIOCreateCompletionQueue(CQsize, NULL);
if (recvCQ == RIO_INVALID_CQ) {
cout << "RIO_INVALID_CQ" << endl;
}
if (sendCQ == RIO_INVALID_CQ) {
cout << "RIO_INVALID_CQ" << endl;
}
//start a loop for newly accept'd socket
while (recvCQ != RIO_INVALID_CQ && sendCQ != RIO_INVALID_CQ) {
//get accept'd socket
struct sockaddr_in saClient;
int iClientSize = sizeof(saClient);
acceptSocket = accept(listenSocket, (SOCKADDR*)&saClient, &iClientSize);
if (acceptSocket == INVALID_SOCKET) {
cout << "Invalid socket" << endl;
printError();
}
//register request queue
requestQueue = rio.RIOCreateRequestQueue(
acceptSocket, //socket
10, //max RECVs on queue
1, //max recv buffers, set to 1
10, //max outstanding sends
1, //max send buffers, set to 1
recvCQ, //recv queue
recvCQ, //send queue
pOperationContext //socket context
);
if (requestQueue == RIO_INVALID_RQ) {
cout << "RIO_INVALID_RQ" << endl;
printError();
}
I now post a RIOReceive:
//start a loop to repin recv buffer for socket
while (acceptSocket != INVALID_SOCKET) {
//pin a recv buffer to wait on data
recvSuccess = rio.RIOReceive(
requestQueue, //socketQueue
&recvBuffer1, //buffer slice
1, //set to 1
RIO_MSG_WAITALL, //flags
0); //requestContext
if (recvSuccess == false) {
cout << "RECV ERROR!!!!!!!!\n";
printError();
}
//wait for recv to post in queue
//std::this_thread::sleep_for(std::chrono::milliseconds(3000));
As soon as I call RIODequeCompletion, it returns 1:
numResults = 0;
while (numResults == 0) numResults = rio.RIODequeueCompletion(recvCQ, recvArray, 10);
if (numResults == RIO_CORRUPT_CQ) {
cout << "RIO_CORRUPT_CQ" << endl;
} else if (numResults == 0) {
cout << "no messages on queue\n";
} else if (numResults > 0) {
but when I inspect the bytesTransferred of the RIORESULT, it's always 0:
if (recvArray[0].BytesTransferred > 0) {
//process results
if (pRecvBufferMain[0] == 'G') {
//set respnose html
strcpy(pSendBufferMain, responseHTTP);
sendSuccess = rio.RIOSend(
requestQueue, //socketQueue
&sendBuffer1, //buffer slice
1, //set to 1
0, //flags
0); //requestContext
} else if (pRecvBufferMain[0] == 'P') {
//process post
} else {
//recv'd a bad message
}
} //end bytesTransferred if statement
//reset everything and post another recv
}//end response if statement
std::this_thread::sleep_for(std::chrono::milliseconds(100));
}//end while loop for recv'ing
std::this_thread::sleep_for(std::chrono::milliseconds(100));
}//end while loop for accept'ing
}// end function
Like I said, I'm probably not using RIOReceive correctly, and/or I'm not setting the correct socket options that I need to (none right now).
I appreciate any help with this.
Try removing RIO_MSG_WAITALL. There may be a bug whereby you're only getting the close notification (bytes == 0) rather than getting a completion with the data in it. Either way it would be interesting to see if the code works without the flag.
Do my example servers and tests work on your hardware?
I encountered a similar issue of having zero bytesReceived in my dequeued completion result while using RioReceive with RioNotify and RioDequeueCompletion. I would also see the 'Status' value of WSAEINVAL (Invalid Parameter = 10022) in my dequeued completion result, this seems to indicate the WSA error code for the Receive call.
The particular reason I had the error is because I had allocated memory for a receiveBuffer and I was trying to pass that buffer pointer as my buffer handle in the RIO_BUFFER_SEGMENT given to RioReceive instead of passing the IntPtr returned by RioRegisterBuffer.
I fully blame myself for using too many untyped IntPtrs and losing type checking. :)

timerfd and read

I have application, that periodically (by timer) check some data storage.
Like this:
#include <iostream>
#include <cerrno>
#include <cstring>
#include <cstdlib>
#include <sys/fcntl.h>
#include <unistd.h>
// EPOLL & TIMER
#include <sys/epoll.h>
#include <sys/timerfd.h>
int main(int argc, char **argv)
{
/* epoll instance */
int efd = epoll_create1(EPOLL_CLOEXEC);
if (efd < 0)
{
std::cerr << "epoll_create error: " << strerror(errno) << std::endl;
return EXIT_FAILURE;
}
struct epoll_event ev;
struct epoll_event events[128];
/* timer instance */
int tfd = timerfd_create(CLOCK_MONOTONIC, TFD_CLOEXEC);
struct timespec ts;
// first expiration in 3. seconds after program start
ts.tv_sec = 3;
ts.tv_nsec = 0;
struct itimerspec new_timeout;
struct itimerspec old_timeout;
bzero(&new_timeout, sizeof(new_timeout));
bzero(&old_timeout, sizeof(old_timeout));
// value
new_timeout.it_value = ts;
// no interval;
// timer will be armed in epoll_wait event trigger
new_timeout.it_interval.tv_sec =
new_timeout.it_interval.tv_nsec = 0;
// Add the timer descriptor to epoll.
if (tfd != -1)
{
ev.events = EPOLLIN | EPOLLERR /*| EPOLLET*/;
ev.data.ptr = &tfd;
epoll_ctl(efd, EPOLL_CTL_ADD, tfd, &ev);
}
int flags = 0;
if (timerfd_settime(tfd, flags, &new_timeout, &old_timeout) < 0)
{
std::cerr << "timerfd_settime error: " << strerror(errno) << std::endl;
}
int numEvents = 0;
int timeout = 0;
bool checkTimer = false;
while (1)
{
checkTimer = false;
numEvents = epoll_wait(efd, events, 128, timeout);
if (numEvents > 0)
{
for (int i = 0; i < numEvents; ++i)
{
if (events[i].data.ptr == &tfd)
{
std::cout << "timeout" << std::endl;
checkTimer = true;
}
}
}
else if(numEvents == 0)
{
continue;
}
else
{
std::cerr << "An error occured: " << strerror(errno) << std::endl;
}
if (checkTimer)
{
/* Check data storage */
uint64_t value;
ssize_t readBytes;
//while ( (readBytes = read(tfd, &value, 8)) > 0)
//{
// std::cout << "\tread: '" << value << "'" << std::endl;
//}
itimerspec new_timeout;
itimerspec old_timeout;
new_timeout.it_value.tv_sec = rand() % 3 + 1;
new_timeout.it_value.tv_nsec = 0;
new_timeout.it_interval.tv_sec =
new_timeout.it_interval.tv_nsec = 0;
timerfd_settime(tfd, flags, &new_timeout, &old_timeout);
}
}
return EXIT_SUCCESS;
}
This is simple description of my app.
After each timeout timer need to be rearmed by some value different in each timeout.
Questions are:
Is it necessary to add timerfd to epoll (epoll_ctl) with EPOLLET flag?
Is it necessary to read timerfd after each timeout?
Is it necessary to epoll_wait infinitely (timeout = -1)?
You can do this in one of two modes, edge triggered or level triggered. If you choose the edge triggered route then you must pass EPOLLET and do not need to read the timerfd after each wakeup. The fact that you receive an event from epoll means one or more time outs have fired. Optionally you may read the timerfd and it will return the number of time outs that have fired since you last read it.
If you choose the level triggered route then you don't need to pass EPOLLET, but you must read the timerfd after each wakeup. If you do not then you will immediately be woken up again until you consume the time out.
You should either pass -1 to epoll as the time out or some positive value. If you pass 0, like you do in the example, then you will never go to sleep, you'll just spin waiting for the time out to fire. That's almost certainly undesirable behaviour.
Answers to the questions:
Is it necessary to add timerfd to epoll (epoll_ctl) with EPOLLET flag?
No. Adding EPOLLET (edge trigger) does changes the behavior of receiving events. Without EPOLLET, you'll continuously receive the event from epoll_wait related to the timerfd until you've read() from the timerfd. With EPOLLET, you'll NOT receive additional events beyond the first one, even if new expiration occurs, until you've read() from the timerfd and a new expiration occur.
Is it necessary to read timerfd after each timeout?
Yes in order to continue and receive events (only) when new expiration occur (see above). No when periodic timer is not used (single expiration only), and you close the timerfd without reading.
Is it necessary to epoll_wait infinitely (timeout = -1)?
No. You can use epoll_wait's timeout instead of timerfd. I personally think it is easier to use timerfd than keep calculating the next timeout for EPOLL, especially if you expect multiple timeout intervals; keeping tabs on what is your next task when timeout occurs is much easier when it is tied to the specific event what woke up.