timerfd and read - c++

I have application, that periodically (by timer) check some data storage.
Like this:
#include <iostream>
#include <cerrno>
#include <cstring>
#include <cstdlib>
#include <sys/fcntl.h>
#include <unistd.h>
// EPOLL & TIMER
#include <sys/epoll.h>
#include <sys/timerfd.h>
int main(int argc, char **argv)
{
/* epoll instance */
int efd = epoll_create1(EPOLL_CLOEXEC);
if (efd < 0)
{
std::cerr << "epoll_create error: " << strerror(errno) << std::endl;
return EXIT_FAILURE;
}
struct epoll_event ev;
struct epoll_event events[128];
/* timer instance */
int tfd = timerfd_create(CLOCK_MONOTONIC, TFD_CLOEXEC);
struct timespec ts;
// first expiration in 3. seconds after program start
ts.tv_sec = 3;
ts.tv_nsec = 0;
struct itimerspec new_timeout;
struct itimerspec old_timeout;
bzero(&new_timeout, sizeof(new_timeout));
bzero(&old_timeout, sizeof(old_timeout));
// value
new_timeout.it_value = ts;
// no interval;
// timer will be armed in epoll_wait event trigger
new_timeout.it_interval.tv_sec =
new_timeout.it_interval.tv_nsec = 0;
// Add the timer descriptor to epoll.
if (tfd != -1)
{
ev.events = EPOLLIN | EPOLLERR /*| EPOLLET*/;
ev.data.ptr = &tfd;
epoll_ctl(efd, EPOLL_CTL_ADD, tfd, &ev);
}
int flags = 0;
if (timerfd_settime(tfd, flags, &new_timeout, &old_timeout) < 0)
{
std::cerr << "timerfd_settime error: " << strerror(errno) << std::endl;
}
int numEvents = 0;
int timeout = 0;
bool checkTimer = false;
while (1)
{
checkTimer = false;
numEvents = epoll_wait(efd, events, 128, timeout);
if (numEvents > 0)
{
for (int i = 0; i < numEvents; ++i)
{
if (events[i].data.ptr == &tfd)
{
std::cout << "timeout" << std::endl;
checkTimer = true;
}
}
}
else if(numEvents == 0)
{
continue;
}
else
{
std::cerr << "An error occured: " << strerror(errno) << std::endl;
}
if (checkTimer)
{
/* Check data storage */
uint64_t value;
ssize_t readBytes;
//while ( (readBytes = read(tfd, &value, 8)) > 0)
//{
// std::cout << "\tread: '" << value << "'" << std::endl;
//}
itimerspec new_timeout;
itimerspec old_timeout;
new_timeout.it_value.tv_sec = rand() % 3 + 1;
new_timeout.it_value.tv_nsec = 0;
new_timeout.it_interval.tv_sec =
new_timeout.it_interval.tv_nsec = 0;
timerfd_settime(tfd, flags, &new_timeout, &old_timeout);
}
}
return EXIT_SUCCESS;
}
This is simple description of my app.
After each timeout timer need to be rearmed by some value different in each timeout.
Questions are:
Is it necessary to add timerfd to epoll (epoll_ctl) with EPOLLET flag?
Is it necessary to read timerfd after each timeout?
Is it necessary to epoll_wait infinitely (timeout = -1)?

You can do this in one of two modes, edge triggered or level triggered. If you choose the edge triggered route then you must pass EPOLLET and do not need to read the timerfd after each wakeup. The fact that you receive an event from epoll means one or more time outs have fired. Optionally you may read the timerfd and it will return the number of time outs that have fired since you last read it.
If you choose the level triggered route then you don't need to pass EPOLLET, but you must read the timerfd after each wakeup. If you do not then you will immediately be woken up again until you consume the time out.
You should either pass -1 to epoll as the time out or some positive value. If you pass 0, like you do in the example, then you will never go to sleep, you'll just spin waiting for the time out to fire. That's almost certainly undesirable behaviour.

Answers to the questions:
Is it necessary to add timerfd to epoll (epoll_ctl) with EPOLLET flag?
No. Adding EPOLLET (edge trigger) does changes the behavior of receiving events. Without EPOLLET, you'll continuously receive the event from epoll_wait related to the timerfd until you've read() from the timerfd. With EPOLLET, you'll NOT receive additional events beyond the first one, even if new expiration occurs, until you've read() from the timerfd and a new expiration occur.
Is it necessary to read timerfd after each timeout?
Yes in order to continue and receive events (only) when new expiration occur (see above). No when periodic timer is not used (single expiration only), and you close the timerfd without reading.
Is it necessary to epoll_wait infinitely (timeout = -1)?
No. You can use epoll_wait's timeout instead of timerfd. I personally think it is easier to use timerfd than keep calculating the next timeout for EPOLL, especially if you expect multiple timeout intervals; keeping tabs on what is your next task when timeout occurs is much easier when it is tied to the specific event what woke up.

Related

SocketCAN with C++ on Raspberry Pi: messages lost when read is delayed

In a C++ application running on a Raspberry Pi, I am using a loop in a thread to continuously wait for SocketCAN messages and process them. The messages come in at around 1kHz, as verified using candump.
After waiting for poll() to return and reading the data, I read the timestamp using ioctl() with SIOCGSTAMP. I then compare the timestamp with the previous one, and this is where it gets weird:
Most of the time, the difference is around 1ms, which is expected. But sometimes (probably when the data processing takes longer than usual or gets interrupted by the scheduler) it is much bigger, up to a few hundred milliseconds. In those instances, the messages that should have come in in the meantime (visible in candump) are lost.
How is that possible? If there is a delay somewhere, the incoming messages get buffered? Why do they get lost?
This is the slightly simplified code:
while(!done)
{
struct pollfd fd = {.fd = canSocket, .events = POLLIN};
int pollRet = poll(&fd, 1, 20); // 20ms timeout
if(pollRet < 0)
{
std::cerr << "Error polling canSocket" << errno << std::endl;
done = true;
return;
}
if(pollRet == 0) // timeout, never happens as expected
{
std::cout << "canSocket poll timeout" << std::endl;
if(done) break;
continue;
}
struct canfd_frame frame;
int size = sizeof(frame);
int readLength = read(canSocket, &frame, size);
if(readLength < 0) throw std::runtime_error("CAN read failed");
else if(readLength < size) throw std::runtime_error("CAN read incomplete");
struct timeval timestamp;
ioctl(canSocket, SIOCGSTAMP, &timestamp);
uint64_t timestamp_us = (uint64_t)timestamp.tv_sec * 1e6 + (uint64_t)timestamp.tv_usec;
static uint64_t timestamp_us_last = 0;
if((timestamp_us - timestamp_us_last) > 20000)
{
std::cout << "timestamp difference large: " << (timestamp_us - timestamp_us_last) << std::endl; // this sometime happens, why?
}
timestamp_us_last = timestamp_us;
// data processing
}

How to properly wait for condition variable in C++?

In trying to create an asynchronous I/O file reader in C++ under Linux. The example I have has two buffers. The first read blocks. Then, for each time around the main loop, I asynchronously launch the IO and call process() which runs the simulated processing of the current block. When processing is done, we wait for the condition variable. The idea is that the asynchronous handler should notify the condition variable.
Unfortunately the notify seems to happen before wait, and it seems like this is not the way the condition variable wait() function works. How should I rewrite the code so that the loop waits until the asynchronous io has completed?
#include <aio.h>
#include <fcntl.h>
#include <signal.h>
#include <unistd.h>
#include <condition_variable>
#include <cstring>
#include <iostream>
#include <thread>
using namespace std;
using namespace std::chrono_literals;
constexpr uint32_t blockSize = 512;
mutex readMutex;
condition_variable cv;
int fh;
int bytesRead;
void process(char* buf, uint32_t bytesRead) {
cout << "processing..." << endl;
usleep(100000);
}
void aio_completion_handler(sigval_t sigval) {
struct aiocb* req = (struct aiocb*)sigval.sival_ptr;
// check whether asynch operation is complete
if (aio_error(req) == 0) {
int ret = aio_return(req);
bytesRead = req->aio_nbytes;
cout << "ret == " << ret << endl;
cout << (char*)req->aio_buf << endl;
}
{
unique_lock<mutex> readLock(readMutex);
cv.notify_one();
}
}
void thready() {
char* buf1 = new char[blockSize];
char* buf2 = new char[blockSize];
aiocb cb;
char* processbuf = buf1;
char* readbuf = buf2;
fh = open("smallfile.dat", O_RDONLY);
if (fh < 0) {
throw std::runtime_error("cannot open file!");
}
memset(&cb, 0, sizeof(aiocb));
cb.aio_fildes = fh;
cb.aio_nbytes = blockSize;
cb.aio_offset = 0;
// Fill in callback information
/*
Using SIGEV_THREAD to request a thread callback function as a notification
method
*/
cb.aio_sigevent.sigev_notify_attributes = nullptr;
cb.aio_sigevent.sigev_notify = SIGEV_THREAD;
cb.aio_sigevent.sigev_notify_function = aio_completion_handler;
/*
The context to be transmitted is loaded into the handler (in this case, a
reference to the aiocb request itself). In this handler, we simply refer to
the arrived sigval pointer and use the AIO function to verify that the request
has been completed.
*/
cb.aio_sigevent.sigev_value.sival_ptr = &cb;
int currentBytesRead = read(fh, buf1, blockSize); // read the 1st block
while (true) {
cb.aio_buf = readbuf;
aio_read(&cb); // each next block is read asynchronously
process(processbuf, currentBytesRead); // process while waiting
{
unique_lock<mutex> readLock(readMutex);
cv.wait(readLock);
}
currentBytesRead = bytesRead; // make local copy of global modified by the asynch code
if (currentBytesRead < blockSize) {
break; // last time, get out
}
cout << "back from wait" << endl;
swap(processbuf, readbuf); // switch to other buffer for next time
currentBytesRead = bytesRead; // create local copy
}
delete[] buf1;
delete[] buf2;
}
int main() {
try {
thready();
} catch (std::exception& e) {
cerr << e.what() << '\n';
}
return 0;
}
A condition varible should generally be used for
waiting until it is possible that the predicate (for example a shared variable) has changed, and
notifying waiting threads that the predicate may have changed, so that waiting threads should check the predicate again.
However, you seem to be attempting to use the state of the condition variable itself as the predicate. This is not how condition variables are supposed to be used and may lead to race conditions such as those described in your question. Another reason to always check the predicate is that spurious wakeups are possible with condition variables.
In your case, it would probably be appropriate to create a shared variable
bool operation_completed = false;
and use that variable as the predicate for the condition variable. Access to that variable should always be controlled by the mutex.
You can then change the lines
{
unique_lock<mutex> readLock(readMutex);
cv.notify_one();
}
to
{
unique_lock<mutex> readLock(readMutex);
operation_completed = true;
cv.notify_one();
}
and change the lines
{
unique_lock<mutex> readLock(readMutex);
cv.wait(readLock);
}
to:
{
unique_lock<mutex> readLock(readMutex);
while ( !operation_completed )
cv.wait(readLock);
}
Instead of
while ( !operation_completed )
cv.wait(readLock);
you can also write
cv.wait( readLock, []{ return operation_completed; } );
which is equivalent. See the documentation of std::condition_varible::wait for further information.
Of course, operation_completed should also be set back to false when appropriate, while the mutex is locked.

socket: multithreading doesn't work when client reads messages

I have a server, which can accept two socket connections. It creates a thread for each socket so that messages can be sent parallel.
Now I'm trying to code my client.
I create a class named SocketThread as a thread of socket. Here is the main code:
void SocketThread::ReadData()
{
int n = 0;
while (!finished)
{
while ((n = read(sockfd, recvBuff, sizeof(Data))) > 0)
{
std::cout<<std::this_thread::get_id()<<std::endl;
std::this_thread::sleep_for(std::chrono::milliseconds(2000));
}
}
}
std::thread SocketThread::run()
{
return std::thread([=] { ReadData(); });
}
in the function main:
SocketThread s0("127.0.0.1", 10000);
SocketThread s1("127.0.0.1", 10000);
std::thread td0{sts[0].run()};
std::thread td1{sts[1].run()};
td0.join(); // stop here
td1.join();
// something else
When I execute the program, it will block at td0.join();, meaning that I can get the id of the thread td0 on the console and I can NEVER get the other thread.
However, when I remove (n = read(sockfd, recvBuff, sizeof(Data))) > 0, which means that now the client is just a simple thread, that it won't receive anything, things gonna be fine ---- I can get two ids of the two threads.
Why?
EDIT
It seems that I used join incorrectly.
What I need is that main doesn't execute //something else until the two threads get 1000 characters together.
What should I do?
You did not use join() incorrectly. If you want main() to block until both threads end, your code is correct : td0.join() will block until thread td0 ends, and the same for td1.
Now, if you want your threads to end after receiving sizeof(Data) bytes, your function void SocketThread::ReadData() should rather look like this :
void SocketThread::ReadData()
{
int n, total = 0;
while (!finished)
{
while ((n = read(sockfd, &recvBuff[total], sizeof(Data) - total)) > 0)
{
total += n;
}
if (n == -1)
{
// manage error here
}
if (n == 0)
{
std::cout << "client shut the socket down; got " << total << " bytes over " << sizeof(Data) << std::endl;
finished = true;
}
}
}
For a short explanation : there is no guarantee that you can get all data sent by client in a single read() operation, so you need to call read() and cumulate data into the buffer until you get a return value of 0 (meaning the client shut down the socket). read(sockfd, &recvBuff[total], sizeof(Data) - total) ensures that the incomming data is properly appended at the right position in the buffer.

How to selectively ignore some inotify events?

Lets say I have two processes (simulated in this example with two threads) in a producer-consumer set up. That is, one process writes data to a file, the other process consumes the data in the file, then clears said file.
The set up I currently have, based on bits and pieces I've thrown together from various resources online, is that I should use a lock file to ensure that only one process can access the data file at a time. The producer acquires the lock, writes to the file, then releases the lock. Meanwhile, the consumer waits for modify events with inotify at which point it acquires the lock, consumes the data, and empties the file.
This seems relatively straightforward, but the part that's tripping me up is that when I empty the file out in my consumer thread, it triggers inotify modify event again, which sets off the whole flow again, and ends with the data file being cleared again, thus repeating forever.
I've tried a few ways to work around this problem, but none of them seem quite right. I'm worried doing this wrong will introduce potential race conditions or I'll end up skipping modify events or something.
Here is my current code:
#include <fstream>
#include <iostream>
#include <string>
#include "pthread.h"
#include "sys/file.h"
#include "sys/inotify.h"
#include "sys/stat.h"
#include "unistd.h"
const char* lock_filename = "./test_lock_file";
const char* data_filename = "./test_data_file";
int AquireLock(char const* lockName) {
mode_t m = umask(0);
int fd = open(lockName, O_RDWR | O_CREAT, 0666);
umask(m);
bool success = false;
if (fd < 0 || flock(fd, LOCK_EX) < 0) {
close(fd);
return -1;
}
return fd;
}
void ReleaseLock(int fd, char const* lockName) {
if (fd < 0) return;
remove(lockName);
close(fd);
}
void* ConsumerThread(void*) {
// Set up inotify.
int file_descriptor = inotify_init();
if (file_descriptor < 0) return nullptr;
int watch_descriptor =
inotify_add_watch(file_descriptor, data_filename, IN_MODIFY);
if (watch_descriptor < 0) return nullptr;
char buf[4096] __attribute__((aligned(__alignof__(inotify_event))));
while (true) {
// Read new events.
const inotify_event* event;
ssize_t numRead = read(file_descriptor, buf, sizeof(buf));
if (numRead <= 0) return nullptr;
// For each event, do stuff.
for (int i = 0; i < numRead; i += sizeof(inotify_event) + event->len) {
event = reinterpret_cast<inotify_event*>(&buf[i]);
// Critical section!
int fd = AquireLock(lock_filename);
// Read from the file.
std::string line;
std::ifstream data_file(data_filename);
if (data_file.is_open()) {
while (getline(data_file, line)) {
std::cout << line << std::endl;
}
data_file.close();
// Clear the file by opening then closing without writing to it.
std::ofstream erase_data_file(data_filename);
erase_data_file.close();
std::cout << "file cleared." << std::endl;
}
ReleaseLock(fd, lock_filename);
// Critical section over!
}
}
return nullptr;
}
int main(int argv, char** argc) {
// Set up other thread.
pthread_t thread;
int rc = pthread_create(&thread, NULL, ConsumerThread, nullptr);
if (rc) return rc;
// Producer thread: Periodically write to a file.
while (true) {
sleep(3);
// Critical section!
int fd = AquireLock(lock_filename);
// Write some text to a file
std::ofstream data_file(data_filename);
int counter = 0;
if (data_file.is_open()) {
std::cout << "Writing to file.\n";
data_file << "This is some example data. " << counter++ << "\n";
data_file.close();
}
ReleaseLock(fd, lock_filename);
// Critical section over!
}
pthread_exit(NULL);
return 0;
}
One idea I had was to disable tracking of modify events at the start of the consumer thread's critical section with inotify_rm_watch, then re-add it right before leaving the critical section. This doesn't seem to work though. Even with the events disabled, modify events are still getting triggered and I'm not sure why.
I've also considered just using a boolean to see if there was any file contents while consuming the file, and only clearing the file if it wasn't empty. This felt kind of hacky since it's still doing a second unnecessary iteration of the loop, but if I can't find a better solution I might just go with that. Ideally there would be a way to have only the producer thread's modifications trigger events, while the consumer could have it's own file modifications somehow ignored or disabled, but I'm not sure how to achieve that effect.

Catching SIGVTALRM signal, cp

I am trying to catch a SIGVTALRM sent by setitimer, and I have no idea why it doesn't work. here's my code:
void time(int time) {
cout << "time" << endl;
exit(0);
}
int main(void) {
signal(SIGVTALRM, time);
itimerval tv;
tv.it_value.tv_sec = 5;
tv.it_value.tv_usec = 0;
tv.it_interval.tv_sec = 5;
tv.it_interval.tv_usec = 0;
setitimer(ITIMER_VIRTUAL, &tv, NULL);
while (true) {
cout << "waiting" << endl;
}
return 0;
}
For some reason it never invokes time() - is it because it doesn't catch the signal or because the signal wasn't sent in the first place I don't know.
It should be pretty simple. Any ideas? thanks
Are you sure it is not working?
Everything looks fine to me. May be you are not waiting enough. Since you are printing the string waiting inside the loop and you are using the virtual timer, as a result the clock ticks only when the process runs (IO time not included). So in reality your timer might expire after several (>5) seconds.
Try commenting out the printing part.
It is due to signal function. As mentioned in http://manpages.ubuntu.com/manpages//precise/en/man2/signal.2.html:
The behavior of signal() varies across UNIX versions, and has also varied historically across different versions of Linux. Avoid its use: use sigaction(2) instead.
So the main method should be:
int main(void) {
itimerval tv;
struct sigaction sa;
sigemptyset(&sa.sa_mask);
sa.sa_flags = 0;
sa.sa_handler = timer_handler;
if (sigaction(SIGVTALRM, &sa, NULL) == -1) {
printf("error with: sigaction\n");
exit(EXIT_FAILURE);
}
tv.it_value.tv_sec = 5;
tv.it_value.tv_usec = 0;
tv.it_interval.tv_sec = 5;
tv.it_interval.tv_usec = 0;
setitimer(ITIMER_VIRTUAL, &tv, NULL);
while (true) {
cout << "waiting" << endl;
}
return 0;
}