I have this function _write_port() being called from a thread, whenever I need to send a message. To ensure the whole message is written, tcdrain() is used.
void Serial_Port::_write_port(char *buf, unsigned &len)
{
// Lock
pthread_mutex_lock(&lock);
// Write packet via serial link
write(fd, buf, len);
// Wait until all data has been written
tcdrain(fd);
// Unlock
pthread_mutex_unlock(&lock);
return;
}
My problem is that tcdrain() blocks forever after a random number of executions of this function _write_port(). This will block the lock, resulting in blocking my other read thread, resulting in blocking everything.
What is a good approach to avoid tcdrain from blocking forever?
Note: I strangely noticed that if I use several printf() throughout the function, tcdrain never blocks. It does not make any sense to me to exist a relationship between printf() and write() because they write to different output files. Since I cannot explain this behaviour I assume it may be a coincidence that it worked like this on my experiments. If someone can explain this behaviour, please let me know.
Related
Let's say I start a thread to receive on a port. The socket call will block on recvfrom.
Then, somehow in another thread, I close the socket.
On Windows, this will unblock recvfrom and my thread execution will terminate.
On Linux, this does not unblock recvfrom, and as a result, my thread is sitting doing nothing forever, and the thread execution does not terminate.
Can anyone help me with what's happening on Linux? When the socket is closed, I want recvfrom to unblock
I keep reading about using select(), but I don't know how to use it for my specific case.
Call shutdown(sock, SHUT_RDWR) on the socket, then wait for the thread to exit. (i.e. pthread_join).
You would think that close() would unblock the recvfrom(), but it doesn't on linux.
Here's a sketch of a simple way to use select() to deal with this problem:
// Note: untested code, may contain typos or bugs
static volatile bool _threadGoAway = false;
void MyThread(void *)
{
int fd = (your socket fd);
while(1)
{
struct timeval timeout = {1, 0}; // make select() return once per second
fd_set readSet;
FD_ZERO(&readSet);
FD_SET(fd, &readSet);
if (select(fd+1, &readSet, NULL, NULL, &timeout) >= 0)
{
if (_threadGoAway)
{
printf("MyThread: main thread wants me to scram, bye bye!\n");
return;
}
else if (FD_ISSET(fd, &readSet))
{
char buf[1024];
int numBytes = recvfrom(fd, buf, sizeof(buf), 0);
[...handle the received bytes here...]
}
}
else perror("select");
}
}
// To be called by the main thread at shutdown time
void MakeTheReadThreadGoAway()
{
_threadGoAway = true;
(void) pthread_join(_thread, NULL); // may block for up to one second
}
A more elegant method would be to avoid using the timeout feature of select, and instead create a socket pair (using socketpair()) and have the main thread send a byte on its end of the socket pair when it wants the I/O thread to go away, and have the I/O thread exit when it receives a byte on its socket at the other end of the socketpair. I'll leave that as an exercise for the reader though. :)
It's also often a good idea to set the socket to non-blocking mode also, to avoid the (small but non-zero) chance that the recvfrom() call might block even after select() indicated the socket is ready-to-read, as described here. But blocking mode might be "good enough" for your purpose.
Not an answer, but the Linux close man page contains the interesting quote:
It is probably unwise to close file descriptors while they may be in
use by system calls in other threads in the same process. Since a file
descriptor may be reused, there are some obscure race conditions that
may cause unintended side effects.
You are asking for the impossible. There is simply no possible way for the thread that calls close to know that the other thread is blocked in recvfrom. Try to write code that guarantees that this happens, you will find that it is impossible.
No matter what you do, it will always be possible for the call to close to race with the call to recvfrom. The call to close changes what the socket descriptor refers to, so it can change the semantic meaning of the call to recvfrom.
There is no way for the thread that enters recvfrom to somehow signal to the thread that calls close that it is blocked (as opposed to being about to block or just entering the system call). So there is literally no possible way to ensure the behavior of close and recvfrom are predictable.
Consider the following:
A thread is about to call recvfrom, but it gets pre-empted by other things the system needs to do.
Later, the thread calls close.
A thread started by the system's I/O library calls socket and gets the same decsriptor as the one you closed.
Finally, the thread calls recvfrom, and now it's receiving from the socket the library opened.
Oops.
Don'd ever do anything even remotely like this. A resource must not be released while another thread is, or might be, using it. Period.
I have a std::thread function that is calling fopen to load a big file into an array:
void loadfile(char *fname, char *fbuffer, long fsize)
{
FILE *fp = fopen(fname, "rb");
fread(fbuffer, 1, fsize, fp);
flose(fp);
}
This is called by:
std::thread loader(loadfile, fname, fbuffer, fsize);
loader.detach();
At some point, something in my program wants to stop reading that file and asks for another file. The problem is that by the time I delete the fbuffer pointer, the loader thread is still going, and I get a race condition that trows an exception.
How can I kill that thread? My idea was to check for the existance of the fbuffer and maybe split the fread in small chunks:
void loadfile(char *fname, char *fbuffer, long fsize)
{
FILE *fp = fopen(fname, "rb");
long ch = 0;
while (ch += 256 < fsize)
{
if (fbuffer == NULL) return;
fread(fbuffer + ch, 1, 256, fp);
}
fclose(fp);
}
Will this slow down the reading of the file? Do you have a better idea?
You should avoid killing a thread at all costs. Doing so causes evil things to happen, like resources left in a permanently locked state.
The thread must be given a reference to a flag, the value of which can be set from elsewhere, to tell the thread to voluntarily quit.
You cannot use a buffer for this purpose; if one thread deletes the memory of the buffer while the other is writing to it, very evil things will happen. (Memory corruption.) So, pass a reference to a boolean flag.
Of course, in order for the thread to be able to periodically check the flag, it must have small chunks of work to do, so splitting your freads to small chunks was a good idea.
256 bytes might be a bit too small though; definitely use 4k or more, perhaps even 64k.
Killing threads is usually not the way to go - doing this may lead to leaked resources, critical sections you cannot exit and inconsistent program state.
Your idea is almost spot-on, but you need a way to signal the thread to finalize. You can use a boolean shared between your thread and the rest of your code that your thread reads after every read, and once it is set, stops reading into the buffer cleans up the file handles and exits cleanly.
On another note, handling the deletion of pointers with owning semantics by yourself is most of the time frowned upon in modern C++ - unless you have a very good reason not to, I'd recommend using the stl fstream and string classes.
You need proper thread synchronization. The comments about resource leaks and the proposal by #Mike Nakis about making the thread exit voluntarily by setting a boolean are almost correct (well, they're correct, but not complete). You need to go even farther than that.
You must ensure not only that the loader thread exits on its own, you must also ensure that it has exited before you delete the buffer it is writing to. Or, at least, you must ensure that it isn't ever touching that buffer in any way after you deleted it. Checking the pointer for null-ness does not work for two reasons. First, it doesn't work anyway, since you are looking at a copy of the original pointer (you would have to use a pointer-pointer or a reference). Second, and more importantly, even if the check worked, there is a race condition between the if statement and fread. In other words, there is no way to guarantee that you aren't freeing the buffer while fread is accessing it (no matter how small you make your chunks).
At the very minimum, you neeed two boolean flags, but preferrably you would use a proper synchronization primitive such as a condition variable to notify the main thread (so you don't have to spin waiting for the loader to exit, but can block).
The correct way of operation would be:
Notify loader thread
Wait for loader thread to signal me (block on cond var)
Loader thread picks up notification, sets condition variable and never touches the buffer afterwards, then exits
Resume (delete buffer, allocate new buffer, etc)
If you do not insist on detaching the loader thread, you could instead simply join it after telling it to exit (so you would not need a cond var).
From posts like this, I know on linux the recv/send functions are thread safe and user is allowed to operate on the same socket from different threads simultaneously.
Though this is not a good design, in following situation I wonder what shall we do from user level code in order to keep data consistency and healthy running state: There are threads operating on the same sockets, the first one for creating and closing socket, the second for reading socket and the last one for sending sockets. See the pseudo code
struct SocketInfo
{
int SockFd;
int SockType;
queue<Packet*> RecvPacks;
};
map<int, SocketInfo*> gSocketInfoMap;
pthread_mutex_t gSocketsLock;
//Thread1
pthread_mutex_lock(&gSocketsLock);
// get information for sock
SocketInfo* info = gSocketInfoMap[sock];
pthread_mutex_unlock(&gSocketsLock);
close(sock); // line-1
.....
//thread-2
pthread_mutex_lock(&gSocketsLock);
SocketInfo* info = gSocketInfoMap[sock];
pthread_mutex_unlock(&gSocketsLock);
recv(sock, buffer, sizeof(buffer)); // line-2
.....
//thread-3
pthread_mutex_lock(&gSocketsLock);
SocketInfo* info = gSocketInfoMap[sock];
pthread_mutex_unlock(&gSocketsLock);
send(sock, buffer, sizeof buffer); // line-3
.....
I wonder if I need to move the Line-1, Line-2 and Line-3 into the protection scope of gSocketsLock? Why?
As the linked question states, socket operations are threadsafe. In any event, receiving and sending data are independent operations which do not interfere with each other.
Obviously, it is not a great idea to close a socket which is actively being read from and written to, but putting the close() inside a critical section does not do anything to prevent that bug. Whatever mechanism ensures that active sockets are not closed or that closed sockets are not accessed is at a higher level than the critical sections shown in the OP.
If one thread closes a socket that another thread is trying to use for I/O, the worst that can happen is that the recv/send call will return an error.
In short: no, it would not be a good idea to put the socket operations inside the critical section. It has not benefit, and it unnecessarily increases the likelihood of lock contention.
So I've made a socket class that uses boost::asio library to make asynchronous reads and writes. It works, but I have a few questions.
Here's a basic code example:
class Socket
{
public:
void doRead()
{
m_sock->async_receive_from(boost::asio::buffer(m_recvBuffer), m_from, boost::bind(&Socket::handleRecv, this, boost::asio::placeholders::error(), boost::asio::placeholders::bytes_transferred()));
}
void handleRecv(boost::system::error_code e, int bytes)
{
if (e.value() || !bytes)
{
handle_error();
return;
}
//do something with data read
do_something(m_recvBuffer);
doRead(); //read another packet
}
protected:
boost::array<char, 1024> m_recvBuffer;
boost::asio::ip::udp::endpoint m_from;
};
It seems that the program will read a packet, handle it, then prepare to read another. Simple.
But what if I set up a thread pool? Should the next call to doRead() be before or after handling the read data? It seems that if it is put before do_something(), the program can immediately begin reading another packet, and if it is put after, the thread is tied up doing whatever do_something() does, which could possibly take a while. If I put the doRead() before the handling, does that mean the data in m_readBuffer might change while I'm handling it?
Also, if I'm using async_send_to(), should I copy the data to be sent into a temporary buffer, because the actual send might not happen until after the data has fallen out of scope? i.e.
void send()
{
char data[] = {1, 2, 3, 4, 5};
m_sock->async_send_to(boost::buffer(&data[0], 5), someEndpoint, someHandler);
} //"data" gets deallocated, but the write might not have happened yet!
Additionally, when the socket is closed, the handleRecv will be called with an error indicating it was interrupted. If I do
Socket* mySocket = new Socket()...
...
mySocket->close();
delete mySocket;
could it cause an error, because there is a chance that mySocket will be deleted before handleRecv() gets called/finished?
Lots of questions here, I'll try to address them one at a time.
But what if I set up a thread pool?
The traditional way to use a thread pool with Boost.Asio is to invoke io_service::run() from multiple threads. Beware this isn't a one-size-fits-all answer though, there can be scalability or performance issues, but this methodology is by far the easiest to implement. There are many similar questions on Stackoverflow with more information.
Should the next call to doRead be before or after handling the read
data? It seems that if it is put before do_something(), the program
can immediately begin reading another packet, and if it is put after,
the thread is tied up doing whatever do_something does, which could
possibly take a while.
This really depends on what do_something() needs to do with m_recvBuffer. If you wish to invoke do_something() in parallel with doRead() using io_service::post() you will likely need to make a copy of m_recvBuffer.
If I put the doRead() before the handling, does
that mean the data in m_readBuffer might change while I'm handling it?
as I mentioned previously, yes this can and will happen.
Also, if I'm using async_send_to(), should I copy the data to be sent
into a temporary buffer, because the actual send might not happen
until after the data has fallen out of scope?
As the documentation describes, it is up to the caller (you) to ensure the buffer remains in scope for the duration of the asynchronous operation. As you suspected, your current example invokes undefined behavior because data[] will go out of scope.
Additionally, when the socket is closed, the handleRecv() will be called
with an error indicating it was interrupted.
If you wish to continue to use the socket, use cancel() to interrupt outstanding asynchronous operations. Otherwise, close() will work. The error passed to outstanding asynchronous operations in either scenario is boost::asio::error::operation_aborted.
I'm writing server using boost.asio. I have read and write buffer for each connection and use asynchronized read/write function (async_write_some / async_read_some).
With read buffer and async_read_some, there's no problem. Just invoking async_read_some function is okay because read buffer is read only in read handler (means in same thread usually).
But, write buffer need to be accessed from several threads so it need to be locked for modifying.
FIRST QUESTION!
Are there any way to avoid LOCK for write buffer?
I write my own packet into stack buffer and copy it to the write buffer. Then, call async_write_some function to send the packet. In this way, if I send two packet in serial, is it okay invoking async_write_some function two times?
SECOND QUESTION!
What is common way for asynchronized writing in socket programming?
Thanks for reading.
Sorry but you have two choices:
Serialise the write statement, either with locks, or better
start a separate writer thread which reads requests from
a queue, other threads can then stack up requests on the
queue without too much contention (some mutexing would be required).
Give each writing thread its own socket!
This is actually the better solution if the program at the other end
of the wire can support it.
Answer #1:
You are correct that locking is a viable approach, but there is a much simpler way to do all of this. Boost has a nice little construct in ASIO called a strand. Any callback that has been wrapped using the strand will be serialized, guaranteed, no matter which thread executes the callback. Basically, it handles any locking for you.
This means that you can have as many writers as you want, and if they are all wrapped by the same strand (so, share your single strand among all of your writers) they will execute serially. One thing to watch out for is to make sure that you aren't trying to use the same actual buffer in memory for doing all of the writes. For example, this is what to avoid:
char buffer_to_write[256]; // shared among threads
/* ... in thread 1 ... */
memcpy(buffer_to_write, packet_1, std::min(sizeof(packet_1), sizeof(buffer_to_write)));
my_socket.async_write_some(boost::asio::buffer(buffer_to_write, sizeof(buffer_to_write)), &my_callback);
/* ... in thread 2 ... */
memcpy(buffer_to_write, packet_2, std::min(sizeof(packet_2), sizeof(buffer_to_write)));
my_socket.async_write_some(boost::asio::buffer(buffer_to_write, sizeof(buffer_to_write)), &my_callback);
There, you're sharing your actual write buffer (buffer_to_write). If you did something like this instead, you'll be okay:
/* A utility class that you can use */
class PacketWriter
{
private:
typedef std::vector<char> buffer_type;
static void WriteIsComplete(boost::shared_ptr<buffer_type> op_buffer, const boost::system::error_code& error, std::size_t bytes_transferred)
{
// Handle your write completion here
}
public:
template<class IO>
static bool WritePacket(const std::vector<char>& packet_data, IO& asio_object)
{
boost::shared_ptr<buffer_type> op_buffer(new buffer_type(packet_data));
if (!op_buffer)
{
return (false);
}
asio_object.async_write_some(boost::asio::buffer(*op_buffer), boost::bind(&PacketWriter::WriteIsComplete, op_buffer, boost::asio::placeholder::error, boost::asio::placeholder::bytes_transferred));
}
};
/* ... in thread 1 ... */
PacketWriter::WritePacket(packet_1, my_socket);
/* ... in thread 2 ... */
PacketWriter::WritePacket(packet_2, my_socket);
Here, it would help if you passed your strand into WritePacket as well. You get the idea, though.
Answer #2:
I think you are already taking a very good approach. One suggestion I would offer is to use async_write instead of async_write_some so that you are guaranteed the whole buffer is written before your callback gets called.
You could queue your modifications and perform them on the data in the write handler.
Network would most probably be the slowest part of the pipe (assuming your modification are not
computationaly expensive), so that you could perform mods while the socket layer is sending the
previous data.
Incase you are handling large number of clients with frequent connect/disconnect take a look at
IO completion ports or similar mechanism.