In a server code I want to use pselect to wait for clients to connect as well monitor the standard output of the prozesses that I create and send it to the client (like a simplified remote shell).
I tried to find examples on how to use pselect but I haven't found any. The socket where the client can connect is already set up and works, as I verified that with accept(). SIGTERM is blocked.
Here is the code where I try to use pselect:
waitClient()
{
fd_set readers;
fd_set writers;
fd_set exceptions;
struct timespec ts;
// Loop until we get a sigterm to shutdown
while(getSigTERM() == false)
{
FD_ZERO(&readers);
FD_ZERO(&writers);
FD_ZERO(&exceptions);
FD_SET(fileno(stdin), &readers);
FD_SET(fileno(stdout), &writers);
FD_SET(fileno(stderr), &writers);
FD_SET(getServerSocket()->getSocketId(), &readers);
//FD_SET(getServerSocket()->getSocketId(), &writers);
memset(&ts, 0, sizeof(struct timespec));
pret = pselect(FD_SETSIZE, &readers, &writers, &exceptions, &ts, &mSignalMask);
// Here pselect always returns with 2. What does this mean?
cout << "pselect returned..." << pret << endl;
cout.flush();
}
}
So what I want to know is how to wait with pselect until an event is received, because currently pselect always returns immediately with a value 2. I tried to set the timeout to NULL but that doesn't change anything.
The returnvalue of pselect (if positive) is the filedescriptor that caused the event?
I'm using fork() to create new prozesses (not implemented yet) I know that I have to wait() on them. Can I wait on them as well? I suppose I need to chatch the signal SIGCHILD, so how would I use that? wait() on the child would also block, or can I just do a peek and then continue with pselect, otherwise I have to concurrent blocking waits.
It returns immediately because the file descriptors in the writers set are ready. The standard output streams will almost always be ready for writing.
And if you check a select manual page you will see that the return value is either -1 on error, 0 on timeout, and a positive number telling you the number of file descriptors that are ready.
Related
So, I've run into this issue where I have many threads calling poll on different file descriptors. When I want to add a new one, I need to cancel one of those polls, add a new one, and continue. That alone sounds bad, but also I can't even see how to do that.
Some relevant code:
struct pollfd fds[size];
for(int i = 0;i<size;i++) {
struct pollfd fd;
fd.fd = body[i];
fd.events = POLLIN;
fd.revents = 0;
fds[i] = fd;
}
if(poll(&fds[0], (nfds_t)size, -1) < 0) return NULL;
(I'm using this through JNI also).
I figure I could set a really low delay on poll, and call it over and over, but I think that would begin to defeat the purpose.
The way you can do it is: open a socket or a pipe where, when there is a new file descriptor to add to a polling set, another thread sends some data. Thus, poll will return, you check this reserved file descriptor. If there is data, it means there is a new file descriptor to add.
You can send your process a signal, causing poll() to return -1 and set errno to EINTR. The signal should obviously not cause the process to terminate, so you may need to get some sigaction() or sigprocmask(). However, any signal received between calling either of those and poll() may get lost, similar to select()/pselect(). For this reason some systems may provide additional, non-standard replacements for poll(), like ppoll(), which include a sigset_t to change the signal disposition atomically.
When doing socket programming, with multi-threading,
if a thread is blocked on Accept Function,
and main thread is trying to shut down the process,
how to break the accept function in order to pthread_join safely?
I have vague memory of how to do this by connection itself to its own port in order to break the accept function.
Any solution will be thankful.
Cheers
Some choices:
a) Use non-blocking
b) Use AcceptEx() to wait on an extra signal, (Windows)
c) Close the listening socket from another thread to make Accept() return with an error/exception.
d) Open a temporary local connection from another thread to make Accept() return with the temp connection
The typical approach to this is not to use accept() unless there is something to accept! The way to do this is to poll() the corresponding socket with a suitable time-out in a loop. The loop checks if it is meant to exit because a suitably synchronized flag was set.
An alternative is to send the blocked thread a signal, e.g., using pthread_kill(). This gets out of the blocked accept() with a suitable error indication. Again, the next step is to check some flag to see if the thread is meant to exit. My preference is the first approach, though.
Depending on your system, if it is available, I would use a select function to wait for the server socket to have a read, indicating a socket is trying to connect. The amount of time to time to wait for a connection can be set/adjusted to to what every time you want to wait for a client to connect(infinity, to seconds, to 0 which will just check and return). The return status needs to be checked to see if the time limit was reached (no socket is trying to connect), or if there is something waiting to be serviced (your server socket indicating there is a client which would like to connect). You can then execute the accept knowing there is a socket to connect based on the returned status.
If available I would use a select function with a timeout in a loop to achieve this functionality.
as Glenn suggested
The select function with a timeout value will wait for a socket to connect for a set period of time. If a socket attempts to connect it can be accepted during that period. By looping this select with a timeout it is possible to check for new connections until the break condition is met.
Here is an example:
std::atomic<bool> stopThread;
void theThread ( std::atomic<bool> & quit )
{
struct timeval tv;
int activity;
...
while(!quit)
{
// reset the time value for select timeout
tv.tv_sec = 0;
tv.tv_usec = 1000000;
...
//wait for an activity on one of the sockets
activity = select( max_sd + 1 , &readfds , NULL , NULL , &tv);
if ((activity < 0) && (errno!=EINTR))
{
printf("select error");
}
if (FD_ISSET(master_socket, &readfds))
{
if ((new_socket = accept(master_socket, (struct sockaddr *)&address, (socklen_t*)&addrlen))<0)
{
perror("accept");
exit(EXIT_FAILURE);
}
...
}
}
int main(int argc, char** argv)
{
...
stopThread = false;
std::thread foo(theThread, std::ref(stopThread));
...
stopThread = true;
foo.join();
return 0;
}
A more complete example of 'Select' http://www.binarytides.com
I am pretty new to C++ so I am sure my code and answer can be improved.
Sounds like what you are looking for is this: You set a special flag variable known to the listening/accepting socket, and then let the main thread open a connection to the listening/accepting socket. The listening/accepting socket/thread has to check the flag every time it accepts a connection in order to know when to shut down.
Typically if you want to do multi-threaded networking, you would spawn a thread once a connection is made (or ready to be made). If you want to lower the overhead, a thread pool isn't too hard to implement.
I wanted to create a multi-threaded socket server using C++11 and standard linux C-Librarys.
The easiest way doing this would be opening a new thread for each incoming connection, but there must be an other way, because Apache isn't doing this. As far as I know Apache handles more than one connection in a Thread. How to realise such a system?
I thought of creating one thread always listening for new clients and assigning this new client to a thread. But if all threads are excecuting an "select()" currently, having an infinite timeout and none of the already assigned client is doing anything, this could take a while for the client to be useable.
So the "select()" needs a timeout. Setting the timeout to 0.5ms would be nice, but I guess the workload could rise too much, couldn't it?
Can someone of you tell me how you would realise such a system, handling more than one client for each thread?
PS: Hope my English is well enough for you to understand what I mean ;)
The standard method to multiplex multiple requests onto a single thread is to use the Reactor pattern. A central object (typically called a SelectServer, SocketServer, or IOService), monitors all the sockets from running requests and issues callbacks when the sockets are ready to continue reading or writing.
As others have stated, rolling your own is probably a bad idea. Handling timeouts, errors, and cross platform compatibility (e.g. epoll for linux, kqueue for bsd, iocp for windows) is tricky. Use boost::asio or libevent for production systems.
Here is a skeleton SelectServer (compiles but not tested) to give you an idea:
#include <sys/select.h>
#include <functional>
#include <map>
class SelectServer {
public:
enum ReadyType {
READABLE = 0,
WRITABLE = 1
};
void CallWhenReady(ReadyType type, int fd, std::function<void()> closure) {
SocketHolder holder;
holder.fd = fd;
holder.type = type;
holder.closure = closure;
socket_map_[fd] = holder;
}
void Run() {
fd_set read_fds;
fd_set write_fds;
while (1) {
if (socket_map_.empty()) break;
int max_fd = -1;
FD_ZERO(&read_fds);
FD_ZERO(&write_fds);
for (const auto& pr : socket_map_) {
if (pr.second.type == READABLE) {
FD_SET(pr.second.fd, &read_fds);
} else {
FD_SET(pr.second.fd, &write_fds);
}
if (pr.second.fd > max_fd) max_fd = pr.second.fd;
}
int ret_val = select(max_fd + 1, &read_fds, &write_fds, 0, 0);
if (ret_val <= 0) {
// TODO: Handle error.
break;
} else {
for (auto it = socket_map_.begin(); it != socket_map_.end(); ) {
if (FD_ISSET(it->first, &read_fds) ||
FD_ISSET(it->first, &write_fds)) {
it->second.closure();
socket_map_.erase(it++);
} else {
++it;
}
}
}
}
}
private:
struct SocketHolder {
int fd;
ReadyType type;
std::function<void()> closure;
};
std::map<int, SocketHolder> socket_map_;
};
First off, have a look at using poll() instead of select(): it works better when you have large number of file descriptors used from different threads.
To get threads currently waiting in I/O out of waiting I'm aware of two methods:
You can send a suitable signal to the thread using pthread_kill(). The call to poll() fails and errno is set to EINTR.
Some systems allow a file descriptor to be obtained from a thread control device. poll()ing the corresponding file descriptor for input succeeds when the thread control device is signalled. See, e.g., Can we obtain a file descriptor for a semaphore or condition variable?.
This is not a trivial task.
In order to achieve that, you need to maintain a list of all opened sockets (the server socket and the sockets to current clients). You then use the select() function to which you can give a list of sockets (file descriptors). With correct parameters, select() will wait until any event happen on one of the sockets.
You then must find the socket(s) which caused select() to exit and process the event(s). For the server socket, it can be a new client. For client sockets, it can be requests, termination notification, etc.
Regarding what you say in your question, I think you are not understanding the select() API very well. It is OK to have concurrent select() calls in different threads, as long as they are not waiting on the same sockets. Then if the clients are not doing anything, it doesn't prevent the server select() from working and accepting new clients.
You only need to give select() a timeout if you want to be able to do things even if clients are not doing anything. For example, you may have a timer to send periodic infos to the clients. You then give select a timeout corresponding to you first timer to expire, and process the expired timer when select() returns (along with any other concurrent events).
I suggest you have a long read of the select manpage.
I have a server with 2 connections SOCKET which is connected with clients and I set this server is non-blocking mode which don't stop when sending or recieving message. I want to set time out for a SOCKET of each connections, but if I use the following code:
void getMessage(SOCKET connectedSocket, int time){
string error = R_ERROR;
// Using select in winsock
fd_set set;
timeval tm;
FD_ZERO(&set);
FD_SET(connectedSocket, &set);
tm.tv_sec = time; // time
tm.tv_usec = 0; // 0 millis
switch (select(connectedSocket, &set, 0, 0, &tm))
{
case 0:
// timeout
this->disconnect();
break;
case 1:
// Can recieve some data here
return this->recvMessage();
break;
default:
// error - handle appropriately.
break;
}
return error;
}
My server is not none-blocking mode any more! I have to wait until the end of 1st connection's time out to get message from the 2nd connection! That's not what I expect! So, is there any way to set time out for non-blocking mode? Or I have to handle it myself?
select is a demultiplexing mechanism. While you are using it to determine when data is ready on a single socket or timeout, it was actually designed to return data ready status on many sockets (hence the fd_set). Conceptually, it is the same with poll, epoll and kqueue. Combined with non-blocking I/O, these mechanisms provide an application writer with the tools to implement a single threaded concurrent server.
In my opinion, your application does not need that kind of power. Your application will only be handling two connections, and you are already using one thread per connection. I believe leaving the socket in blocking I/O mode is more appropriate.
If you insist on non-blocking mode, my suggestion is to replace the select call with something else. Since what you want from select is an indication of read readiness or timeout for a single socket, you can achieve a similar effect with recv passed with appropriate parameters and with the appropriate timeout set on the socket.
tm.tv_sec = time;
tm.tv_usec = 0;
setsockopt(connectedSocket, SOL_SOCKET, SO_RCVTIMEO, (char *)&tm, sizeof(tm));
char c;
swtich (recv(connectedSocket, &c, 1, MSG_PEEK|MSG_WAITALL)) {
case -1:
if (errno == EAGAIN) {
// handle timeout ...
} else {
// handle other error ...
}
break;
case 0: // FALLTHROUGH
default:
// handle read ready ...
break;
}
From man recv:
MSG_PEEK -- This flag causes the receive operation to return data from the beginning of the receive queue without removing that data from the queue. Thus, a subsequent receive call will return the same data.
MSG_WAITALL (since Linux 2.2) -- This flag requests that the operation block until the full request is satisfied. However, the call may still return less data than requested if a signal is caught, an error or disconnect occurs, or the next data to be received is of a different type than that returned.
As to why select is behaving in the way you observed. While the select call is thread-safe, it is likely fully guarded against reentrancy. So, one thread's call to select will only come in after another thread's call completes (the calls to select are serialized). This is inline with its function as a demultiplexer. It's purpose is to serve as a single arbiter for which connections are ready. As such, it wants to be controlled by a single thread.
I have a thread that listens for new connections
new_fd = accept(Listen_fd, (struct sockaddr *) & their_addr, &sin_size);
and another thread that closes Listen_fd when when it's time to close the program. After Listen_fd is closed however, it still blocks. When I use GDB to try and debug accept() doesn't block. I thought that it could be a problem with SO_LINGER, but it shouldn't be on by default, and shouldn't change when using GDB. Any idea whats going on, or any other suggestion to closing the listing socket?
Use: sock.shutdown (socket.SHUT_RD)
Then accept will return EINVAL. No ugly cross thread signals required!
From the Python documentation:
"Note close() releases the resource associated with a connection but does not necessarily close the connection immediately. If you want to close the connection in a timely fashion, call shutdown() before close()."
http://docs.python.org/3/library/socket.html#socket.socket.close
I ran into this problem years ago, while programming in C. But I only found the solution today, after running into the same problem in Python, AND pondering using signals (yuck!), AND THEN remembering the note about shutdown!
As for the comments that say you should not close/use sockets across threads... in CPython the global interpreter lock should protect you (assuming you are using file objects rather than raw, integer file descriptors).
Here is example code:
import socket, threading, time
sock = socket.socket (socket.AF_INET, socket.SOCK_STREAM)
sock.setsockopt (socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
sock.bind (('', 8000))
sock.listen (5)
def child ():
print ('child accept ...')
try: sock.accept ()
except OSError as exc : print ('child exception %s' % exc)
print ('child exit')
threading.Thread ( target = child ).start ()
time.sleep (1)
print ('main shutdown')
sock.shutdown (socket.SHUT_RD)
time.sleep (1)
print ('main close')
sock.close ()
time.sleep (1)
print ('main exit')
The behavior of accept when called on something which is not a valid socket FD is undefined. "Not a valid socket FD" includes numbers which were once valid sockets but have since been closed. You might say "but Borealid, it's supposed to return EINVAL!", but that's not guaranteed - for instance, the same FD number might be reassigned to a different socket between your close and accept calls.
So, even if you were to isolate and correct whatever makes your program fail, you could still begin to fail again in the future. Don't do it - correct the error that causes you to attempt to accept a connection on a closed socket.
If you meant that a call which was previously made to accept continues blocking after close, then what you should do is send a signal to the thread which is blocked in accept. This will give it EINTR and it can cleanly disengage - and then close the socket. Don't close it from a thread other than the one using it.
The shutdown() function may be what you are looking for. Calling shutdown(Listen_fd, SHUT_RDWR) will cause any blocked call to accept() to return EINVAL. Coupling a call to shutdown() with the use of an atomic flag can help to determine the reason for the EINVAL.
For example, if you have this flag:
std::atomic<bool> safe_shutdown(false);
Then you can instruct the other thread to stop listening via:
shutdown_handler([&]() {
safe_shutdown = true;
shutdown(Listen_fd, SHUT_RDWR);
});
For completeness, here's how your thread could call accept:
while (true) {
sockaddr_in clientAddr = {0};
socklen_t clientAddrSize = sizeof(clientAddr);
int connSd = accept(Listen_fd, (sockaddr *)&clientAddr, &clientAddrSize);
if (connSd < 0) {
// If shutdown_handler() was called, then exit gracefully
if (errno == EINVAL && safe_shutdown)
break;
// Otherwise, it's an unrecoverable error
std::terminate();
}
char clientname[1024];
std::cout << "Connected to "
<< inet_ntop(AF_INET, &clientAddr.sin_addr, clientname,
sizeof(clientname))
<< std::endl;
service_connection(connSd);
}
It's a workaround, but you could select on Listen_fd with a timeout, and if a timeout occured check that you're about to close the program. If so, exit the loop, if not, go back to step 1 and do the next select.
Are you checking the return value of close?
From linux manpages, (http://www.kernel.org/doc/man-pages/online/pages/man2/close.2.html)
"It is probably unwise to close file descriptors while they may be in use by system calls in other threads in the same process. Since a file descriptor may be reused, there are some obscure race conditions that may cause unintended side effects".
You can use a select instead of an accept and wait for some event from the other thead, then close the socket in the listener thread.