I'm trying to use C/C++ to communicate with a CAN bus. I'm using sockets to read and write on the bus. I've created a write thread and a read thread. The read thread is constantly trying to read and the socket, and when a write request arrives, the write thread take control of the socket using a mutex and do the write.
I'm having a big issue with speed using this method, as a write request can sometimes take 500 ms to be completed (which is completely unfeasible for my application). I've tried to put a timeout on the read command to make it non-blocking when nothing comes on the bus, but if the timeout is too short, I have reliability issues with the read. On the other hand, if I make it too long, the speed increase is insufficient.
It's my first time working with CAN. Would you have some advices on implementation of fast, two-way CAN communication node in C/C++ ? Which library should I use to interface with the bus itself ? Which architecture would yield the lowest read and write latency ?
To give a few metrics for the application, the bus bitrate is 1MBits/sec, I'm using CAN-Open with 64 bits data packets (each message contains 32 bits for indexes and 32 bits of data). I would like a write frequency from 300 to 500hz, same for the read frequency.
Thanks a lot for your help !
EDIT :
Thanks a lot for all your comments. Here are some clarifications on my application and problems.
I'm working on a mobile robot project, and I'm using CAN-Open to communicate with motor drivers and sensors. The code will run on a Raspberry Pi CM4 running Raspbian OS mounted on a custom IO board integrating a MCP2515 CAN controller. I want to implement fast communication interface between the ROS architecture and the CAN bus. The language used could be either C or C++.
I'm currently using a homemade interface build around standard C sockets, but the speed is very low, and is a big bottleneck to the robot's performance. So I'm looking for a better solution, either:
An open-source library build for this purpose
Architecture suggestion to implement such a program
A combination of both
Here are the socket creation, the read and the write functions I use. Read and write being each called in a while loop in different threads (I'm using pthread):
bool connectCanBus(int* socketIDOut, std::string canInterfaceName){
// Socket and can variables
struct sockaddr_can addr;
struct ifreq ifr;
// Openning the socket to send commands over the can bus
if ((*socketIDOut = socket(PF_CAN, SOCK_RAW, CAN_RAW)) < 0) {
perror("[Can Controler] Unable to create socket.");
return false;
}
strcpy(ifr.ifr_name, canInterfaceName.c_str());
ioctl(*socketIDOut, SIOCGIFINDEX, &ifr);
memset(&addr, 0, sizeof(addr));
addr.can_family = AF_CAN;
addr.can_ifindex = ifr.ifr_ifindex;
// Setting option to gte errors as frames
can_err_mask_t err_mask = 0xFFFFFFFF;
setsockopt(*socketIDOut, SOL_CAN_RAW, CAN_RAW_ERR_FILTER, &err_mask, sizeof(err_mask));
struct timeval tv;
tv.tv_sec = 0;
tv.tv_usec = 10000;
setsockopt(*socketIDOut, SOL_SOCKET, SO_RCVTIMEO, (const char*)&tv, sizeof tv);
// Binding Socket
if (bind(*socketIDOut, (struct sockaddr *)&addr, sizeof(addr)) < 0)
{
perror("[Can Controler] Unable to bind socket.");
close(*socketIDOut);
return false;
}
ROS_INFO("CAN bus connected.");
return true;
}
int sendCommand(const char* id, const char* message, int socket, std::mutex& mutex)
{
struct canfd_frame frame;
int err = parseFrame(id, message, &frame);
if(err == 0){
ROS_ERROR_STREAM("[Can Utils] Unable to parse frame : " << id << ", " << message);
return 0;
}
mutex.lock();
int res = write(socket, &frame, sizeof(struct can_frame));
mutex.unlock();
if (res != sizeof(struct can_frame)) {
perror("[Can Utils] CAN bus Write error");
return 0;
}
return 1;
}
int readBus(CanFrame *outFrame, int socketID, std::mutex& mutex)
{
struct can_frame frame;
// Reading on bus
mutex.lock();
int nbytes = read(socketID, &frame, sizeof(struct can_frame));
mutex.unlock();
if (nbytes < 0) {
perror("[Can Utils] CAN bus Read error");
return 0;
}
// Converting frame to strings
sprint_canframe(outFrame, &frame);
return nbytes;
}
I hope this make the question clearer and better focused.
Thanks to all comments on the question, I was able to solve the speed issue. I was not aware that SocketCan supports multiple socket on the same bus. Being able to create per process sockets for read and write allowed to reduce talk to the bus in a distributed manner, greatly improving speed.
Related
I have a simple socket server set up using sys/socket and OpenSSL. For each connection, the client is required to send a message to the server, receive a response and then reply to that response.
I can't find any clear mechanism for making these sockets non-blocking? The system has to be able to handle multiple sockets concurrently...
My server code for listening for connections:
while(1)
{
struct sockaddr_in addr;
uint len = sizeof(addr);
SSL *ssl;
int client = accept(sock, (struct sockaddr*)&addr, &len);
if (client > 0)
{
std::cout<<"Client accepted..."<<std::endl;
}
else
{
perror("Unable to accept");
exit(EXIT_FAILURE);
}
ssl = SSL_new(ctx);
SSL_set_fd(ssl, client);
if (SSL_accept(ssl) <= 0)
{
std::cout<<"ERROR"<<std::endl;
}
else
{
char buff[1024];
SSL_read(ssl, buff, 1024);
std::cout<<buff<<std::endl;
std::string reply="Thanks from the server";
char buff_response[1024];
reply.copy(buff_response, 1024);
const void *buf=&buff_response;
SSL_write(ssl, buf, 1024);
char another_buff[1024];
SSL_read(ssl,another_buff,1024);
std::cout<<another_buff<<std::endl;
}
}
I've looked into 'select()', however this doesn't seem to allow concurrency as such, but allows the system to know when a socket is freed?
Does anyone have any experience in solving this basic problem?
First, with server code, it's important to differentiate between concurrency and parallelism. A reasonable server will typically handle many more connections concurrently than its number of cores. Consequently, it's important to make the code concurrent in the sense that it can (efficiently) handle many concurrent connections, in a way that does not rely on parallelism (in the sense of having each connection handled by a thread).
In this sense, select is actually a reasonable choice for concurrency, and gives you the effect of being non-blocking.
When your system handles multiple sockets concurrently, select indicates on which socket(s) you can perform operations such as send and recv without their blocking when you do so. If you use select well you won't have cases where your thread is idling, waiting indefinitely for some operation to proceed, while other sockets are ready.
The minimal example from gnu.org shows a reasonably efficient server which it seems you can adapt to your needs.
fd_set active_fd_set, read_fd_set;
FD_ZERO (&active_fd_set);
FD_ZERO (&read_fd_set);
// Use FD_SET to add sockets according to what you want to do with them
/* This call (checking to see who can be read) is the
* only thing that blocks. But if it does, no socket is ready for reading. */
if (select (FD_SETSIZE, &read_fd_set, NULL, NULL, NULL) < 0) {
// Handle error;
for (i = 0; i < FD_SETSIZE; ++i)
if (FD_ISSET (i, &read_fd_set))
// Here you can read without its blocking.
Firstly I'm coding in c++ and running in Linux/CentOS 6.4
So after a really long time and reading a lot of different books on sockets, I finally have at least my client and my server partially working.
First I want to continuously accept messages from different clients, I have already setup the client, and it finally successfully compiled at least. Now I need to set up my server so that I can properly test.
What I'm doing is implementing the dining philosopher problem with sockets, with each client/philosopher representing a different process. I was going to go through this whole thing, where the server was going to keep track of everything, like the states of all the client. That was too difficult, I have now just created the client just to send their status to the server and the server prints it out.
I was thinking of putting a do/while loop to continuously accept messages, but not sure what I should use to stop the loop. Note that I will have a while loop set up in my client, which is signaled to stop after an elapsed amount of time. It should then close that particular client. I do have a signal in my serve, but I am not sure it works.
#include "helper.h"
char buffer[4096];
void sigchld_handler(int signo)
{
while (waitpid(-1, NULL, WNOHANG) > 0);
}
void client(int &newsock, int nread)
{
do
{
int nread = recv(newsock, buffer,sizeof(buffer), 0);
puts(buffer);
}while(nread!=0);
}
int main(int argc, char *argv[])
{
struct sockaddr_in sAddr, cli_addr;
socklen_t client_len;
int listensock;
int newsock;
int result;
int nread=1;
pid_t childid; ;
int status;
if((listensock = socket(AF_INET, SOCK_STREAM, 0))<0)
{
perror("Problem in creating socket");
exit(2);
}
sAddr.sin_family = AF_INET;
sAddr.sin_port = htons(3333);
sAddr.sin_addr.s_addr = htonl(INADDR_ANY);
bind(listensock, (struct sockaddr *) &sAddr, sizeof(sAddr));
if (result < 0) {
perror("exserver2");
return 0;
}
result = listen(listensock, 5);
if (result < 0) {
perror("exserver2");
return 0;
}
signal(SIGCHLD, sigchld_handler);
while (1) {
client_len = sizeof(cli_addr);
newsock = accept(listensock,(struct sockaddr *)&cli_addr, &client_len);
if ((childid = fork()) == 0) {
printf("child process %i created.\n", getpid());
close(listensock);
client(newsock, nread);
}
if(status<0)
{
printf("%s\n" "Read error");
exit(1);
}
close(newsock);
}
}
You need a multiplexing syscall like poll(2) (or the old, nearly obsolete, select(2)syscall). You may want to use some (or implement your own) event loop. See this & that answer. Read about the C10K problem.
Every server needs an event loop.
Read Advanced Linux Programming (or some Posix network programming book).
You may want to simply run your server program under tcpserver (see http://cr.yp.to/ucspi-tcp.html). This will spawn a new instance of your program each time a client connects to your program (on the port that you specify). This way, you can focus on the core logic of your program, and let tcpserver handle all of the heavy lifting as far as the socket programming, etc. tcpserver will pass input from the client to your program's stdin, and output from your programs stdout will be sent back to the client.
In got the following problem:
I made a server which is able to handle multiple connection by using select(). But select returns a client(index of FD_SET) also if the socket just got an error like "client disconnect" or whatever.
Is it possible to check a socket without calling recv(). Because to receive I need to get a buffer out of my "BufferPool"
Sample code:
int ret = recv(client, buffer_pool->get(), BUFFER_SIZE, 0);
if(ret == -1) ... // something went wrong
Well then I have to release the buffer again, and it was pretty much a waste of one buffer in my pool. (for a short time)
So isn't it possible to check the socket without calling recv()
I am not sure about the Windows, but using getsockopt() works like a charm on POSIX-compliant systems. Though before you use it - make sure that getting your buffer from the pool is more expensive than making an extra system call. Here is a code snippet:
int my_get_socket_error(int fd)
{
int err_code;
socklen_t len = sizeof(err_code);
if (getsockopt(fd, SOL_SOCKET, SO_ERROR, &err_code, &len) != 0)
err_code = errno;
else
errno = err_code;
return err_code;
}
UPDATE:
According to this document, it seems like Windows supports it too.
No, there is no way to avoid the recv() call. If select() reports that the socket is readable, then you have to read from the socket to determine its new state. If the client disconnected gracefully, recv() will return 0, not -1. If you do not want to waste a pooled buffer, then you will have to read into a temporary local buffer first, and then if recv() returns any data, you can retrieve a pooled buffer and copy the read data into it.
Calling recv and similar function does not work directly with networking devices or something similar.
When you send or receive data, all you do is questioning OS for available data, or to put data in queue for sending. Then OS will do the other job when your code is already went further.
That is why you receive errors after next call of socket function that will "contact" OS networking layers.
It is normal to get errors on that point, and you have to deal with them.
But to prevent blocking sockets and wasting buffers, check out online techniques of implementing or ready libraries that gives you asynchronous way of working with sockets, that way you don't need to define anything before socket will trigger receive callback function where you have to do actual receiving.
As well, it is not good technique to receive big amount of data in one go, because you will face problems with merged or broken apart data through TCP layer, because it is stream based layer. It is recommended to have header in you packets (few bytes) and receive them, that way you don't need pull for header, but only after header you want to read rest of message based on length provided in header. This is just possible example.
After some minutes of work and your help I just receive 1byte before receiving the full amount:
SOCKET client = ...;
char temp = 0x00;
int len = recv(client, &temp, 1, 0);
if(len == 0)
{
// .. client error handling
return;
}
char* buffer = m_memory_pool->Get();
len = recv(client, buffer + 1, m_memory_pool->buffer_size() - 1, 0);
buffer[0] = temp;
// data handling
I tried also to set a timeout for recv() but seems that under Windows it does not work, this is my code:
...
long timeout_ms = 10;
struct timeval interval = {timeout_ms / 1000, (timeout_ms % 1000) * 1000};
if (interval.tv_sec < 0 || (interval.tv_sec == 0 && interval.tv_usec <= 0))
{
interval.tv_sec = 0;
interval.tv_usec = 10000;
}
setsockopt(s_sktIx, SOL_SOCKET, SO_RCVTIMEO, (char *)&interval, sizeof(struct timeval));
...
I'm designing a distributed server/client system with C++, in which many clients send request to many servers through TCP and server throw a thread to handle the request and send back it's response. In my use case only limited number of clients will access the server and I need very high performance.The data sent from client and server are all small, but are very frequent. So creating a connection and tearing it down it after use is expensive. So I want to use connection caching to solve this problem: once connection created, it will be stored in a cache for future use.(Assume that the number of clients will not beyond the size of cache).
My question is:
I saw someone said that connection pooling is a client side technique. If this connection pooling is only used in client side, then first time it make connection to a server, and send data. This action of making connection triggers the accept() function in server side which return a socket for receiving from client. So when client wants to use a existing connection(in cache), it doesn't make new connection, but just send data. The problem is, if no making connection, who would trigger accept() in server side and to throw a thread?
If connection pooling also need to be implemented in server side, how can I know where a request come from? Since only from accept() I can get the client address, but meanwhile accept() already make a new socket for that request, so no point to use a cached connection.
Any answer and suggestion will be appreciated. Or any one can give me an example of connection pool or connection caching?
I saw someone said that connection pooling is a client side technique. ... if no making connection, who would trigger accept() in server side and to throw a thread?
Firstly, connection pooling is not just a client-side technique; it's a connection-mode technique. It applies to both types of peer (the "server" and the "client").
Secondly, accept doesn't need to be called to start a thread. Programs can start threads for any reason they like... They could start threads just to start more threads, in a massively parallelised loop of thread creation. (edit: we call this a "fork bomb")
Finally, an efficient thread-pooling implementation won't start a thread for each client. Each thread typically occupies between 512KB-4MB (counting stack space and other context information), so if you have 10000 clients each occupying that much, that's a lot of wasted memory.
I want to do so, but just don't know how to do it in multithreading case.
You shouldn't use multithreading here... At least, not until you have a solution that uses a single thread, and you decide that it's not fast enough. At the moment you don't have that information; you're just guessing, and guessing doesn't guarantee optimisation.
At the turn of the century there were FTP servers that solved the C10K problem; they were able to handle 10000 clients at any given time, browsing, downloading or idling as users tend to do on FTP servers. They solved that problem not by using threads, but by using non-blocking and/or asynchronous sockets and/or calls.
To clarify, those web servers handled thousands of connections on a single thread! One typical way is to use select, but I'm not particularly fond of that method because it requires a rather ugly series of loops. I prefer to use ioctlsocket for Windows and fcntl for other POSIX OSes to set the file descriptor into non-blocking mode, e.g.:
#ifdef WIN32
ioctlsocket(fd, FIONBIO, (u_long[]){1});
#else
fcntl(fd, F_SETFL, fcntl(fd, F_GETFL, 0) | O_NONBLOCK);
#endif
At this point, recv and read won't block when operating on fd; if there's no data available, they'll return an error value immediately rather than waiting for data to arrive. That means you can loop on multiple sockets.
If connection pooling also need to be implemented in server side, how can I know where a request come from?
Store the client fd along-side its struct sockaddr_storage and any other stateful information you need to store about clients, in a struct that you declare however you feel. If this ends up being 4KB (which is a fairly large struct, usually about as large as they need to get) then 10000 of these will only occupy about 40000KB (~40MB). Even the mobile phones of today should have no problems handling that. Consider completing the following code for your needs:
struct client {
struct sockaddr_storage addr;
socklen_t addr_len;
int fd;
/* Other stateful information */
};
#define BUFFER_SIZE 4096
#define CLIENT_COUNT 10000
int main(void) {
int server;
struct client client[CLIENT_COUNT] = { 0 };
size_t client_count = 0;
/* XXX: Perform usual bind/listen */
#ifdef WIN32
ioctlsocket(server, FIONBIO, (u_long[]){1});
#else
fcntl(server, F_SETFL, fcntl(server, F_GETFL, 0) | O_NONBLOCK);
#endif
for (;;) {
/* Accept connection if possible */
if (client_count < sizeof client / sizeof *client) {
struct sockaddr_storage addr = { 0 };
socklen_t addr_len = sizeof addr;
int fd = accept(server, &addr, &addr_len);
if (fd != -1) {
# ifdef WIN32
ioctlsocket(fd, FIONBIO, (u_long[]){1});
# else
fcntl(fd, F_SETFL, fcntl(fd, F_GETFL, 0) | O_NONBLOCK);
# endif
client[client_count++] = (struct client) { .addr = addr
, .addr_len = addr_len
, .fd = fd };
}
}
/* Loop through clients */
char buffer[BUFFER_SIZE];
for (size_t index = 0; index < client_count; index++) {
ssize_t bytes_recvd = recv(client[index].fd, buffer, sizeof buffer, 0);
# ifdef WIN32
int closed = bytes_recvd == 0
|| (bytes_recvd < 0 && WSAGetLastError() == WSAEWOULDBLOCK);
# else
int closed = bytes_recvd == 0
|| (bytes_recvd < 0 && errno == EAGAIN) || errno == EWOULDBLOCK;
# endif
if (closed) {
close(client[index].fd);
client_count--;
memmove(client + index, client + index + 1, (client_count - index) * sizeof client);
continue;
}
/* XXX: Process buffer[0..bytes_recvd-1] */
}
sleep(0); /* This is necessary to pass control back to the kernel,
* so it can queue more data for us to process
*/
}
}
Supposing you want to pool connections on the client-side, the code would look very similar, except obviously there would be no need for the accept-related code. Supposing you have an array of clients that you want to connect, you could use non-blocking connect calls to perform all of the connections at once like this:
size_t index = 0, in_progress = 0;
for (;;) {
if (client[index].fd == 0) {
client[index].fd = socket(/* TODO */);
# ifdef WIN32
ioctlsocket(client[index].fd, FIONBIO, (u_long[]){1});
# else
fcntl(client[index].fd, F_SETFL, fcntl(client[index].fd, F_GETFL, 0) | O_NONBLOCK);
# endif
}
# ifdef WIN32
in_progress += connect(client[index].fd, (struct sockaddr *) &client[index].addr, client[index].addr_len) < 0
&& (WSAGetLastError() == WSAEALREADY
|| WSAGetLastError() == WSAEWOULDBLOCK
|| WSAGetLastError() == WSAEINVAL);
# else
in_progress += connect(client[index].fd, (struct sockaddr *) &client[index].addr, client[index].addr_len) < 0
&& (errno == EALREADY
|| errno == EINPROGRESS);
# endif
if (++index < sizeof client / sizeof *client) {
continue;
}
index = 0;
if (in_progress == 0) {
break;
}
in_progress = 0;
}
As for optimisation, given that this should be able to handle 10000 clients with perhaps a few minor tweaks, you shouldn't need multiple threads.
Nonetheless, by associating items from a mutex collection with clients and preceding the non-blocking socket operation with a non-blocking pthread_mutex_trylock, the above loops could be adapted to run simultaneously in multiple threads whilst processing the same group of sockets. This provides a working model for all POSIX-compliant platforms, be it Windows, BSD or Linux, but it's not a perfectly optimal one. To achieve optimality, we must step into the asynchronous world, which varies from system to system:
Windows uses WSA* functions with call-backs.
BSD and Linux use the somewhat similar kqueue and epoll, respectively.
It may pay to codify that "non-blocking socket operation" abstraction mentioned earlier, as the two asynchronous mechanisms vary significantly in respect to their interface. Like everything else, unfortunately we must write abstractions so that our Windows-relevant code remains legible on POSIX-compliant systems. As a bonus, this'll allow us to mingle server-processing (i.e. accept and anything that follows) with client-processing (i.e. connect and anything that follows), so our server loop can become a client loop (or vice-versa).
i have a server and client classes but the problem is: when i make infinite loop to accept incoming connection i cant receive all the data received from the client while accepting the connections because accept blocks until the connection is accepted, my code:
for (;;)
{
boost::thread thread(boost::bind(&Irc::Server::startAccept, &s));
thread.join();
for (ClientsMap::const_iterator it = s.begin(); it != s.end(); ++it)
{
std::string msg = getData(it->second->recv());
std::clog << "Msg: " << msg << std::endl;
}
}
You need either multiple threads or a call to select/poll to find out which connections have unprocessed data. IBM has a nice example here, which will work on any flavor of Unix, Linux, BSD, etc. (you might need different header files depending on the OS).
Right now you're starting a thread and then waiting for it immediately, which results in sequential execution and completely defeats the purpose of threads.
Take a look here : http://www.boost.org/doc/libs/1_38_0/doc/html/boost_asio/examples.html
especially the HTTP Server 3 example, thats exactly what you are looking for , all you have to do is change that code a little bit for your needs :) and your done
A good approach would be to create one thread that only accepts new connections. That's where you have a listener socket. Then, for every connection that gets accepted, you have a new connected socket, so you can spawn another thread, giving it the connected socket as a parameter. That way, your thread that accepts connections doesn't get blocked, and can connect to many clients very fast. The processing threads deal with the clients and then they exit.
I don't even know why need to wait for them, but if you do, you may deal with it in some other way, depending on the OS and/or libraries that you use (messages, signals etc can be used).
If you don't want to spawn a new thread for each connected client, then as Ben Voigt suggested, you can use select. That is another good approach if you want to make it single threaded. Basically, all your sockets will be in an array of socket descriptors and using select you will know what happened (someone connected, socket is ready for read/write, socket got disconnected etc) and act accordingly.
Here's one example Partial one, but it works. you just accept connections in the acceptConnections(), which will then spawn a separate thread for each client. That's where you communicate with the clients. It's from a windows code that i have lying around, but it's very easy to be reimplemented for any platform.
typedef struct SOCKET_DATA_ {
SOCKET sd;
/* other parameters that you may want to pass to the clientProc */
} SOCKET_DATA;
/* In this function you communicate with the clients */
DWORD WINAPI clientProc(void * param)
{
SOCKET_DATA * pSocketData = (SOCKET_DATA *)param;
/* Communicate with the new client, and at the end deallocate the memory for
SOCKET_DATA and return.
*/
delete pSocketData;
return 0;
}
int acceptConnections(const char * pcAddress, int nPort)
{
sockaddr_in sinRemote;
int nAddrSize;
SOCKET sd_client;
SOCKET sd_listener;
sockaddr_in sinInterface;
SOCKET_DATA * pSocketData;
HANDLE hThread;
sd_listener = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);
if (INVALID_SOCKET == sd_listener) {
fprintf(stderr, "Could not get a listener socket!\n");
return 1;
}
sinInterface.sin_family = AF_INET;
sinInterface.sin_port = nPort;
sinInterface.sin_addr.S_un.S_addr = INADDR_ANY;
if (SOCKET_ERROR != bind(sd_listener, (sockaddr*)&sinInterface, sizeof(sockaddr_in))) {
listen(sd_listener, SOMAXCONN);
} else {
fprintf(stderr, "Could not bind the listening socket!\n");
return 1;
}
while (1)
{
nAddrSize = sizeof(sinRemote);
sd_client = accept(sd_listener, (sockaddr*)&sinRemote, &nAddrSize);
if (INVALID_SOCKET == sd_client) {
fprintf(stdout, "Accept failed!\n");
closesocket(sd_listener);
return 1;
}
fprintf(stdout, "Accepted connection from %s:%u.\n", inet_ntoa(sinRemote.sin_addr), ntohs(sinRemote.sin_port));
pSocketData = (SOCKET_DATA *)malloc(sizeof(SOCKET_DATA));
if (!pSocketData) {
fprintf(stderr, "Could not allocate memory for SOCKET_DATA!\n");
return 1;
}
pSocketData->sd = sd_client;
hThread = CreateThread(0, 0, clientProc, pSocketData, 0, &nThreadID);
if (hThread == INVALID_HANDLE_VALUE) {
fprintf(stderr, "An error occured while trying to create a thread!\n");
delete pSocketData;
return 1;
}
}
closesocket(sd_listener);
return 0;
}