The following is a simple code which just sends data of a processor i to i+1 using Isend and probes if all sends are complete.
Code:
std::vector<double> sendbuffer;
int myrank, nprocs;
std::vector <MPI::Request> req_v;
MPI::Init ();
nprocs = MPI::COMM_WORLD.Get_size ();
myrank = MPI::COMM_WORLD.Get_rank ();
sendbuffer.resize(20000);
int startin = 0;
if(myrank != nprocs-1)
for (int i = myrank+1; i <= (myrank+1);i++)
{
int sendrank = i;
int msgtag = myrank * sendrank;
int msgsz = sendbuffer.size();
double *sdata = &(sendbuffer[startin]);
MPI::Request req;
req = MPI::COMM_WORLD.Isend (sdata, msgsz, MPI::DOUBLE, sendrank, msgtag);
req_v.push_back (req);
}
printf("Size-%d %d\n", myrank,(int) req_v.size());
if(req_v.size() > 0)
MPI::Request::Waitall (req_v.size(), &(req_v[0]));
printf("Over-%d\n", myrank);
MPI::COMM_WORLD.Barrier();
MPI::Finalize ();
The code completes for a buffer size of 1500 but for 20000 it stalls. The behavior seems a bit strange. I thought matching receives are not needed for Isend. Please suggest possible reasons for this behavior.
You never actually receive the messages. Without the corresponding calls to Recv (or Irecv), the send calls may never complete.
You can get more details at this question, but generally, sends are allowed to complete without the corresponding receive having been posted as long as the messages can be buffered within MPI (either on the sender side or the receiver side). Eventually, your system will run out of buffers and the send call will stop completing until you call the corresponding receive calls and free up some system buffers.
Related
What I have to do is a class which basically has 3 methods :
send(message, dst) which keeps sending messages (and maybe add delay with a sleep(t) and an increasing t) to dst until receiving an ACK.
listen() which receives messages and delivers ACKs. If the message is an ACK, destroys the thread who sent the msg acquitted.
shutdown() which stops every communication (every thread) and writes a log.
For the ACK mechanism, I thought of tables :
host_map[port][ipadress][id] // Also used for other things which require (port,adress) =>id mapping and also because all host have ids.
ACK[id][message][thread_to_stop] // Will be used to destroy threads except I didn't know how to put infos about the thread here and don't know where to put a join() if I even have to
A vector of threads (or threads id, idk) to stop all the threads when I call shutdown().
I want send() and listen() to be parallelized. In other words, listen() should not block the program so it should be in a thread. Also it needs to keep receiving stuff so it would need a while loop.
In my main.cpp :
A link = A(my_ip,port);
A.listen();
for (int i, i < 5, i++)
link.send(to_string(i), dst_ip, dst_port);
This should is supposed to make 5 threads which have while loop where they send i and then sleep a little, repeat until I receive an ACK.
I am new to C++ and never did multi-threading before so I don't even know if I can do this and don't even know where to put my join() if there is any.
Another thing that I thought and don't know if it's possible is to have a queue inside my class Link which keeps sending stuff and have send(msg) just adding it to the queue.
Here is something I made in A.hpp.
void sendm(std::string m, in_addr_t dst_ip, unsigned short dst_port){
int id = resolveId(dst_port, dst_ip);
int seq_number = resolveSeq(); // TODO
std::thread th= (&A::sendMessage, this, m, dst_ip, dst_port);
// I need something to be able to add th or an ID in my ACK table and thread vector.
th.join();
}
void sendMessage(std::string m, in_addr_t dst_ip, unsigned short dst_port){
//dst
struct in_addr dst_ipt;
struct sockaddr_in dst;
dst_ipt.s_addr = dst_ip;
dst.sin_family = AF_INET;
dst.sin_port = htons(dst_port);
dst.sin_addr = dst_ipt;
int t = 50;
while(true){
std::cout << "send message m = " << m << "\n";
//Sends a message to dst_ip through dst_port and increments the number of messages sent.
if (sendto(obj_socket, m.c_str(), m.size(), 0, reinterpret_cast<const sockaddr*>(&dst), sizeof(dst)) < 0){
std::cerr << " Error sendto\n";
exit(EXIT_FAILURE);
};
std::cout<< "message sent\n";
std::this_thread::sleep_for(std::chrono::milliseconds(t));
t+=10;
}
}
void receive(){
char buffer[1500];
sockaddr_in from;
socklen_t fromlen = sizeof(from);
ssize_t tmp = recvfrom(obj_socket, buffer, 1500, 0, reinterpret_cast<sockaddr*>(&from), &fromlen);
if (tmp < 0){
std::cout << "Exit\n";
std::cerr << "Error receive from\n";
exit(EXIT_FAILURE);
} else{
if (tmp >= 0){
int id = resolveId(from.sin_port, from.sin_addr.s_addr);
if (!verifySomething(m,id)){
doSomethingCool(m,id);
}
}
}
}
My listen() would just be a threaded version of while(true)receive();
Idk if this version compiles to be honest. I keep changing it every 2 minutes. Without the while loop in the send() and the threading, it works so far.
I didn't really implement the ACK mechanism yet.
Thank you for reading and for your help.
I'm adapting a tcp PubSub example to using inproc with multithread. It ends up hanging forever.
My setup
macOS Mojave, Xcode 10.3
zmq 4.3.2
The source code reeproducing the issue:
#include <string.h>
#include <stdio.h>
#include <unistd.h>
#include <thread>
#include "zmq.h"
void hello_pubsub_inproc() {
void* context = zmq_ctx_new();
void* publisher = zmq_socket(context, ZMQ_PUB);
printf("Starting server...\n");
int pub_conn = zmq_bind(publisher, "inproc://*:4040");
void* subscriber = zmq_socket(context, ZMQ_SUB);
printf("Collecting stock information from the server.\n");
int sub_conn = zmq_connect(subscriber, "inproc://localhost:4040");
sub_conn = zmq_setsockopt(subscriber, ZMQ_SUBSCRIBE, 0, 0);
std::thread t_pub = std::thread([&]{
const char* companies[2] = {"Company1", "Company2"};
int count = 0;
for(;;) {
int which_company = count % 2;
int index = (int)strlen(companies[0]);
char update[12];
snprintf(update, sizeof update, "%s",
companies[which_company]);
zmq_msg_t message;
zmq_msg_init_size(&message, index);
memcpy(zmq_msg_data(&message), update, index);
zmq_msg_send(&message, publisher, 0);
zmq_msg_close(&message);
count++;
}
});
std::thread t_sub = std::thread([&]{
int i;
for(i = 0; i < 10; i++) {
zmq_msg_t reply;
zmq_msg_init(&reply);
zmq_msg_recv(&reply, subscriber, 0);
int length = (int)zmq_msg_size(&reply);
char* value = (char*)malloc(length);
memcpy(value, zmq_msg_data(&reply), length);
zmq_msg_close(&reply);
printf("%s\n", value);
free(value);
}
});
t_pub.join();
// Give publisher time to set up.
sleep(1);
t_sub.join();
zmq_close(subscriber);
zmq_close(publisher);
zmq_ctx_destroy(context);
}
int main (int argc, char const *argv[]) {
hello_pubsub_inproc();
return 0;
}
The result
Starting server...
Collecting stock information from the server.
I've also tried adding this before joining threads to no avail:
zmq_proxy(publisher, subscriber, NULL);
The workaround: Replacing inproc with tcp fixes it instantly. But shouldn't inproc target in-process usecases?
Quick research tells me that it couldn't have been the order of bind vs. connect, since that problem is fixed in my zmq version.
The example below somehow tells me I don't have a missing shared-context issue, because it uses none:
ZeroMQ Subscribers not receiving message from Publisher over an inproc: transport class
I read from the Guide in the section Signaling Between Threads (PAIR Sockets) that
You can use PUB for the sender and SUB for the receiver. This will correctly deliver your messages exactly as you sent them and PUB does not distribute as PUSH or DEALER do. However, you need to configure the subscriber with an empty subscription, which is annoying.
What does it mean by an empty subscription?
Where am I doing wrong?
You can use PUB for the sender and SUB for the receiver. This will correctly deliver your messages exactly as you sent them and PUB does not distribute as PUSH or DEALER do. However, you need to configure the subscriber with an empty subscription, which is annoying.
Q : What does it mean by an empty subscription?
This means to set ( configure ) a subscription, driving a Topic-list message-delivery filtering, using an empty subscription string.
Q : Where am I doing wrong?
Here :
// sub_conn = zmq_setsockopt(subscriber, ZMQ_SUBSCRIBE, 0, 0); // Wrong
sub_conn = zmq_setsockopt(subscriber, ZMQ_SUBSCRIBE, "",0); // Empty string
Doubts also here, about using a proper syntax and naming rules :
// int pub_conn = zmq_bind(publisher, "inproc://*:4040");
int pub_conn = zmq_bind(publisher, "inproc://<aStringWithNameMax256Chars>");
as inproc:// transport-class does not use any kind of external stack, but maps the AccessPoint's I/O(s) onto 1+ memory-locations ( a stack-less, I/O-thread not requiring transport-class ).
Given this, there is nothing like "<address>:<port#>" being interpreted by such (here missing) protocol, so the string-alike text gets used as-is for identifying which Memory-location are the message-data going to go into.
So, the "inproc://*:4040" does not get expanded, but used "literally" as a named inproc:// transport-class I/O-Memory-location identified as [*:4040] ( Next, asking a .connect()-method of .connect( "inproc://localhost:4040" ) will, and must do so, lexically miss the prepared Memory-location: ["*:4040"] as the strings do not match
So this ought fail to .connect() - error-handling might be silent, as since the versions +4.x there is not necessary to obey the historical requirement to first .bind() ( creating a "known" named-Memory-Location for inproc:// ) before one may call a .connect() to get it cross-connected with an "already existing" named-Memory-location, so the v4.0+ will most probably not raise any error on calling and creating a different .bind( "inproc://*:4040" ) landing-zone and next asking a non-matching .connect( "inproc://localhost:4040" ) ( which does not have a "previously prepared" landing-zone in an already existing named-Memory-location.
I am using CodeBlocks with MinGW compiler and wxWidgets library.
I am writing a program that read some data from the microcontroller, by sending messages (using index and subindex) and getting response messages with said data.
My plan was to send messages one-by-one and waiting for response message using __atomic int variables__ to check when I get response message.
This is my function for sending a message:
typedef std::chrono::high_resolution_clock Clock;
void sendSDO(int index, int subindex)
{
int nSent = 0;
atomic_index.store(index);
atomic_subindex.store(subindex);
canOpenClient->SDORead(index, subindex);
auto start = Clock::now();
nSentMessages++;
nSent++;
Sleep(10);
while ((atomic_index.load() != 0) && (atomic_subindex.load() != 0))
{
auto t = chrono::duration_cast<chrono::milliseconds>(Clock::now() - start);
if(t.count() > 20)
{
if (nSent > 5)
{
MainFrame->printTxt("[LOG] response not received\n");
return;
}
atomic_index.store(index);
atomic_subindex.store(subindex);
canOpenClient->SDORead(index, subindex);
nSentMessages++;
nSent++;
start = Clock::now();
}
}
}
Pseudocode is set atomic int to index and subindex of what I want value I want to read from microcontroller, then send message to it SDORead(), and if no response was received in 20 ms, send the message again, up to 5 times.
For receiving messages, I have a __separate thread__ with a callback function which is called when I get response message from the controller:
void notifyEvent(unsigned char ev_type)
{
SDO_msg_t msg;
msg = canOpenClient->Cmd_CustomMessageGet(); //get response message
if(ev_type == CO_EVENT_SDO_READ)
{
if ((msg.index == atomic_index.load()) && (msg.subindex == atomic_subindex.load()))
{
//does stuff, like saves message data to set container
atomic_index.store(0);
atomic_subindex.store(0);
}
}
if (message data not in container)
printf("not in container!")
}
Here I set the same atomic int values to 0, when the correct response message is received, and save response message data
I also have variables nSentMessages and nReceivedMessages, which hold the number of messages sent and messages received. I check at the end if these values are the same. Normally, I wouldn't need this (since I wait for every response), I put it there as an extra safety measure.
Now onto the problem:
1) My problem is in callback function notifyEvent(), where I presumably save response message data to a container, but I still sometimes get "not in container!" from that if statement and I don't know why. (My container is just normal set set<EDSobject, cmp> container, it's not atomic or anything, since I know there won't be reads/writes to it at the same time from different threads.)
2) If you check my function sendSDO(), there is a line Sleep(10). The program works ok with it, but if I remove it, the program returns a different value for nSentMessages and nReceivedMessages - 576 and 575. This happens every time I run the program and I don't understand why.
I am trying to write a gSoap server, which needs to save the incoming message. I want to save it in a buffer. Here is how gSoap Documentation does it on a client side to save the outgoing message...
//from gSoap Documentation
//SOURCE: https://www.genivia.com/doc/soapdoc2.html#tth_sEc19.7
int mysend(struct soap *soap, const char *s, size_t n)
{
struct buffer *h = (struct buffer*)soap->user; // get buffer through handle
int m = h->max, k = h->len + n;
// need to increase space?
if (m == 0)
m = 1024;
else
while (k >= m)
m *= 2;
if (m != h->max)
{
char *buf = malloc(m);
memcpy(buf, h->buf, h->len);
h->max = m;
if(h->buf)
free(h->buf);
h->buf = buf;
}
memcpy(h->buf + h->len, s, n);
h->len += n;
return SOAP_OK;
}
this works with some modification, but if I take this same idea to the server side, store to a buffer and end it with this return statement...
size_t myrecv(struct soap *soap, char *s, size_t n){
//do similar to above example...
...
return default_frecv(soap,s,n);
}
it only stores the message going from the server back to the client. I need to save the message coming to the server from the client. I thought recv would give me the incoming message, but this is not the case. Any Ideas? Any help, suggestions, or ideas are appreciated! Thanks in Advance!
source to the mysend example: https://www.genivia.com/doc/soapdoc2.html#tth_sEc19.7
I need to save the message coming to the server from the client. I
thought recv would give me the incoming message, but this is not the
case.
Hmmm. The documentation clearly says that frecv() is the callback that receives all messages, regardless whether it is the client side implementation or the server side implementation. So defining your own frecv() callback should so it. If you get stuck, take a look at the gsoap/plugin/logging.c message logger that implements this callback to capture the message received and sends it to a file/pipe.
I was able to hook into frecv() and do my own additional processing after invoking the default frecv() function. However, a new purpose requires me to parse the buffer before the default frecv(), at which point it doesn't yet exist. Once the default frecv() is called, the buffer exists, but the action taken on the buffer (namely soap_serve()), which I wanted to intercept, has already occurred, so I'm one request behind in my derived context.
My intent was to parse the buffer for the name of the WSDL, in order to assign a unique namespace to soap_serve(), but it's not working.
I'm coding a TCP Server class based on the I/O multiplexing (select) way.
The basic idea is explained in this chunk of code:
GenericApp.cpp
TServer *server = new Tserver(/*parameters*/);
server->mainLoop();
For now the behavior of the server is independent from the context but in a way that i nedd to improove.
Actual Status
receive(sockFd , buffer);
MSGData * msg= MSGFactory::getInstance()->createMessage(Utils::getHeader(buffer,1024));
EventHandler * rightHandler =eventBinder->getHandler(msg->type());
rightHandler->callback(msg);
At this version the main loop reads from the socket, instantiates the right type of message object and calls the appropriate handler(something may not work properly because it compiles but i have not tested it).
As you can notice this allows a programmer to define his message types and appropriate handlers but once the main loop is started nothing can be done.
I need to make this part of the server more customizable to adapt this class to a bigger
quantity of problems.
MainLoop Code
void TServer::mainLoop()
{
int sockFd;
int connFd;
int maxFd;
int maxi;
int i;
int nready;
maxFd = listenFd;
maxi = -1;
for(i = 0 ; i< FD_SETSIZE ; i++) clients[i] = -1; //Should be in the constructor?
FD_ZERO(&allset); //Should be in the constructor?
FD_SET(listenFd,&allset); //Should be in the constructor?
for(;;)
{
rset = allset;
nready = select (maxFd + 1 , &rset , NULL,NULL,NULL);
if(FD_ISSET( listenFd , &rset ))
{
cliLen = sizeof(cliAddr);
connFd = accept(listenFd , (struct sockaddr *) &cliAddr, &cliLen);
for (i = 0; i < FD_SETSIZE; i++)
{
if (clients[i] < 0)
{
clients[i] = connFd; /* save descriptor */
break;
}
}
if (i == FD_SETSIZE) //!!HANDLE ERROR
FD_SET(connFd, &allset); /* add new descriptor to set */
if (connFd > maxFd) maxFd = connFd; /* for select */
if (i > maxi) maxi = i; /* max index in client[] array */
if (--nready <= 0) continue;
}
for (i = 0; i <= maxi; i++)
{
/* check all clients for data */
if ( (sockFd = clients[i]) < 0) continue;
if (FD_ISSET(sockFd, &rset))
{
//!!SHOULD CLEAN BUFFER BEFORE READ
receive(sockFd , buffer);
MSGData * msg = MSGFactory::getInstance()->createMessage(Utils::getHeader(buffer,1024));
EventHandler * rightHandler =eventBinder->getHandler(msg->type());
rightHandler->callback(msg);
}
if (--nready <= 0) break; /* no more readable descriptors */
}
}
}
Do you have any suggestions on a good way to do this?
Thanks.
Your question requires more than just a stack overflow question. You can find good ideas in these book:
http://www.amazon.com/Pattern-Oriented-Software-Architecture-Concurrent-Networked/dp/0471606952/ref=sr_1_2?s=books&ie=UTF8&qid=1405423386&sr=1-2&keywords=pattern+oriented+software+architecture
http://www.amazon.com/Unix-Network-Programming-Volume-Networking/dp/0131411551/ref=sr_1_1?ie=UTF8&qid=1405433255&sr=8-1&keywords=unix+network+programming
Basically what you're trying to do is a reactor. You can find open source library implementing this pattern. For instance:
http://www.cs.wustl.edu/~schmidt/ACE.html
http://pocoproject.org/
If you want yout handler to have the possibility to do more processing you could give them a reference to your TCPServer and a way to register a socket for the following events:
read, the socket is ready for read
write, the socket is ready for write
accept, the listening socket is ready to accept (read with select)
close, the socket is closed
timeout, the time given to wait for the next event expired (select allow to specify a timeout)
So that the handler can implement all kinds of protocols half-duplex or full-duplex:
In your example there is no way for a handler to answer the received message. This is the role of the write event to let a handler knows when it can send on the socket.
The same is true for the read event. It should not be in your main loop but in the socket read handler.
You may also want to add the possibility to register a handler for an event with a timeout so that you can implement timers and drop idle connections.
This leads to some problems:
Your handler will have to implement a state-machine to react to the network events and update the events it wants to receive.
You handler may want to create and connect new sockets (think about a Web proxy server, an IRC client with DCC, an FTP server, and so on...). For this to work it must have the possibility to create a socket and to register it in your main loop. This means the handler may now receive callbacks for one of the two sockets and there should be a parameter telling the callback which socket it is. Or you will have to implement a handler for each socket and they will comunnicate with a queue of messages. The queue is needed because the readiness of one socket is independent of the readiness of the other. And you may read something on one and not being ready to send it on the other.
You will have to manage the timeouts specified by each handlers which may be different. You may end up with a priority queue for timeouts
As you see this is no simple problem. You may want to reduce the genericity of your framework to simplify its design. (for instance handling only half-duplex protocols like simple HTTP)