I'm writing a C++ program that uses OpenMPI. It executes in "rounds", where in each round, process 0 sends chunks of data to the other processes, they do stuff to it and send results back, and when there are no more chunks to send, process 0 sends a "done" message to each other process. A "done" message is just a single-int message with tag 3. My first round executes fine. However, when I get to round two, processes 1-p "probe" and "receive" a done message before process 0 has had a chance to send anything (let alone a done message).
I've gone over my code many times now and it seems like the only place this message could be coming from is where process 0 sent it in the previous round - but each process had already received that. I'd rather not post my code since it's pretty big, but does anyone know if MPI messages can be received twice like this?
I think I may have the answer... Since the actual data in the done message doesn't matter, I didn't think to have the processes actually receive it. It turns out that in the previous round, the processes were "probing" the message and finding that the tag was 3, then breaking out of their loop. Therefore, in round two, the message was still waiting to be received, so when they called MPI_Probe, they found the same message as in the previous round.
To solve this I just put in a call to MPI_Recv. I looked at MPI_Cancel but I can't find enough information about it to see if it would be appropriate. Sorry for being misleading in my question!
Related
I have a queue which is supposed to receive the messages sent by a lambda function. This function is supposed to send each different message once only. However, I saw a scary amount of receive count on the console:
Since I cannot find any explanation about receive count in the plain English, I need to consult StackOverflow Community. I have 2 theories to verify:
There are actually not so many messages and the reason why "receive count" is that high is simply because I polled the messages for a looooong time so the messages were captured more than once;
the function that sends the messages to the queue is SQS-triggered, those messages might be processed by multiple processors. Though I set VisibilityTimeout already, are the messages which are processed going to be deleted? If they aren't remained, there are no reasons for them to be caught and processed for a second time.
Any debugging suggestion will be appreciated!!
So, receive count is basically the amount of times the lambda (or any other consumer) has received the message. It can be that a consumer receives a message more than once (this is by design, and you should handle that in your logic).
That being said, the receive count also increases if your lambda fails to process the message (or even hits the execution limits). The default is 3 times, so if something with your lambda is wrong, you will have at least 3 receives per message.
Also, when you are polling the message, via the AWS console, you are basically increasing the receive count.
I send messages using ZMQ_PUSH socket with ZMQ_DONTWAIT flag. Each message separated by 300ms (so I seriously doubt, that my thread scheduling is so off). However from time to time several (up to 7 so far) messages are packed to the same TCP frame yet I don't see any retransmissions or reconnections in tcpdump.
Why is it happening? And how can I determine when my message would be sent?
Q : "And how can I determine when my message would be sent?"
You have zero-chance to know when a message, dispatched to be sent, would finally get down to the wire-level.
This is a property of the concept, not a bug. It was explicitly explained in the documentation, since the API v2.0+ ( and will most probably remain so forever ). It is fair to add that some insight into the zmq_context-instance's internalities might be derived from a socket_monitor available in the more recent API versions.
In my MPI program, I want to send and receive information to adjacent processes. But if a process ends and doesn't send anything, its neighbors will wait forever. How can I resolve this issue? Here is what I am trying to do:
if (rank == 0) {
// don't do anything until all slaves are done
} else {
while (condition) {
// send info to rank-1 and rank+1
// if can receive info from rank-1, receive it, store received info locally
// if cannot receive info from rank-1, use locally stored info
// do the same for process rank+1
// MPI_Barrier(slaves); (wait for other slaves to finish this iteration)
}
}
I am going to check the boundaries of course. I won't check rank-1 when process number is 1 and I won't check rank+1 when process is the last one. But how can I achieve this? Should I wrap it with another while? I am confused.
I'd start by saying that MPI wasn't originally designed with your use case in mind. In general, MPI applications all start together and all end together. Not all applications fit into this model though, so don't lose hope!
There are two relatively easy ways of doing this and probably thousands of hard ones:
Use RMA to set flags on neighbors.
As has been pointed out in the comments, you can set up a tiny RMA window that exposes a single value to each neighbor. When a process is done working, it can do an MPI_Put on each neighbor to indicate that it's done and then MPI_Finalize. Before sending/receiving data to/from the neighbors, check to see if the flag is set.
Use a special tag when detecting shutdowns.
The tag value often gets ignored when sending and receiving messages, but this is a great time to use it. You can have two flags in your application. The first (we'll call it DATA) just indicates that this message contains data and you can process it as normal. The second (DONE) indicates that the process is done and is leaving the application. When receiving messages, you'll have to change the value for tag from whatever you're using to MPI_ANY_TAG. Then, when the message is received, check which tag it is. If it's DONE, then stop communicating with that process.
There's another problem with the pseudo-code that you posted however. If you expect to perform an MPI_Barrier at the end of every iteration, you can't have processes leaving early. When that happens, the MPI_Barrier will hang. There's not much you can do to avoid this unfortunately. However, given the code you posted, I'm not sure that the barrier is really necessary. It seems to me that the only inter-loop dependency is between neighboring processes. If that's the case, then the sends and receives will accomplish all of the necessary synchronization.
If you still need a way to track when all of the ranks are done, you can have each process alert a single rank (say rank 0) when it leaves. When rank 0 detects that everyone is done, it can just exit. Or, if you want to leave after some other number of processes is done, you can have rank 0 send out a message to all other ranks with a special tag like above (but add MPI_ANY_SOURCE so you can receive from rank 0).
When there is a "FAULT" and "max.delivery.attempts" processes the configured number of times and even then the process continues in "FAULT" logo in the following section "Message Processor Turns" without manual intervention it activates again? The fact that the "Message Processor" DISABLED can not impact the reading of new messages in JMS queue.
Since the Message store and process story implemented in the way to served as First Come First Out basis it is not possible to skip the message that got fault and continue the message flow.
Nevertheless up coming release has a new improvement where you can drop the message out from the queue after x number of fail attempts. Having said that, it is not good practice while you do the schedule and process.
To understand further about Message-stores and Message-processors read on the given article
In order to avoid this situation you can use the Sampling processor and send the message to back-end. Sample process will immediately remove it from the queue and process further. If the delivery of the message is failed or if you find fault you can re added in to store in Fault sequence.
I am using $SUB for the first time and have come across this problem. Both, client and server use overlapped operations and here is the specific situation I have a problem with.
Client
C1. Connects to the server.
C2. Sends the message bigger than a pipe buffer and buffer passed to overlapped read operation in the server.
C3. Successfully cancels the send operation.
Server
S1. Creates and waits for the client.
S2. When the client is connected, it reads the message.
S21. Because message doesn't fit into the buffer(ERROR_MORE_DATA), it is read part by part.
It seems to me that there is no way to tell when is the whole message, as an isolated unit, canceled. In particular, if client cancels the send operation, server does not receive the whole message, just a part of it, and consequent read operation returns with ERROR_IO_PENDING (in my case), which means there is no data to be read and read operation has been queued. I would expect to have some kind of means telling the reader that the message has been canceled, so that reader can act upon it.
However, relevant documentation is scatter over MSDN, so I may as well be missing something. I would really appreciate if anyone can shed some light on it. Thanks.
You are correct, there is no way to tell.
If you cancel the Writefile partway through, only part of the message will be written, so only that part will be read by the server. There is no "bookkeeping" information sent about how large the message was going to be before you cancelled it - what is sent is just the raw data.
So the answer is: Don't cancel the IO, just wait for it to succeed.
If you do need to cancel IO partway through, you should probably cut the connection and start again from the beginning, just as you would for a network outage.
(You could check your OVERLAPPED structure to find out how much was actually written, and carry on from there, but if you wanted to do that you would probably just not cancel the IO in the first place.)
Why did you want to cancel the IO anyway? What set of circumstances triggers this requirement?