I have a simple program that when given a tupple message containing {pid,integer} will send a message back to the processor with its PID and the interger+1. The problem is that I need this program to be left active so i can send it multiple messages, and then when i flush() it, will send back its mailbox all at once. It only works 1 message at a time. I tried a recursion but it doesn't work. Here is what I have.
defmodule Spawner do
def start() do
spawn(fn ->
receive do
{pid,y} -> send(pid,y+1)
Spawner.start()
end
end)
end
end
Then on the terminal i would do:
> x = Spawner.start()
> send x, {self(),3}
> send x, {self(),5}
> flush()
#⇒ output: {PID,4}
I need the output to be {PID,4} and {PID,6}.
Thank you for your time.
Think about send as about ping-pong game. The rule is: one send ⇒ one consume. Exactly as in ping-pong one can not expect the proper behaviour from the opposite side, serving ten balls at once.
To accomplish what you want you are going to have a GenServer that collects all the received messages (instead of immediately answering to each of them.)
Also it would provide, say, get_all call, that would retrieve all the collected messages from it’s state and respond with the {int, list} tuple:
{PID, [msg1, msg2, ..., msgN]}
The implementation of that won’t fit the margins here, but since you have your question tagged with elixir, GenServer tutorial would be a good start. Then you might want to read about Agent to hold the state.
Other way round (I do not recommend it) would be flush() the consumer recursively with a timeout. The empty queue would trigger a timeout. But, again, it’s not how it’s supposed to be done, because you probably want all the already sent messages to be collected somehow on the other side.
Related
I'm creating an async gRPC server in C++. One of the methods streams data from the server to clients - it's used to send data updates to clients. The frequency of the data updates isn't predictable. They could be nearly continuous or as infrequent as once per hour. The model used in the gRPC example with the "CallData" class and the CREATE/PROCESS/FINISH states doesn't seem like it would work very well for that. I've seen an example that shows how to create a 'polling' loop that sleeps for some time and then wakes up to check for new data, but that doesn't seem very efficient.
Is there another way to do this? If I use the "CallData" method can it block in the 'PROCESS' state until there's data (which probably wouldn't be my first choice)? Or better, can I structure my code so I can notify a gRPC handler when data is available?
Any ideas or examples would be appreciated.
In a server-side streaming example, you probably need more states, because you need to track whether there is currently a write already in progress. I would add two states, one called WRITE_PENDING that is used when a write is in progress, and another called WRITABLE that is used when a new message can be sent immediately. When a new message is produced, if you are in state WRITABLE, you can send immediately and go into state WRITE_PENDING, but if you are in state WRITE_PENDING, then the newly produced message needs to go into a queue to be sent after the current write finishes. When a write finishes, if the queue is non-empty, you can grab the next message from the queue and immediately start a write for it; otherwise, you can just go into state WRITABLE and wait for another message to be produced.
There should be no need to block here, and you probably don't want to do that anyway, because it would tie up a thread that should otherwise be polling the completion queue. If all of your threads wind up blocked that way, you will be blind to new events (such as new calls coming in).
An alternative here would be to use the C++ sync API, which is much easier to use. In that case, you can simply write straight-line blocking code. But the cost is that it creates one thread on the server for each in-progress call, so it may not be feasible, depending on the amount of traffic you're handling.
I hope this information is helpful!
I am working on a project that will require multiple workers to access the same queue to get information about a file which they will manipulate. Files are ranging from size, from mere megabytes to hundreds of gigabytes. For this reason, a visibility timeout doesn't seem to make sense because I cannot be certain how long it will take. I have though of a couple of ways but if there is a better way, please let me know.
The message is deleted from the original queue and put into a
‘waiting’ queue. When the program finished processing the file, it
deletes it, otherwise the message is deleted from the queue and put
back into the original queue.
The message id is checked with a database. If the message id is
found, it is ignored. Otherwise the program starts processing the
message and inserts the message id into the database.
Thanks in advance!
Use the default-provided SQS timeout but take advantage of ChangeMessageVisibility.
You can specify the timeout in several ways:
When the queue is created (default timeout)
When the message is retrieved
By having the worker call back to SQS and extend the timeout
If you are worried that you do not know the appropriate processing time, use a default value that is good for most situations, but don't make it so big that things become unnecessarily delayed.
Then, modify your workers to make a ChangeMessageVisiblity call to SQS periodically to extend the timeout. If a worker dies, the message stops being extended and it will reappear on the queue to be processed by another worker.
See: MessageVisibility documentation
In my MPI program, I want to send and receive information to adjacent processes. But if a process ends and doesn't send anything, its neighbors will wait forever. How can I resolve this issue? Here is what I am trying to do:
if (rank == 0) {
// don't do anything until all slaves are done
} else {
while (condition) {
// send info to rank-1 and rank+1
// if can receive info from rank-1, receive it, store received info locally
// if cannot receive info from rank-1, use locally stored info
// do the same for process rank+1
// MPI_Barrier(slaves); (wait for other slaves to finish this iteration)
}
}
I am going to check the boundaries of course. I won't check rank-1 when process number is 1 and I won't check rank+1 when process is the last one. But how can I achieve this? Should I wrap it with another while? I am confused.
I'd start by saying that MPI wasn't originally designed with your use case in mind. In general, MPI applications all start together and all end together. Not all applications fit into this model though, so don't lose hope!
There are two relatively easy ways of doing this and probably thousands of hard ones:
Use RMA to set flags on neighbors.
As has been pointed out in the comments, you can set up a tiny RMA window that exposes a single value to each neighbor. When a process is done working, it can do an MPI_Put on each neighbor to indicate that it's done and then MPI_Finalize. Before sending/receiving data to/from the neighbors, check to see if the flag is set.
Use a special tag when detecting shutdowns.
The tag value often gets ignored when sending and receiving messages, but this is a great time to use it. You can have two flags in your application. The first (we'll call it DATA) just indicates that this message contains data and you can process it as normal. The second (DONE) indicates that the process is done and is leaving the application. When receiving messages, you'll have to change the value for tag from whatever you're using to MPI_ANY_TAG. Then, when the message is received, check which tag it is. If it's DONE, then stop communicating with that process.
There's another problem with the pseudo-code that you posted however. If you expect to perform an MPI_Barrier at the end of every iteration, you can't have processes leaving early. When that happens, the MPI_Barrier will hang. There's not much you can do to avoid this unfortunately. However, given the code you posted, I'm not sure that the barrier is really necessary. It seems to me that the only inter-loop dependency is between neighboring processes. If that's the case, then the sends and receives will accomplish all of the necessary synchronization.
If you still need a way to track when all of the ranks are done, you can have each process alert a single rank (say rank 0) when it leaves. When rank 0 detects that everyone is done, it can just exit. Or, if you want to leave after some other number of processes is done, you can have rank 0 send out a message to all other ranks with a special tag like above (but add MPI_ANY_SOURCE so you can receive from rank 0).
I have been thinking about this: Send a "message" to the client and then whenever the server receives it i compare the send date with the receive one and then i'll get the ms out of there.
But my question is, Is there any other efficient way to do this?
As for the basic concept:
Simply send out a signal ( like a ping ) and wait till it returns. The execution time tells you the latency between server and client.
It is, however, something you need to do quite often to get an accurate value. The problem being that network latency is highly variable.
Then again, doing this might not be viable at all, since there is more involved than just sending and receiving the signal. For instance, the actual route a packet might take can vary.
Also, one would think that dividing the execution time by 2 results in an accurate result, but it does not.
Server to client might be faster than the return trip, screwing up the results.
I am using $SUB for the first time and have come across this problem. Both, client and server use overlapped operations and here is the specific situation I have a problem with.
Client
C1. Connects to the server.
C2. Sends the message bigger than a pipe buffer and buffer passed to overlapped read operation in the server.
C3. Successfully cancels the send operation.
Server
S1. Creates and waits for the client.
S2. When the client is connected, it reads the message.
S21. Because message doesn't fit into the buffer(ERROR_MORE_DATA), it is read part by part.
It seems to me that there is no way to tell when is the whole message, as an isolated unit, canceled. In particular, if client cancels the send operation, server does not receive the whole message, just a part of it, and consequent read operation returns with ERROR_IO_PENDING (in my case), which means there is no data to be read and read operation has been queued. I would expect to have some kind of means telling the reader that the message has been canceled, so that reader can act upon it.
However, relevant documentation is scatter over MSDN, so I may as well be missing something. I would really appreciate if anyone can shed some light on it. Thanks.
You are correct, there is no way to tell.
If you cancel the Writefile partway through, only part of the message will be written, so only that part will be read by the server. There is no "bookkeeping" information sent about how large the message was going to be before you cancelled it - what is sent is just the raw data.
So the answer is: Don't cancel the IO, just wait for it to succeed.
If you do need to cancel IO partway through, you should probably cut the connection and start again from the beginning, just as you would for a network outage.
(You could check your OVERLAPPED structure to find out how much was actually written, and carry on from there, but if you wanted to do that you would probably just not cancel the IO in the first place.)
Why did you want to cancel the IO anyway? What set of circumstances triggers this requirement?