Is it possible to create a Source to which I'm going to be able to push data "manually" (or I can do it somehow to a "regular" Source)?
Something like:
var source = Source.Empty<int>();
source.Push(10); //is something like this possible?
My use case would be creating a source to which I'm able to push data whenever my API endpoint is called.
Yes, it's possible. Check out Source.Queue:
Source.Queue can be used for emitting elements to a stream from an actor (or from anything running outside the stream). The elements will be buffered until the stream can process them. You can Offer elements to the queue and they will be emitted to the stream if there is demand from downstream, otherwise they will be buffered until request for demand is received.
Another option is Source.ActorRef:
Messages sent to the actor that is materialized by Source.ActorRef will be emitted to the stream if there is demand from downstream, otherwise they will be buffered until request for demand is received.
Unlike Source.Queue, Source.ActorRef does not support backpressure.
Related
I'm creating an async gRPC server in C++. One of the methods streams data from the server to clients - it's used to send data updates to clients. The frequency of the data updates isn't predictable. They could be nearly continuous or as infrequent as once per hour. The model used in the gRPC example with the "CallData" class and the CREATE/PROCESS/FINISH states doesn't seem like it would work very well for that. I've seen an example that shows how to create a 'polling' loop that sleeps for some time and then wakes up to check for new data, but that doesn't seem very efficient.
Is there another way to do this? If I use the "CallData" method can it block in the 'PROCESS' state until there's data (which probably wouldn't be my first choice)? Or better, can I structure my code so I can notify a gRPC handler when data is available?
Any ideas or examples would be appreciated.
In a server-side streaming example, you probably need more states, because you need to track whether there is currently a write already in progress. I would add two states, one called WRITE_PENDING that is used when a write is in progress, and another called WRITABLE that is used when a new message can be sent immediately. When a new message is produced, if you are in state WRITABLE, you can send immediately and go into state WRITE_PENDING, but if you are in state WRITE_PENDING, then the newly produced message needs to go into a queue to be sent after the current write finishes. When a write finishes, if the queue is non-empty, you can grab the next message from the queue and immediately start a write for it; otherwise, you can just go into state WRITABLE and wait for another message to be produced.
There should be no need to block here, and you probably don't want to do that anyway, because it would tie up a thread that should otherwise be polling the completion queue. If all of your threads wind up blocked that way, you will be blind to new events (such as new calls coming in).
An alternative here would be to use the C++ sync API, which is much easier to use. In that case, you can simply write straight-line blocking code. But the cost is that it creates one thread on the server for each in-progress call, so it may not be feasible, depending on the amount of traffic you're handling.
I hope this information is helpful!
I am working on a project that will require multiple workers to access the same queue to get information about a file which they will manipulate. Files are ranging from size, from mere megabytes to hundreds of gigabytes. For this reason, a visibility timeout doesn't seem to make sense because I cannot be certain how long it will take. I have though of a couple of ways but if there is a better way, please let me know.
The message is deleted from the original queue and put into a
‘waiting’ queue. When the program finished processing the file, it
deletes it, otherwise the message is deleted from the queue and put
back into the original queue.
The message id is checked with a database. If the message id is
found, it is ignored. Otherwise the program starts processing the
message and inserts the message id into the database.
Thanks in advance!
Use the default-provided SQS timeout but take advantage of ChangeMessageVisibility.
You can specify the timeout in several ways:
When the queue is created (default timeout)
When the message is retrieved
By having the worker call back to SQS and extend the timeout
If you are worried that you do not know the appropriate processing time, use a default value that is good for most situations, but don't make it so big that things become unnecessarily delayed.
Then, modify your workers to make a ChangeMessageVisiblity call to SQS periodically to extend the timeout. If a worker dies, the message stops being extended and it will reappear on the queue to be processed by another worker.
See: MessageVisibility documentation
I have trouble with using general_work function for a block which takes a vector as an input and outputs a message.
The block is a kind of demodulator. In fact it is working great if I send some data after and after (periodically).
But I need to create only one data (frame) which has a predefined size and sent it to this block. And I want this block to handle all of the items in its buffer without waiting for more data.
As I understand, it is about the buffering and scheduler structure of GNU Radio, but, I couldn't figure it out how to provide an ability to this block to handle all the symbols of the frame that I've sent without waiting for another frame.
For example, lets say my frame has 150 symbols. The scheduler calls my general_work function two, three, or four times (I don't know how it decides the number of calls for my general_work).
However, it stops lets say at symbol #141, or 143. Every time I run it, it stops at different symbol number. If I send another frame, it completes to handle remaining items (symbols) in its buffer.
Does anybody know how can I tell the scheduler to not wait for another frame to complete the remaining items in its buffer from the previously sent data.
First of all, thank you for your advices. In fact, I am studying on a link layer protocol and its implementation using SDR for my graduate thesis. Because I'm not a DSP expert, I need a wifi phy layer (transceiver). So, I decided to use an OOT module, "802.11 a/g/p Transceiver" project developed by Bastian Bloessl which is available on https://github.com/bastibl/gr-ieee802-11.git. He provided an example flow-graph (wifi_loopback.crc) to simulate the transceiver. By the way, besides the transceiver (DSP stuff) itself, he also developed some part of the data link layer issues for 802.11 such as framing and error control. In the example flow-graph, the "Message Strobe" block is used as a kind of application layer for producing data periodically and send them to a block called "OFDM MAC" which has 4 message ports (app_in, app_out, phy_in, and phy_out). In this block, the raw data which is coming from the "Message Strobe" is encapsulated by adding a header and FCS information. Then, the encapsulated data is sent (phy_out) to a hierarchical block called "Wifi PHY Hier" in order to do some DSP issues such as scrambling, coding, interleaving, symbol mapping and modulation etc. In some way, the data is converted to signal and received by the same block ("Wifi PHY Hier") and the opposite process is handled such as descrambling, decoding etc. And it gives the decoded frame to "OFDM MAC" block (phy_in). If you run this flow-graph, everything is normal. I mean, the data sent by "Message Strobe" is received correctly.
However, because I am trying to implement a kind of link layer protocol, I need some feedback from destination to source such as an ACK message. So, I decided to start by implementing a simple stop&wait protocol that the source sends a message and wait for an ACK from the destination, DATA -> ACK -> DATA -> ACK... and so on. In order to do that, I create a simple source block which sends only one data and wait for an ACK message to send another data. The data I produce with my source block is the same as the data produced by "Message Strobe". When I replace the "Message Strobe" block with my source block, I realized that something is wrong because I couldn't receive my data. So, I've followed my data in order to find which step cause this situation. There is no problem with the transmission process. In the receive process, I found the problematic block which is in the "Wifi PHY Hier" block and is the last block before this hierarchical block gives its data to "OFDM MAC" block. This problematic block which is called "OFDM Decode MAC" has two ports. The output port is a message port and the input port is complex vector. So, I reviewed the code of this block, specially, the general_work() function of it. For my particular test data, in order to complete its job correctly, it should consume 177 items to produce an output to "OFDM MAC". However, it stops consuming items after 172 items are consumed. I override the forecast() method and set ninput_items_required[0] = 177. But in this case, nothing is happened, because, as I understand, the scheduler has never see 177 items in the input buffer. As you said, this is because the block ("OFDM Decode Signal") that writes into this block's input buffer produce 172 items.
I did not go deep further yet but the interesting point is when I send a second data (in the runtime) after a period, without waiting for an ACK, the remaining 5 items of the first data I've sent are consumed in some way and received correctly by the "OFDM MAC" block. And now the second data is in the same problematic situation that the previus data has experienced.. If I send third data, the second one is also received correctly. I'm really confused. How can this be ?
I'll comment quickly on your text, and then advise below:
I have trouble with using general_work function for a block which
takes a vector as an input and outputs a message.
That block is, from a sample stream perspective, a sink. You will find that when using sink as a block type in gr_modtool, you will get a sync_block, which means you will only have to implement a work, not a general_work, and a forecast.
The block is a kind of demodulator. In fact it is working great if I
send some data after and after (periodically).
So that's great!
But I need to create only one data (frame) which has a predefined size
and sent it to this block. And I want this block to handle all of the
items in its buffer without waiting for more data.
That sounds like your block doesn't actually take streams of samples, but blocks. That is either a job for
message passing (so your block would have no input stream, just a message port) or
tagged stream blocks.
Sounds like the second to me.
As I understand, it is about the buffering and scheduler structure of
GNU Radio, but, I couldn't figure it out how to provide an ability to
this block to handle all the symbols of the frame that I've sent
without waiting for another frame.
Frame is what you make of this – to GNU Radio, your samples are just items that get written to and read from a buffer.
For example, lets say my frame has 150 symbols. The scheduler calls my
general_work function two, three, or four times (I don't know how it
decides the number of calls for my general_work).
It doesn't decide -- that's probably the chunks in which the symbols get written into the input buffer of your block. You don't have to consume all of these (or any of these) if your block isn't able to produce output with the input given. Just let GNU Radio know how many items were consumed (in the sync block case, it's implicitly done with the return value; in the general_work case, you might have to manually call consume – another reason to change your block type!).
However, it stops lets say at symbol #141, or 143. Every time I run
it, it stops at different symbol number. If I send another frame, it
completes to handle remaining items (symbols) in its buffer.
That sounds like a bug in your algorithm, not in GNU Radio. Maybe your input buffer is simply full, or maybe the block that writes into it simply doesn't provide more data?
Does anybody know how can I tell the scheduler to not wait for
another frame to complete the remaining items in its buffer from the
previously sent data.
The scheduler doesn't wait; as soon as there is data to be processed, it instantly "wakes" your block, and asks it to process the items.
I've reached Bastian, the guy who developed this OOT module. He said that the reason of the problem was a kind of padding issue. If a block called "Packet Padding2", which can be found in another OOT module that also developed by him, is used after "Wifi PHY Hier" and set the Pad Tail parameter of this block to appropriate value, the problem is solved.
I'm trying to spread data across multiple workers using OpenMPI, however, I'm doing the data division in a fairly custom way that is not amenable to MPI_Scatter or MPI_Broadcast. What I would like to do is to give each processor some work in a queue (or, some other async mechanism) such that they can do their work on the first chunk of data, take the next chunk, repeat until no more chunks.
I know of MPI_Isend, however if I send data with MPI_Isend I can't modify it until it's finished sending; forcing me to use MPI_Wait and thus having to wait until the thread is finished receiving the data anyway!
Is there a standard a solution to this problem, or must I rethink my approach?
Using MPI_ISEND doesn't necessarily mean that the message is received on the remote end. It just means that the buffer is available for reuse. It could be that the message has been buffered internally by Open MPI or that the message actually has been received on the other end. It depends on your message size.
Another option would be to have your workers ask the master process for work when they need it instead of having it pushed to them. Then the master can work only as needed. You could do an MPI_SCATTER for the first message since everyone will be receiving some data. Then after that, have the master do an MPI_RECV(MPI_ANY_SOURCE) to get a message from one of the worker processes.
I'm searching for different options for implementing communication between a service and other services/applications.
What I would like to do:
I have a service that is constantly running, polling a device connected to a serial port. At certain points, this service should send a message to interested clients containing data retrieved from the device. Data is uncomplicated, most likely just a single string.
Ideally, the clients would not have to subscribe to receive these messages, which leads me to some sort of event 'broadcast' setup (similar to Windows events). The message sending process should not block, and does not need a response from any clients (or that there even are any clients for that matter).
I've been reading about IPC (COM in particular) and windows events, but am yet to come across something that really fits with what I want to do.
So is this possible? If so, what technologies should I be using? If not, what are some viable communication alternatives?
Here's the particulars of the setup:
Windows 2000/XP environments
'Server' service is a windows service, using VC++2005
Clients would vary, but always be in the windows environment (usual clients would be VC++6 windows services, VB6 applications)
Any help would be appreciated!
Windows supports broadcasting messages, check here. You can SendMessage to HWND_BROADCAST from the service, and receive it in each client.
There are a number of ways to do a broadcast system, but you'll have to either give up reliability (ie, some messages must be lost) or use a proper subscription system.
If you're willing to give up reliability, you can create a shared memory segment and named manual-reset event object. When a new message arrives, write it to the shared memory segment, signal the event object, then close the event object and create a new one with a different name (the name should be in the shmem segment somewhere). Clients open the shmem segment, find the current event object, wait for it to be signaled, then read off the message and new event segment.
In this option, you must be careful to deal with the case of a client reading at the same time as the shmem segment is updated properly. One way to do this is to have two sequence number fields in the shmem segment - one is updated before the new message is written, one after. Clients read the second sequence number prior to reading the message, then re-read both sequence numbers after, and check that they are all equal (and discard the message and retry after a delay if they are not). Be sure to place memory barriers around accesses to these sequence numbers to ensure the compiler does not reorder them!
Of course, this is all a bit hairy. Named pipes are a lot simpler, but a subscription (of a sort) is required. The server calls CreateNamedPipe, then accepts connections with ConnectNamedPipe. Clients use CreateFile to connect to the server's pipe. The server then just loops to send data (using WriteFile) to all of its clients. Note that you will need to create addititonal instance of the pipe using CreateNamedPipe each time you accept a connection. An example of a named pipe server can be found here: http://msdn.microsoft.com/en-us/library/aa365588(v=vs.85).aspx