Using the MailboxProcessor in F#, what is the preferred way to communicate between them? - Wrapping the agents into objects like:
type ProcessAgent(saveAgent:SaveAgent) = ...
type SaveAgent() = ...
let saveAgent = new SaveAgent()
let processAgent = new ProcessAgent(mySaveAgent)
or what about:
type ProcessAgent(cont:string -> unit) = ...
type SaveAgent() = ...
let saveAgent = new SaveAgent()
let processAgent = new ProcessAgent(fun z -> saveAgent.Add z)
or maybe even something like:
type ProcessAgent() = ...
type SaveAgent() = ...
let saveAgent = new SaveAgent()
let processAgent = new ProcessAgent()
processAgent.Process item (fun z -> saveAgent.Add z)
Also is there ever any reason to wrap a normal function, that's is not maintaining some kind of state, into an agent?
The key thing about encapsulating the agents in classes is that it lets you break the direct dependencies between them. So, you can create the individual agents and then connect them into a bigger "agent network" just by registering event handlers, calling methods, etc.
An agent can essentially expose three kinds of members:
Actions are members of type 'T -> unit. They send some message to the agent without waiting for any reply from the agent. This is essentially wrapping a call to agent.Post.
Blocking actions are members of type 'T -> Async<'R>. This is useful when you're sending some message to the agent, but then want to wait for a response (or confirmation that the action was processed). These do not block the logical thread (they are asynchronous) but they block the execution of the caller. This is essentially wrapping a call to agent.PostAndAsyncReply.
Notifications are members of type IEvent<'T> or IObservable<'T> representing some sort of notification reported from the agent - e.g. when the agent finishes doing some work and wants to notify the caller (or other agents).
In your example, the processing agent is doing some work (asynchronously) and then returns the result, so I think it makes sense to use "Blocking action". The operation of the saving agent is just an "Action", because it does not return anything. To demonstrate the last case, I'll add "flushed" notification, which gets called when the saving agent saves all queued items to the actual storage:
// Two-way communication processing a string
type ProcessMessage =
PM of string * AsyncReplyChannel<string>
type ProcessAgent() =
let agent = MailboxProcessor.Start(fun inbox -> async {
while true do
let! (PM(s, repl)) = inbox.Receive()
repl.Reply("Echo: " + s) })
// Takes input, processes it and asynchronously returns the result
member x.Process(input) =
agent.PostAndAsyncReply(fun ch -> PM(input, ch))
type SaveAgent() =
let flushed = Event<_>()
let agent = (* ... *)
// Action to be called to save a new processed value
member x.Add(res) =
agent.Post(res)
// Notification triggered when the cache is flushed
member x.Flushed = flushed.Publish
Then you can create both agents and connect them in various ways using the members:
let proc = ProcessAgent()
let save = SaveAgent()
// Process an item and then save the result
async {
let! r = proc.Process("Hi")
save.Save(r) }
// Listen to flushed notifications
save.Flushed |> Event.add (fun () ->
printfn "flushed..." )
You don't need to create a class for your agents. Why not just write a function that returns your MailboxProcessor?
let MakeSaveAgent() =
MailboxProcessor<SaveMessageType>.Start(fun inbox ->
(* etc... *))
let MakeProcessAgent (saveAgent: MailboxProcessor<SaveMessageType>) =
MailboxProcessor<ProcessMessageType>.Start(fun inbox ->
(* etc... you can send messages to saveAgent here *))
For your final question: no, not really, that would be adding unnecessary complication when a simple function returning Async<_> would suffice.
Related
I have a system using Akka which currently handles incoming streaming data over message queues. When a record arrives then it is processed, mq is acked and record is passed on for further handling within the system.
Now I would like to add support for using DBs as input.
What would be a way to go for the input source to be able to handle DB (should stream in > 100M records at the pace that the receiver can handle - so I presume reactive/akka-streams?)?
Slick Library
Slick streaming is how this is usually done.
Extending the slick documentation a bit to include akka streams:
//SELECT Name from Coffees
val q = for (c <- coffees) yield c.name
val action = q.result
type Name = String
val databasePublisher : DatabasePublisher[Name] = db stream action
import akka.stream.scaladsl.Source
val akkaSourceFromSlick : Source[Name, _] = Source fromPublisher databasePublisher
Now akkaSourceFromSlick is like any other akka stream Source.
"Old School" ResultSet
It is also possible to use a plain ResultSet, without slick, as the "engine" for an akka stream. We will utilize the fact that a stream Source can be instantiated from an Iterator.
First create the ResultSet using standard jdbc techniques:
import java.sql._
val resultSetGenerator : () => Try[ResultSet] = Try {
val statement : Statement = ???
statement executeQuery "SELECT Name from Coffees"
}
Of course all ResultSet instances have to move the cursor before the first row:
val adjustResultSetBeforeFirst : (ResultSet) => Try[ResultSet] =
(resultSet) => Try(resultSet.beforeFirst()) map (_ => resultSet)
Once we start iterating through rows we'll have to pull the value from the correct column:
val getNameFromResultSet : ResultSet => Name = _ getString "Name"
And now we can implement the Iterator Interface to create a Iterator[Name] from a ResultSet:
val convertResultSetToNameIterator : ResultSet => Iterator[Name] =
(resultSet) => new Iterator[Try[Name]] {
override def hasNext : Boolean = resultSet.next
override def next() : Try[Name] = Try(getNameFromResultSet(resultSet))
} flatMap (_.toOption)
And finally, glue all the pieces together to create the function we'll need to pass to Source.fromIterator:
val resultSetGenToNameIterator : (() => Try[ResultSet]) => () => Iterator[Name] =
(_ : () => Try[ResultSet])
.andThen(_ flatMap adjustResultSetBeforeFirst)
.andThen(_ map convertResultSetToNameIterator)
.andThen(_ getOrElse Iterator.empty)
This Iterator can now feed a Source:
val akkaSourceFromResultSet : Source[Name, _] =
Source fromIterator resultSetGenToNameIterator(resultSetGenerator)
This implementation is reactive all the way down to the database. Since the ResultSet pre-fetches a limited number of rows at a time, data will only come off the hard drive through the database as the stream Sink signals demand.
I find Alpakka documentation to be excellent and a much easier way to work with reactive-streams than than the Java Publisher interface.
The Alpakka project is an open source initiative to implement stream-aware, reactive, integration pipelines for Java and Scala. It is built on top of Akka Streams, and has been designed from the ground up to understand streaming natively and provide a DSL for reactive and stream-oriented programming, with built-in support for backpressure
Document for Alpakka with Slick: https://doc.akka.io/docs/alpakka/current/slick.html
Alpakka Github: https://github.com/akka/alpakka
In my system I want an actor A to send the same messages to actors B,C,D.
Instead of creating the three actors, I was thinking of just combining their behaviors with an And behavior contaminator, and then passing that behavior to A.
If I do this, how many actors will get created? Will I get just one actor with three behaviors in it, or three actors with separate behaviors?
Here is my real code using the non-And approach, for concreteness (see how ReplyGenerator gets passed the references to other actors):
object Foobar {
def foobar(): Behavior[Request] =
ContextAware[Request] {
context =>
val foo1 = context.spawn(Props(Foo1.behavior()), "foo1")
val foo2 = context.spawn(Props(Foo2.behavior()), "foo2")
val foo3 = context.spawn(Props(Foo3.behavior()), "foo3")
val generator = context.spawn(Props(ReplyGenerator.behavior(List(foo1, foo2, foo3))),
"generator")
Static {
case request: Request =>
generator ! request
}
}
}
and here is the ReplyGenerator behavior that sends the same message to all subscribers:
object ReplyGenerator {
def behavior(subscribers: List[ActorRef[Reply]]): Behavior[Request] = {
Static {
case request: Request =>
subscribers.foreach(_ ! Reply.empty)
}
}
Considering that I want the actors foo 1,2,3 to run in parallel, can And combinator be used here instead?
Thank you.
If you mean execution parallelism then you’ll have to create separate Actors (by separate spawn calls) as you do in the example code—using And will only create a single Actor that runs the contained behaviors one after the other.
I want to create a service who generates its HTML according to the parameter given and a map. Given the parameter, the service search in the map for the html, and a function to launch on client side.
type sample =
(string (* little text *)*
Html5_types.html Eliom_content.Html5.elt (* html page *) *
(unit -> unit)(* Demonstration function *))
Given that the function is to be launched on client side, I insert it in the map as a client value :
{client{
let demo_function = ignore (Ojquery.add_html
(Ojquery.jQ "li") "<p id='test1'>new paragraph</p>") }}
let get_samples () =
let samples_map = Samples.empty in
let samples_map = Samples.add "add_html"
("text",
(Eliom_tools.F.html
(** html stuff **)
),
{unit->unit{demo_function}}) samples_map in
samples_map
And then I register the service like this :
let sample_service =
Eliom_service.service
~path:["examples"]
~get_params:Eliom_parameter.(string "entry")
()
let () =
Examples_app.register
~service:sample_service
(fun (entry) () ->
try
(let entry = Samples.find entry samples_map in
let html = ((function (name, html, func) -> html) entry) in
let func = ((function (name, html, func) -> func) entry) in
ignore {unit{%func ()}};
Lwt.return (html))
with Not_found -> Lwt.return (not_found)
)
The rest of the code is pretty much only the result of a classic eliom-distillery, with the inclusion of the ojquery package for the client function used.
The compilation phase goes smoothly, but when I try to launch the server, I get the following error message :
ocsigenserver: main: Fatal - Error in configuration file: Error while parsing configuration file: Eliom: while loading local/lib/examples/examples.cma: Failure("That function cannot be called here because it needs information about the request or the site.")
My first guess was that it is due to the fact that I store client values outside of a service, but is there any way to store this kind of values on the server?
I tried to wrap them in regular functions :
let demo_serv_func () = {unit{demo_client_func ()}}
But the problem remained...
I found the issue. The problem was not because I stored client functions, but because I used Eliom_tools.F.html outside of a service.
It happens that Eliom_tools needs the context of the service to function, and since I was storing it outside of the service, it could not work.
I solved the issue by using Eliom_tools inside the service, and storing the body of the HTML page in the map.
I have a mailbox processor which receives a fixed number of messages:
let consumeThreeMessages = MailboxProcessor.Start(fun inbox ->
async {
let! msg1 = inbox.Receive()
printfn "msg1: %s" msg1
let! msg2 = inbox.Receive()
printfn "msg2: %s" msg2
let! msg3 = inbox.Receive()
printfn "msg3: %s" msg3
}
)
consumeThreeMessages.Post("First message")
consumeThreeMessages.Post("Second message")
consumeThreeMessages.Post("Third message")
These messages should be handled in exactly the order sent. During my testing, it prints out exactly what it should:
First message
Second message
Third message
However, since message posting is asynchronous, it sounds like posting 3 messages rapidly could result in items being processed in any order. For example, I do not want to receive messages out of order and get something like this:
Second message // <-- oh noes!
First message
Third message
Are messages guaranteed to be received and processed in the order sent? Or is it possible for messages to be received or processed out of order?
The code in your consumeThreeMessages function will always execute in order, because of the way F#'s async workflows work.
The following code:
async {
let! msg1 = inbox.Receive()
printfn "msg1: %s" msg1
let! msg2 = inbox.Receive()
printfn "msg2: %s" msg2
}
Roughly translates to:
async.Bind(
inbox.Receive(),
(fun msg1 ->
printfn "msg1: %s" msg1
async.Bind(
inbox.Receive(),
(fun msg2 -> printfn "msg2: %s" msg2)
)
)
)
When you look at the desugared form, it is clear that the code executes in serial. The 'async' part comes into play in the implementation of async.Bind, which will start the computation asynchronously and 'wake up' when it completes to finish the execution. This way you can take advantage of asynchronous hardware operations, and not waste time on OS threads waiting for IO operations.
That doesn't mean that you can't run into concurrency issues when using F#'s async workflows however. Imagine that you did the following:
let total = ref 0
let doTaskAsync() =
async {
for i = 0 to 1000 do
incr total
} |> Async.Start()
// Start the task twice
doTaskAsync()
doTaskAsync()
The above code will have two asynchronous workflows modifying the same state at the same time.
So, to answer your question in brief: within the body of a single async block things will always execute in order. (That is, the next line after a let! or do! doesn't execute until the async operation completes.) However, if you share state between two async tasks, then all bets are off. In that case you will need to consider locking or using Concurrent Data Structures that come with CLR 4.0.
Anyone know of a good way to unit test Scala actors? In the general sense I have an actor that receives a message and will send out other messages in response. This is done on multiple threads, and an actor that is not correct may either send the wrong messages or no message at all. I need a simple way of creating a mockup actor that send and receives messages to the actor being tested. Any experiences in this area?
Because of the dynamic nature of actor-style message passing, mocking actors is usually no trouble at all. Just create an actor which receives the desired message and you're home free. You will of course need to ensure that this mock actor is the one to which messages are passed, but that shouldn't be a problem as long as the actor you are attempting to test is reentrant.
I think the complexity depends on a couple factors...
How stateful is the actor?
If it behaves like a idempotent function, only asynchronous, then it should be a simple matter of mocking up an actor that sends a message and then checks that it receives the expected messages back. You probably want to use a react/receiveWithin on the mock actor in case there is response within a reasonable period of time you can fail rather than hanging.
However if the messages aren't independent of one another, then you should test it with various sequences of messages and expected results.
How many actors will the actor being tested interact with?
If an actor is expected to interact with many others, and it is stateful, then it should be tested with several actors sending/receiving messages. Since you probably have no guarantee of the order in which the messages will arrive, you should be sure to either permute the orders in which the actors send the messages or introduce random pauses in the actors generating messages and run the test many times.
I'm not aware of any prebuilt frameworks for testing actors, but you could possibly look to Erlang for inspiration.
http://svn.process-one.net/contribs/trunk/eunit/doc/overview-summary.html
I have been wondering about how to test Actors myself.
Here is what I came up with, does anybody see problems with this approach?
Rather than send messages directly, what if your actor delegated message sending to a function?
Then your tests can swap out the function with one that tracks the number of times called and/or the arguments with which the method was called:
class MyActor extends Actor {
var sendMessage:(Actor, ContactMsg) => Unit = {
(contactActor, msg) => {
Log.trace("real sendMessage called")
contactActor ! msg
}
}
var reactImpl:PartialFunction(Any, Unit) = {
case INCOMING(otherActor1, otherActor2, args) => {
/* logic to test */
if(args){
sendMessage(otherActor1, OUTGOING_1("foo"))
} else {
sendMessage(otherActor2, OUTGOING_2("bar"))
}
}
}
final def act = loop {
react {
reactImpl
}
}
Your test case might contain code like:
// setup the test
var myActor = new MyActor
var target1 = new MyActor
var target2 = new MyActor
var sendMessageCalls:List[(Actor, String)] = Nil
/*
* Create a fake implementation of sendMessage
* that tracks the arguments it was called with
* in the sendMessageCalls list:
*/
myActor.sendMessage = (actor, message) => {
Log.trace("fake sendMessage called")
message match {
case OUTGOING_1(payload) => {
sendMessageCalls = (actor, payload) :: sendMessageCalls
}
case _ => { fail("Unexpected Message sent:"+message) }
}
}
// run the test
myActor.start
myActor.reactImpl(Incoming(target1, target2, true))
// assert the results
assertEquals(1, sendMessageCalls.size)
val(sentActor, sentPayload) = sendMessageCalls(0)
assertSame(target1, sentActor)
assertEquals("foo", sentPayload)
// .. etc.
My attempt at unit testing an actor (it works). I'm using Specs as a framework.
object ControllerSpec extends Specification {
"ChatController" should{
"add a listener and respond SendFriends" in{
var res = false
val a = actor{}
val mos = {ChatController !? AddListener(a)}
mos match{
case SendFriends => res = true
case _ => res = false
}
res must beTrue
}
How this works is by sending a synchronous call to the singleton ChatController. ChatController responds by use of reply(). The response is sent as a return of the called function, which gets stored into mos. Then a match is applied to mos getting the case class that was sent from ChatController. If the result is what is expected (SendFriends) set res to true. The res must beTrue assertion determines the success or failure of test.
My actor singleton that I'm testing
import ldc.socialirc.model._
import scala.collection.mutable.{HashMap, HashSet}
import scala.actors.Actor
import scala.actors.Actor._
import net.liftweb.util.Helpers._
//Message types
case class AddListener(listener: Actor)
case class RemoveListener(listener: Actor)
case class SendFriends
//Data Types
case class Authority(usr: Actor, role: String)
case class Channel(channelName: String, password: String, creator: String, motd: String, users: HashSet[Authority])
object ChatController extends Actor {
// The Channel List - Shows what actors are in each Chan
val chanList = new HashMap[String, Channel]
// The Actor List - Shows what channels its in
val actorList = new HashMap[Actor, HashSet[String]]
def notifyListeners = {
}
def act = {
loop {
react {
case AddListener(listener: Actor)=>
actorList += listener -> new HashSet[String]
reply(SendFriends)
}
}
}
start //Dont forget to start
}
Though its not complete it does return the Sendfriends case class as expected.
Suite for unit testing of Actors has recently been added to Akka. You can find some information and code snippets in this blogpost.