I know that Erlang uses Actor model to support concurrency and Erlang processes are the same as Actors: they send messages, immutable and so on. But according to Carl Hewitt one important thing in the Actor Model is indeterminism and Arbiters (Given an arbiter, you can have multiple inputs (e.g. I0 and I1) into the arbiter at the same time, but only one of the possible outcomes (e.g. O0 or O1) will come out on the other end).
So, I'm interesting how does Erlang implemented this concept? And what is used in the role of Arbiters in the Erlang concurrency model/actor model implementation?
This gets pretty deeply philosophical (see e.g. https://en.wikipedia.org/wiki/Indeterminacy_in_concurrent_computation), but as far as I can tell, he's saying that in the Actor Model, whenever an actor has multiple inputs, there's a magic box that decides the ordering of the incoming messages any way it wants to, even if it means delaying some of the messages for an arbitrarily long (but bounded) time. I.e., you can never rely on any particular order or time for receiving parallel messages, even if the program structure seems to favour a certain arrival order. (Note that this is a theoretical concept for reasoning about actor programs - you wouldn't try to make a system unnecessarily random in practice, except for testing purposes.)
The semantics of Erlang message passing say pretty much the same thing: whenever two processes send a message each to a third process, and there is no ordering constraint on the individual send events, you can never rely on which message will end up first in the receiver's mailbox. They could be arbitrarily delayed, even if all processes run within the same Erlang VM. Again, this is about what guarantees you get as a programmer (none), not about making the Erlang VM insert random delays. (Random delays can be introduced naturally by other things, such as OS-level pauses for page faults.)
When working with Scala Akka, we still think about blocking IO, and try to avoid it with patterns, I feel it makes no difference from using threads. Implicitly async IO is a huge differentiator, Erlang and Go provide it at language level, while Scala Akka does not. I feel Scala Akka is not real actor model.
There's a popular blog post Don't use Actors for concurrency, but it's not really a problem of actor model, it's purely a problem of Akka.
Akka implements the Actor Model as specified by Carl Hewitt et al in 1973: an actor can upon reception of a message
send a finite number of messages to actors it knows
create a finite number of actors
determine the behavior to be applied to the next message
Nowhere does this say anything about how exactly I/O is supposed to be handled. Translating blocking method calls into actor-suspending method calls automatically is on the other hand a violation of the model: actors only act upon messages, nothing else, and declaring some arbitrary method calls to prevent that from happening for some (possibly unbounded) time is not part of the model.
Akka provides IO facilities for network operations that are presented in an Actor Model fashion by exposing an ActorRef that commands can be sent to and results can be received from. Akka Streams includes asynchronous consumption and production of data from and to I/O channels.
I hope this explains why my overall answer is that the premise of the question is flawed.
The premise of your question is inaccurate
("...Scala Akka does not."). From the akka documentation:
Actors give you:
Simple and high-level abstractions for concurrency and parallelism.
Asynchronous, non-blocking and highly performant event-driven
programming model. Very lightweight event-driven processes (several
million actors per GB of heap memory).
A basic example of asynchronous message passing to an Actor:
val actorSystem = ActorSystem()
val actorRef = actorSystem actorOf Props[SomeActor]
val message = ??? //some message value
actorRef ! message //asynchronous message passing
In fact, akka exceeds the original Actor Model specification with the ability to lookup Actors by name.
For IO specifically, akka offers many non-blocking solutions:
TCP
HTTP
Streaming File IO
Granted, Scala the language does not have an embedded Actor Model. But it provides Futures and parallel collections for concurrent computation.
The gist of the blog you quoted was that Futures are much better for concurrent computation (which I agree with), while Actors are for maintaining state. It wasn't claiming that Actors are incapable of concurrency.
Plus, akka is not just Actors. There are Agents, finite state machines, akka streams (implementing the reactive manifesto), and the akka-http library.
Assume that I have 2 actors, A and B, A ask B for a response within a 10 seconds timeout. After that B still busy processing other messages which cannot response for A within 10 seconds, so Future in actor A get timeout exception.
The questions are:
1) After actor A got exception, does the message that A send to B still be inside B's mailbox waiting for B to process it?
2) If yes, how to prevent/detect B form overwhelm of messages?
The answer to this is that yes, the message is still in B's box. This is quite a common misconception about Futures in many contexts like the ask pattern, where you have some request going on represented by a future that times out after a limited amount of time; most people would assume that this would mean that when the process being awaited would be cancelled but there is no way that a generic abstraction like a Future could do it (also remember that Future ≠ Thread).
As for the second question the answer is that there is no easy answer and you need to think about a strategy that fits your environment. If all your actors reside in the same VM you could embed timestamps in your messages to validate them on the target actor; this would still be racy as the actor B could take enough time to process the message to trigger the timeout on the A side, generating inconsistency (i.e. A thinks that the operation failed, B things that it succeeded).
Otherwise, in a distributed environment, an idea could be to send a compensating action to the actor B when the actor A detects a timeout. You may want to have a look at the Saga Pattern (you can find a nice talk here) if you are doing such a distributed system based on actors and you are having troubles with this sort of situations. In any case, just remember that distributed systems are hard and there is no silver bullet or any way around studying hard to get them right.
I am trying to setup a TCP communication framework between two computers. I would like each computer to send data to the other. So computer A would perform a calculation, and send it to computer B. Computer B would then read this data, perform a calculation using it, and send a result back to computer A. Computer A would wait until it receives something from computer B before proceeding with performing another calculation, and sending it to computer B.
This seems conceptually straightforward, but I haven't been able to locate an example that details two-way (bidirectional) communication via TCP. I've only found one-way server-client communication, where a server sends data to a client. These are some examples that I have looked at closely so far:
Server-Client communication
Synchronized server-client communication
I'm basically looking to have two "servers" communicate with each other. The synchronized approach above is, I believe, important for what I'm trying to do. But I'm struggling to setup a two-way communication framework via a single socket.
I would appreciate it greatly if someone could point me to examples that describe how to setup bidirectional communication with TCP, or give me some pointers on how to set this up, from the examples I have linked above. I am very new to TCP and network communication frameworks and there might be a lot that I could be misunderstanding, so it would be great if I could get some clear pointers on how to proceed.
This answer does not go into specifics, but it should give you a general idea, since that's what you really seem to be asking for. I've never used Qt before, I do all my networking code with BSD-style sockets directly or with my own wrappers.
Stuff to think about:
Protocol. Hand-rolled or existing?
Existing protocols can be heavyweight, depending on what your payload looks like. Examples include HTTP and Google ProtoBuf; there are many more.
Handrolled might mean more work, but more controlled. There are two general approaches: length-based and sentinel-based.
Length-based means embedding the length into the first bytes. Requires caring about endianness. Requires thinking about what if a message is longer than can be embedded in the length byte. If you do this, I strongly recommend that you define your packet formats in some data file, and then generate the low-level packet encoding logic.
Sentinel-based means ending the message when some character (or sequence) is seen. Common sentinels are '\0', '\n', and "\r\n". If the rest of your protocol is also text-based, this means it is much easier to debug.
For both designs, you have to think about what happens if the other side tries to send more data than you are willing (or able) to store in memory. In either case, limiting the payload size to a 16-bit unsigned integer is probably a good idea; you can stream replies with multiple packets. Note that serious protocols (based on UDP + crypto) typically have a protocol-layer size limit of 512-1500 bytes, though application-layer may be larger of course.
For both designs, EOF on the socket without having a sentinel means you must drop the message and log an error.
Main loop. Qt probably has one you can use, but I don't know about it.
It's possible to develop simple operations using solely blocking operations, but I don't recommend it. Always assume the other end of a network connection is a dangerous psychopath who knows where you live.
There are two fundamental operations in a main loop:
Socket events: a socket reports being ready for read, or ready to write. There are also other sorts of events that you probably won't use, since most useful information can be found separately in the read/write handlers: exceptional/priority, (write)hangup, read-hangup, error.
Timer events: when a certain time delta has passed, interrupt the wait-for-socket-events syscall and dispatch to the timer heap. If you don't have any, either pass the syscalls notion of "infinity". But if you have long sleeps, you might want some arbitrary, relatively number like "10 seconds" or "10 minutes" depending on your application, because long timer intervals can do all sorts of weird things with clock changes, hibernation, and such. It's possible to avoid those if you're careful enough and use the right APIs, but most people don't.
Choice of multiplex syscall:
The p versions below include atomic signal mask changing. I don't recommend using them; instead if you need signals either add signalfd to the set or else emulate it using signal handlers and a (nonblocking, be careful!) pipe.
select/pselect is the classic, available everywhere. Cannot have more than FD_SETSIZE file descriptors, which may be very small (but can be #defined on the command-line if you're careful enough. Inefficient with sparse sets. Timeout is microseconds for select and nanonseconds for pselect, but chances are you can't actually get that. Only use this if you have no other choice.
poll/ppoll solves the problems of sparse sets, and more significantly the problem of listening to more than FD_SETSIZE file descriptors. It does use more memory, but it is simpler to use. poll is POSIX, ppoll is GNU-specific. For both, the API provides nanosecond granularity for the timeout, but you probably can't get that. I recommend this if you need BSD compatibility and don't need massive scalability, or if you only have one socket and don't want to deal with epoll's headaches.
epoll solves the problem of having to respecify the file descriptor and event list every time. by keeping the list of file descriptors. Among other things, this means that when, the low-level kernel event occurs, the epoll can immediately be made aware, regardless of whether the user program is already in a syscall or not. Supports edge-triggered mode, but don't use it unless you're sure you understand it. Its API only provides millisecond granularity for the timeout, but that's probably all you can rely on anyway. If you are able to only target Linux, I strongly suggest you use this, except possibly if you can guarantee only a single socket at once, in which case poll is simpler.
kqueue is found on BSD-derived systems, including Mac OS X. It is supposed to solve the same problems as epoll, but instead of keeping things simple by using file descriptors, it has all sorts of strange structures and does not follow the "do only one thing" principle. I have never used it. Use this if you need massive scalability on BSD.
IOCP. This only exists on Windows and some obscure Unixen. I have never used it and it has significantly different semantics. Use this, but be aware that much of this post is not applicable because Windows is weird. But why would you use Windows for any sort of serious system?
io_uring. A new API in Linux 5.1. Significantly reducing the number of syscalls and memory copies. Worth it if you have a lot of sockets, but since it's so new, you must provide a fallback path.
Handler implementation:
When the multiplex syscall signifies an event, look up the handler for that file number (some class with virtual functions) and call the relevant events (note there may be more than one).
Make sure all your sockets have O_NONBLOCK set and also disable Nagle's algorithm (since you're doing buffering yourself), except possibly connect's before the connection is made, since that requires confusing logic, especially if you want to play nice with multiple DNS results.
For TCP sockets, all you need is accept in the listening socket's handler, and read/write family in the accept/connected handler. For other sorts of sockets, you need the send/recv family. See the "see also" in their man pages for more info - chances are one of them will be useful to you sometimes, do this before you hard-code too much into your API design.
You need to think hard about buffering. Buffering reads means you need to be able to check the header of a packet to see if there are enough bytes to do anything with it, or if you have to store the bytes until next time. Also remember that you might receive more than one packet at once (I suggest you rethink your design so that you don't mandate blocking until you get the reply before sending the next packet). Buffering writes is harder than you think, since you don't want to be woken when there is a "can write" even on a socket for which you have no data to write. The application should never write itself, only queue a write. Though TCP_CORK might imply a different design, I haven't used it.
Do not provide a network-level public API of iterating over all sockets. If needed, implement this at a higher level; remember that you may have all sorts of internal file descriptors with special purposes.
All of the above applies to both the server and the client. As others have said, there is no real difference once the connection is set up.
Edit 2019:
The documentation of D-Bus and 0MQ are worth reading, whether you use them or not. In particular, it's worth thinking about 3 kinds of conversations:
request/reply: a "client" makes a request and the "server" does one of 3 things: 1. replies meaningfully, 2. replies that it doesn't understand the request, 3. fails to reply (either due to a disconnect, or due to a buggy/hostile server). Don't let un-acknowledged requests DoS the "client"! This can be difficult, but this is a very common workflow.
publish/subscribe: a "client" tells the "server" that it is interested in certain events. Every time the event happens, the "server" publishes a message to all registered "clients". Variations: , subscription expires after one use. This workflow has simpler failure modes than request/reply, but consider: 1. the server publishes an event that the client didn't ask for (either because it didn't know, or because it doesn't want it yet, or because it was supposed to be a oneshot, or because the client sent an unsubscribe but the server didn't process it yet), 2. this might be a magnification attack (though that is also possible for request/reply, consider requiring requests to be padded), 3. the client might have disconnected, so the server must take care to unsubscribe them, 4. (especially if using UDP) the client might not have received an earlier notification. Note that it might be perfectly legal for a single client to subscribe multiple times; if there isn't naturally discriminating data you may need to keep a cookie to unsubscribe.
distribute/collect: a "master" distributes work to multiple "slaves", then collects the results, aka map/reduce any many other reinvented terms for the same thing. This is similar to a combination of the above (a client subscribes to work-available events, then the server makes a unique request to each clients instead of a normal notification). Note the following additional cases: 1. some slaves are very slow, while others are idle because they've already completed their tasks and the master might have to store the incomplete combined output, 2. some slaves might return a wrong answer, 3. there might not be any slaves, 4.
D-Bus in particular makes a lot of decisions that seem quite strange at first, but do have justifications (which may or may not be relevant, depending on the use case). Normally, it is only used locally.
0MQ is lower-level and most of its "downsides" are solved by building on top of it. Beware of the MxN problem; you might want to artificially create a broker node just for messages that are prone to it.
#include <QAbstractSocket>
#include <QtNetwork>
#include <QTcpServer>
#include <QTcpSocket>
QTcpSocket* m_pTcpSocket;
Connect to host: set up connections with tcp socket and implement your slots. If data bytes are available readyread() signal will be emmited.
void connectToHost(QString hostname, int port){
if(!m_pTcpSocket)
{
m_pTcpSocket = new QTcpSocket(this);
m_pTcpSocket->setSocketOption(QAbstractSocket::KeepAliveOption,1);
}
connect(m_pTcpSocket,SIGNAL(readyRead()),SLOT(readSocketData()),Qt::UniqueConnection);
connect(m_pTcpSocket,SIGNAL(error(QAbstractSocket::SocketError)),SIGNAL(connectionError(QAbstractSocket::SocketError)),Qt::UniqueConnection);
connect(m_pTcpSocket,SIGNAL(stateChanged(QAbstractSocket::SocketState)),SIGNAL(tcpSocketState(QAbstractSocket::SocketState)),Qt::UniqueConnection);
connect(m_pTcpSocket,SIGNAL(disconnected()),SLOT(onConnectionTerminated()),Qt::UniqueConnection);
connect(m_pTcpSocket,SIGNAL(connected()),SLOT(onConnectionEstablished()),Qt::UniqueConnection);
if(!(QAbstractSocket::ConnectedState == m_pTcpSocket->state())){
m_pTcpSocket->connectToHost(hostname,port, QIODevice::ReadWrite);
}
}
Write:
void sendMessage(QString msgToSend){
QByteArray l_vDataToBeSent;
QDataStream l_vStream(&l_vDataToBeSent, QIODevice::WriteOnly);
l_vStream.setByteOrder(QDataStream::LittleEndian);
l_vStream << msgToSend.length();
l_vDataToBeSent.append(msgToSend);
m_pTcpSocket->write(l_vDataToBeSent, l_vDataToBeSent.length());
}
Read:
void readSocketData(){
while(m_pTcpSocket->bytesAvailable()){
QByteArray receivedData = m_pTcpSocket->readAll();
}
}
TCP is inherently bidirectional. Get one way working (client connects to server). After that both ends can use send and recv in exactly the same way.
Have a look at QWebSocket, this is based on HTTP and it also allows for HTTPS
I'm trying to understand when and where to use the different built-in Akka mailboxes as well as when it is appropriate to roll your own. However, nowhere on that page does it explain what a "bounded mailbox" actually is, or how it behaves different than an unbounded mailbox. Also, that page categorizes mailboxes as "blocking" vs "non-blocking". And while I have a strong idea of what they mean by this (a message can be sent to a mailbox unless the mailbox is first emptied) I'm not 100% sure that I understand this. So seeing that I have no idea what the docs mean when they categorize a mailbox as bounded or blocking, it's tough for me to tell when I should be using each type.
Also, it seems like it is the default Akka behavior to clear out an actor's mailbox if that actor is restarted. I'd like to prevent this, but not sure if the solution is to use one of these built-in mailbox types (no mention of message persistence is mentioned on this page) or to somehow use persistent actors to accomplish such lossless-ness.
First, if an actor crashes and is restarted you only lose the current message that was being processed and not the entire mailbox.
A bounded mailbox has a limit to the number of messages it can have queued before it starts blocking the sender and not allowing the item if the queue doesn't go down while the sender is trying to put an item on. If you have concerns about memory and you can deal with potential message loss then you might want something like this. An unbounded mailbox has no limit on capacity at all so it could possible suffer memory issues if it gets flooded.
Whether it's bounded or not will affect whether or not it blocks. Blocking is generally not great for performance and should be avoided if the situation does not call for a bounded mailbox. That's why the default mailbox is unbounded; it will yield much better performance than a bounded counterpart.
The single consumer unbounded mailbox will most likely be the fastest because it is optimized to only have one consumer ever taking things off the queue. This means that you can not use a dispatcher that allows an actor instance to steal items from another actor instances mailbox (work distributing/stealing) but if you don't care about that then this mailbox might be the best bet for performance.
The priority based mailboxes allow you to provide code that allows the placement within the queue to vary depending on some attributes on the messages itself. This allows you to define the priority of the messages yourself and this will then shift higher priority items to the front of the queue regardless of the normal FIFO rules.