Using PyMQI, which is fine on IBM MQ Single instance queues, but does anyone know if I can pass dual IP Addresses and Ports on the connection string and if the MQ CLient under the hood handles the IBM MW Multi Instance queue management ?
PyMQI sits on top of the underlying MQ libraries. If you are using it with MQ v7.0 or higher then you can specify multiple connection names that are separated by a comma. It will then try each one in order and loop back to the first one if it can not connect to any of them. Some settings related to how long it will retry and how often can be set in the mqclient.ini.
The IBM Knowledge center page "Automatic client reconnection" has good general information on the reconnect options. All of it related to the C/C++ clients applies to PyMQI.
I'm going to implement a distributed message bus over ZeroMQ and I'm trying to make it as efficient as possible. My requirements are:
multiple processes are connected to a bus, some of them are on the same machine, some not.
processes can subscribe to some topics
unfortunately, no multicast (it is not supported in the production environment - Amazon)
I need multilanguage soultion (at least for C++, Haskell and Python)
Approaches I'm considering are:
1. Directory Service + Mesh topology
there is a single Directory Service which has a list of all connected processes and their addresses.
each process connects to DS on start and asks for addresses of others
each process has a Pub and Sub sockets connected to all other processes (mesh topology)
2. Broker
all processes are connected to a broker which distributes messages using Pub socket.
Are there any other/better architectures to use with ZeroMQ to create such message bus?
I suggest you to have a look to nanomsg, which has a built-in BUS topology and some
other interesting ones like SURVEY. It is a library by Martin Sustrik the original zmq
author.
You can find some discussion about BUS on Martin Sustrik blog: http://250bpm.com/blog:17
I'm trying to come up with the best solution for scaling a chat service in AWS. I've come up with a couple potential solutions:
Redis Pub/Sub - When a user establishes a connection to a server that server subscribes to that user's ID. When someone sends a message to that user, a server will perform a publish to the channel with the user's id. The server the user is connected to will receive the message and push it down to the appropriate client.
SQS - I've thought of creating a queue for each user. The server the user is connected to will poll (or use SQS long-polling) that queue. When a new message is discovered, it will be pushed to the user from the server.
SNS - I really liked this solution until I discovered the 100 topic limit. I would need to create a topic for each user, which would only support 100 users.
Are their any other ways chat could be scaled using AWS? Is the SQS approach viable? How long does it take AWS to add a message to a queue?
Building a chat service isn't as easy as you would think.
I've built full XMPP servers, clients, and SDK's and can attest to some of the subtle and difficult problems that arise. A prototype where users see each other and chat is easy. A full features system with account creation, security, discovery, presence, offline delivery, and friend lists is much more of a challenge. To then scale that across an arbitrary number of servers is especially difficult.
PubSub is a feature offered by Chat Services (see XEP-60) rather than a traditional means of building a chat service. I can see the allure, but PubSub can have drawbacks.
Some questions for you:
Are you doing this over the Web? Are users going to be connecting and long-poling or do you have a Web Sockets solution?
How many users? How many connections per user? Ratio of writes to reads?
Your idea for using SQS that way is interesting, but probably won't scale. It's not unusual to have 50k or more users on a chat server. If you're polling each SQS Queue for each user you're not going to get anywhere near that. You would be better off having a queue for each server, and the server polls only that queue. Then it's on you to figure out what server a user is on and put the message into the right queue.
I suspect you'll want to go something like:
A big RDS database on the backend.
A bunch of front-end servers handling the client connections.
Some middle tier Java / C# code tracking everything and routing messages to the right place.
To get an idea of the complexity of building a chat server read the XMPP RFC's:
RFC 3920
RFC 3921
SQS/ SNS might not fit your chatty requirement. we have observed some latency in SQS which might not be suitable for a chat application. Also SQS does not guarantee FIFO. i have worked with Redis on AWS. It is quite easy and stable if it is configured taking all the best practices in mind.
I've thought about building a chat server using SNS, but instead of doing one topic per user, as you describe, doing one topic for the entire chat system and having each server subscribe to the topic - where each server is running some sort of long polling or web sockets chat system. Then, when an event occurs, the data is sent in the payload of the SNS notification. The server can then use this payload to determine what clients in its queue should receive the response, leaving any unrelated clients untouched. I actually built a small prototype for this, but haven't done a ton of testing to see if it's robust enough for a large number of users.
HI realtime chat doesn't work well with SNS. It's designed for email/SMS or service 1 or a few seconds latency is acceptable. In realtime chat, 1 or a few seconds are not acceptable.
check this link
Latency (i.e. “Realtime”) for PubNub vs SNS
Amazon SNS provides no latency guarantees, and the vast majority of latencies are measured over 1 second, and often many seconds slower. Again, this is somewhat irrelevant; Amazon SNS is designed for server-to-server (or email/SMS) notifications, where a latency of many seconds is often acceptable and expected.
Because PubNub delivers data via an existing, established open network socket, latencies are under 0.25 seconds from publish to subscribe in the 95% percentile of the subscribed devices. Most humans perceive something as “realtime” if the event is perceived within 0.6 – 0.7 seconds.
the way i would implement such a thing (if not using some framework) is the following:
have a webserver (on ec2) which accepts the msgs from the user.
use Autoscalling group on this webserver. the webserver can update any DB on amazon RDS which can scale easily.
if you are using your own db, you might consider to decouple the db from the webserver using the sqs (by sending all requests the same queue), and then u can have a consumer which consume the queue. this consumer can also be placed behind an autoscalling group, so that if the queue is larger than X msgs, it will scale (u can set it up with alarms)
sqs normally updates pretty fast i.e less than one second. (from the moment u sent it, to the moment it appears on the on the queue), and rarely more than that.
Since a new AWS IoT service started to support WebSockets, Keepalive and Pub/Sub couple months ago, you may easily build elastic chat on it. AWS IoT is a managed service with lots of SDKs for different languages including JavaScript that was build to handle monster loads (billions of messages) with zero administration.
You can read more about update here:
https://aws.amazon.com/ru/about-aws/whats-new/2016/01/aws-iot-now-supports-websockets-custom-keepalive-intervals-and-enhanced-console/
Edit:
Last SQS update (2016/11): you can now use Amazon Simple Queue Service (SQS) for applications that require messages to be processed in a strict sequence and exactly once using First-in, First-out (FIFO) queues. FIFO queues are designed to ensure that the order in which messages are sent and received is strictly preserved and that each message is processed exactly once.
Source:
https://aws.amazon.com/about-aws/whats-new/2016/11/amazon-sqs-introduces-fifo-queues-with-exactly-once-processing-and-lower-prices-for-standard-queues/
Now on, implementing SQS + SNS looks like a good idea too.
I have an AMQP application that has a persistent RabbitMQ queue in the client side and a RabbitMQ queue in the server side. The client always writes in the local persistent queue and those messages are transmited to server using shovel plugin.
Producer -> Local Queue --------- SHOVEL ---------- Remote queue -> Consumer
Whether the server is not present the app stills works and shovel does the send when possible. In the other hand the server doesn't require to know the location of the clients becaiuse it consmes always from local queues. I would like to migrate this topology to AKKA using the FilePersistent Mailbox. Is it even possible? Is there something like Federation or Shovel plugin in Akka core libraries.
PS: What I want to achieve is replacing completetly AMQP to get rid of RabbitMQ. It works fine but is another piece of software to install, configure and mantain. I would like to provide all this functionality from my application using just libraries and not another server like RabbitMQ.
Just to clarify a little more what I'm looking to achieve is something like this:
Actor1 -> DurableMailBox1 ----Shovel? Federation?---- DurableMailbox2 <- Actor2
[EDIT]
It looks like there's no way to communicate directly mailbox to mailbox. The possible topologies that can be implemented with AKKA are these:
Remote Actor1 -> [DurableMailBox1 <- Actor2]
Where the arrow can be secured in order to ensure message delivery but is not possible to copy messages from one Mailbox to other Mailbox automatically.
Take a look at Akka Remoting and the Reliable Proxy Pattern.
Sending via a ReliableProxy makes the message send exactly as reliable
as if the represented target were to live within the same JVM,
provided that the remote actor system does not terminate. In effect,
both ends (i.e. JVM and actor system) must be considered as one when
evaluating the reliability of this communication channel. The benefit
is that the network in-between is taken out of that equation.
See this enhancement to the ReliableProxy that mitigates the problem with the remote actor system terminating.
I am using SonicMQ (version 7.6).
I have a Topic set up in production and code that publishes messages to that Topic.
I want to replicate these messages to a Queue running in a Sonic broker in another site.
Is it possible to configure SonicMQ to do this Topic-to-Queue replication, across brokers and sites?
There is no SonicMQ standard feature for doing this, as SonicMQ is mostly a standard JMS provider, so it has no built-in standard feature for manipulation or moving messages. SonicESB would be the "Sonic product" to solve such kind of problems.