I am using SonicMQ (version 7.6).
I have a Topic set up in production and code that publishes messages to that Topic.
I want to replicate these messages to a Queue running in a Sonic broker in another site.
Is it possible to configure SonicMQ to do this Topic-to-Queue replication, across brokers and sites?
There is no SonicMQ standard feature for doing this, as SonicMQ is mostly a standard JMS provider, so it has no built-in standard feature for manipulation or moving messages. SonicESB would be the "Sonic product" to solve such kind of problems.
Related
I have a doubt - that which data structure does AMQP internally uses for storing messages.
I know it has
Transport (TCP/IP)
Messaging Protocol & Type System
Portable Representation of Message Properties
Message Broker – the MOM Layer
API Mappings
Routing between Brokers
Management –in a forthcoming addition
The AMQP Network Protocol
I believe it should be queue only .
If so then all the operations like blocking the queue and making it durable (etc) are performed on it ?
Any example of how it works (url or blogs will work)
Does AWS SQS follows AMQP architecture only?
AMQP standard (or better to say standards, since 0.9.x and 1.x are quite different) does not state, or prescribe how are messages to be stored. And in fact different implementations choose different ways of storing them.
Regarding AWS SQS I am not aware that it follows AMQP architecture.
Log files of my application keep accumulating on a server.I want to dump them into HDFS through KAFKA.I want the Kafka producer to read the log files,send them to Kafka broker and then move those files to another folder.Can the Kafka producer read log files ? Also, is it possible to have the copying logic in Kafka producer ?
Kafka maintains feeds of messages in categories called topics.
We'll call processes that publish messages to a Kafka topic producers.
We'll call processes that subscribe to topics and process the feed of published messages consumers..
Kafka is run as a cluster comprised of one or more servers each of which is called a broker.
So, at a high level, producers send messages over the network to the Kafka cluster which in turn serves them up to consumers like this:
So this is not a suitable for your application where you want to injest log files. Instead you can try flume.
Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data. It has a simple and flexible architecture based on streaming data flows. It is robust and fault tolerant with tunable reliability mechanisms and many failover and recovery mechanisms. It uses a simple extensible data model that allows for online analytic application.
As you know, Apache Kafka is publish-subscribe messaging system. you can send message from your application. To send message from your application you can use kafka clients or kafka rest api.
In short, you can read your log with your application and can send these logs to kafka topics.
To handle these logs, you can use apache storm. You can find many integrated solution for these purposes. And by using storm you can
add any logic your stream processing.
You can read many useful detailed information about storm kafka integration.
Also to put your processed logs to hdfs, you can easily integrate your storm with hadoop. You can check this repo for it.
Kafka was developed to support high volume event streams such as real-time log aggregation. From the kafka documentation
Many people use Kafka as a replacement for a log aggregation solution. Log aggregation typically collects physical log files off servers and puts them in a central place (a file server or HDFS perhaps) for processing. Kafka abstracts away the details of files and gives a cleaner abstraction of log or event data as a stream of messages. This allows for lower-latency processing and easier support for multiple data sources and distributed data consumption
Also I got this little piece of information from this nice article which almost similar to your use-case
Today, Kafka has been used in production at LinkedIn for a number of projects. There are both offline and online usage. In the offline case, we use Kafka to feed all activity events to our data warehouse and Hadoop, from which we then run various batch analysis
I want to connect a single queue to multiple queue managers
(atleast 2 qmgrs). Lets say I have qmgrA and qmgrB as queue managers and it is connected to queueName. I will put a message "Hello" to queueName connected to qmgrA as well as another message "World" on qmgrB. So suppose to be the queueName contains "Hello" and "World".
The question is how can I get those messages simultaneously? Can you give me an example code fragment/snippet for me to atleast have an overview of how can I start coding with that design.
Note: *I am asking for this because for example the qmgrA got disconnected/down for unknown reason, atleast qmgrB is still active
and will get messages on queueName even though qmgrA is disconnected.
By the way, I'm using Websphere MQ v7 C++.
Thanks in advance! :)
The question appears to be asking how to do something that IBM MQ does not do. QMgrs do not "connect" to queues, they host them. Applications do not "connect" to queues except in the abstract JMS sense. They connect to QMgrs and then open queues. However, the requirement that is described of keeping MQ highly available can be met with MQ Clusters, hardware clusters, Multi-Instance QMgrs, shared queues (on z/OS only), or some combination of these.
The two queue managers in your example each host a local copy of queueName. Here are several scenarios as to how messages are distributed in this situation.
PUTting messages
A message put to queueName by an application connected to qmgrA will by default result in the message landing in the local instance of that queue.
When there is a local instance and an MQ cluster with at least one other instance, the QMgr can be configured to allow messages to go to non-local instances.
When qmgrA, qmgrB, and qmgrC are in a cluster, and qmgrC does not host a local instance of the queue, messages put to that queue name will round-robin between the instances on qmgrA and qmgrB .
GETting messages
A message landing in queueName on qmgrA can only be retrieved from that queue by an application connected to qmgrA.
A message landing in queueName on qmgrB can only be retrieved from that queue by an application connected to qmgrB.
An application connected to qmgrC cannot retrieved messages from the queues hosted on qmgrA or qmgrB.
MQ HA
The requirement to make a queue highly available can be achieved in several ways. Some of these provide recoverability of the service provided by MQ. Some of them provide recoverability of messages that would be stranded in-flight. In addition to reading topology design section in the manuals, please see the MQ HA presentation from IBM Interconnect 2015.
Your question does make very much sense. Go look up MQ Clustering and MQ Multi-Instance in the MQ Knowledge Center. http://www-01.ibm.com/support/knowledgecenter/SSFKSJ_8.0.0/com.ibm.mq.con.doc/q017820_.htm
I'm newbie to AWS and trying to work on the SQS for the first time. I've an Oracle Service Bus (OSB) in non-cloud environment and would like to configure OSB to consume messages from Amazon SQS. The documentation mentions to use REST API and poll repeatedly for messages. I also read about the 'client library for JMS' so that the OSB could treat SQS as JMS provider. What is the best approach to achieve this? Appreciate your inputs.
The easiest (not necessarily the purest way) would be to create a Java EE app that imports the SQS libraries and pulls messages from AWS and puts them on a local queue for OSB to process. The example code snippets are in Java, so it should be relatively straight forward.
The purest way would be to set it up as a remote JMS provider. However, how to set that up is not so clear - you may end up writing most of the code that went into option #1 above, but making a JMS client library instead of a MDB.
I'm trying to come up with the best solution for scaling a chat service in AWS. I've come up with a couple potential solutions:
Redis Pub/Sub - When a user establishes a connection to a server that server subscribes to that user's ID. When someone sends a message to that user, a server will perform a publish to the channel with the user's id. The server the user is connected to will receive the message and push it down to the appropriate client.
SQS - I've thought of creating a queue for each user. The server the user is connected to will poll (or use SQS long-polling) that queue. When a new message is discovered, it will be pushed to the user from the server.
SNS - I really liked this solution until I discovered the 100 topic limit. I would need to create a topic for each user, which would only support 100 users.
Are their any other ways chat could be scaled using AWS? Is the SQS approach viable? How long does it take AWS to add a message to a queue?
Building a chat service isn't as easy as you would think.
I've built full XMPP servers, clients, and SDK's and can attest to some of the subtle and difficult problems that arise. A prototype where users see each other and chat is easy. A full features system with account creation, security, discovery, presence, offline delivery, and friend lists is much more of a challenge. To then scale that across an arbitrary number of servers is especially difficult.
PubSub is a feature offered by Chat Services (see XEP-60) rather than a traditional means of building a chat service. I can see the allure, but PubSub can have drawbacks.
Some questions for you:
Are you doing this over the Web? Are users going to be connecting and long-poling or do you have a Web Sockets solution?
How many users? How many connections per user? Ratio of writes to reads?
Your idea for using SQS that way is interesting, but probably won't scale. It's not unusual to have 50k or more users on a chat server. If you're polling each SQS Queue for each user you're not going to get anywhere near that. You would be better off having a queue for each server, and the server polls only that queue. Then it's on you to figure out what server a user is on and put the message into the right queue.
I suspect you'll want to go something like:
A big RDS database on the backend.
A bunch of front-end servers handling the client connections.
Some middle tier Java / C# code tracking everything and routing messages to the right place.
To get an idea of the complexity of building a chat server read the XMPP RFC's:
RFC 3920
RFC 3921
SQS/ SNS might not fit your chatty requirement. we have observed some latency in SQS which might not be suitable for a chat application. Also SQS does not guarantee FIFO. i have worked with Redis on AWS. It is quite easy and stable if it is configured taking all the best practices in mind.
I've thought about building a chat server using SNS, but instead of doing one topic per user, as you describe, doing one topic for the entire chat system and having each server subscribe to the topic - where each server is running some sort of long polling or web sockets chat system. Then, when an event occurs, the data is sent in the payload of the SNS notification. The server can then use this payload to determine what clients in its queue should receive the response, leaving any unrelated clients untouched. I actually built a small prototype for this, but haven't done a ton of testing to see if it's robust enough for a large number of users.
HI realtime chat doesn't work well with SNS. It's designed for email/SMS or service 1 or a few seconds latency is acceptable. In realtime chat, 1 or a few seconds are not acceptable.
check this link
Latency (i.e. “Realtime”) for PubNub vs SNS
Amazon SNS provides no latency guarantees, and the vast majority of latencies are measured over 1 second, and often many seconds slower. Again, this is somewhat irrelevant; Amazon SNS is designed for server-to-server (or email/SMS) notifications, where a latency of many seconds is often acceptable and expected.
Because PubNub delivers data via an existing, established open network socket, latencies are under 0.25 seconds from publish to subscribe in the 95% percentile of the subscribed devices. Most humans perceive something as “realtime” if the event is perceived within 0.6 – 0.7 seconds.
the way i would implement such a thing (if not using some framework) is the following:
have a webserver (on ec2) which accepts the msgs from the user.
use Autoscalling group on this webserver. the webserver can update any DB on amazon RDS which can scale easily.
if you are using your own db, you might consider to decouple the db from the webserver using the sqs (by sending all requests the same queue), and then u can have a consumer which consume the queue. this consumer can also be placed behind an autoscalling group, so that if the queue is larger than X msgs, it will scale (u can set it up with alarms)
sqs normally updates pretty fast i.e less than one second. (from the moment u sent it, to the moment it appears on the on the queue), and rarely more than that.
Since a new AWS IoT service started to support WebSockets, Keepalive and Pub/Sub couple months ago, you may easily build elastic chat on it. AWS IoT is a managed service with lots of SDKs for different languages including JavaScript that was build to handle monster loads (billions of messages) with zero administration.
You can read more about update here:
https://aws.amazon.com/ru/about-aws/whats-new/2016/01/aws-iot-now-supports-websockets-custom-keepalive-intervals-and-enhanced-console/
Edit:
Last SQS update (2016/11): you can now use Amazon Simple Queue Service (SQS) for applications that require messages to be processed in a strict sequence and exactly once using First-in, First-out (FIFO) queues. FIFO queues are designed to ensure that the order in which messages are sent and received is strictly preserved and that each message is processed exactly once.
Source:
https://aws.amazon.com/about-aws/whats-new/2016/11/amazon-sqs-introduces-fifo-queues-with-exactly-once-processing-and-lower-prices-for-standard-queues/
Now on, implementing SQS + SNS looks like a good idea too.