I'm an AWS noob, I'm trying to figure out what the difference between Amazon's Kinesis Data Stream and EventBridge products. Can someone explain this for someone not familiar with the AWS tech stack?
Kinesis is a real-time stream processing service. Typically gets used for storing logs or end-user data coming from browser.
Event bridge is typically used to reliably communicate between apps / microservices, so it's quite similar to SQS, but has some added features.
Differences between SQS and Event Bridge are explained in the post below:
https://www.reddit.com/r/aws/comments/cjnw2l/what_makes_eventbridge_different_than_sqs_and/
Related
I want my local Java application to know that some changes made to the item in AWS DynamoDB table. Is it posible without polling? I understand that I should use DynamoDB Stream to invoke AWS Lamda trigger but what should Lambda function do to send information to my local app? If it's possible.
In my opinion Lambda is not a good fit here. AWS provides reference architectures that will meet your needs. Usually it goes like this:
DynamoDB Stream -> Kinesis Adapter for DynamoDB Streams -> Kinesis Client Library
Take a closer look at these documents:
https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.KCLAdapter.html
https://aws.amazon.com/blogs/database/dynamodb-streams-use-cases-and-design-patterns/
You can also design your app as this (the top part) and consume events from SQS long polling, but using Kinesis Client Library seems more straightforward and easier:
I have a process which publish some data(json) onto a queue on Aws-Sqs. Another process reads from this queue. All this is working fine.
Now I want to create a topic which can be listened by mutiple processes and the data is delivered to all the processes. For example Activemq and many other messaging servers have this capability to create a topic. I could not find any such thing on AWS. The closest I could find is AWS SNS.
From what I understand AWS-SNS allows multiple clients to subscribe to a topic. But the type of subscription is either Email, Http, or Sms and so on ... This does not really serve my purpose. I want to recieve json data in all my clients just like Sqs.
Is that achievable? If so how?
You can subscribe multiple SQS into single SNS topic: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-subscribe-queue-sns-topic.html
Then it will be distributed to all of them.
The other option is to use Kinesis - https://aws.amazon.com/kinesis/ but it is more difficult to set up. There you can also read from multiple clients from the stream.
amazon mq is a managed active mq service. maybe this will help with your needs?
I'm evaluating AWS Kinesis vs Managed Service Kafka (MSK). Our requirement is sending some messages (JSON) to AWS to from the on-prem system (system develop using c++). Then we need to persist above messages into the relational database like PostgreSQL, and same time we need to stream above data into some other microservices (java) which hosted in AWS.
I have the following queries:
i) How can I access(connect and send messages) to AWS Kinesis from my on-premise system? Is there any C++ API supporting that? (There are java client API, but our on-prem system written on C++)
ii) How can I access(connect and send messages) to AWS MSK from my on-premise system?
iii) Is it possible to integrate MSK with other AWS service (e.g lambda, Redshift, EMR, etc)?
iv) To persist data into a database can we use AWS lambda? (AWS Kinesis supporting that functionality, what about AWS MSK)
v) Our message rate is 50msg/second and what is the cost-effective solution?
To be blunt, your use case sounds simple and 50 messages a second is a very low rate.
Kinesis is a firehose where you need a straw. Kinesis is meant to ingest, transform and process terabytes of moving data. ]
Have you considered rather looking at SQS or Amazon MQ ? Both are considerably simpler to use and manage than Kafka or Kinesis. Just from your questions it's clear you have not interacted with Kafka at all, so you're going to have a steep learning curve. SQS is a simple api-based queueing system - you publish to an SQS queue, and you consume from the queue. If you don't need to worry about ordering, routing, etc it is a persistent and reliable (if clunky) technology that lots of people use to great success.
To answer your actual questions:
Amazon publishes a C++ SDK for their services - I would be stunned if there wasn't a Kinesis client as part of this. You would either need a public Kinesis endpoint, or a private Kinesis endpoint accessible via some sort of tunnel or gateway between your on-prem network and your AWS vpc.
MSK is Kafka. You need an Apache Kafka C++ client, and similar to kinesis above you will need some sort of tunnel or gateway from your on-prem network to the AWS vpc where you have provisioned MSK
It's possible, but it's unlikely there are any turn-key solutions for this. You will have to write some sort of bridging software from Kafka -> Other systems
You can possibly use Lambda, so long as you cater for failures, timeouts, and other failure modes. To be honest, a stand-alone consumer running as a service in your vpc or on-prem is a better idea.
SQS or Amazon MQ as previously mentioned are likely to be simpler and more cost-effective than MSK, and will almost certainly be cheaper than Kinesis.
My company currently uses Azure for our Data Warehousing infrastructure. In the past we have used Azure Event Hubs for streaming data. When working on previous projects this hasn't been an issue we just provide the connection details and they start sending us data.
However we have recently started working on a new project where most of their infrastructure is hosted on AWS, we have been asked to set up a Amazon Kinesis endpoint instead as they do not support Azure Event Hubs.
I don't know much about sending the data, but is it asking a lot to send to an Event Hub instead of Kinesis?
My suggestion for this is you could introduce a middle layer which understands both Kinesis and Event hub. And the middle layer I know is Spring Cloud Stream. It provides binder abstraction to supports various message middleware such as Kafka, Kinesis and Event hub.
Im writing an app using the AWS framework. It would be the perfect framework if the SQS had FIFO access!
I am using SNS to publish messages to queues, and the order of the messages is important. Does anyone know of a MQ that can be used, that is scalable etc.. like SQS that can be used?
I have been looking at using ironMQ but the SNS seems to have issues with receiving the messages?
(or even better yet,if you know of a good way of ordering messages with SQS)
Thanks,
Ben
These days, Amazon Kinesis would be a good choice. (It wasn't available when the question was raised.)
Kinesis captures data in streams, retains order and allows the data to be replayed and even consumed by multiple applications.