My company currently uses Azure for our Data Warehousing infrastructure. In the past we have used Azure Event Hubs for streaming data. When working on previous projects this hasn't been an issue we just provide the connection details and they start sending us data.
However we have recently started working on a new project where most of their infrastructure is hosted on AWS, we have been asked to set up a Amazon Kinesis endpoint instead as they do not support Azure Event Hubs.
I don't know much about sending the data, but is it asking a lot to send to an Event Hub instead of Kinesis?
My suggestion for this is you could introduce a middle layer which understands both Kinesis and Event hub. And the middle layer I know is Spring Cloud Stream. It provides binder abstraction to supports various message middleware such as Kafka, Kinesis and Event hub.
Related
My company is doing a POC on some streaming data and one of the tasks is sending data from AWS Kinesis to Azure Event Hubs.
Has anyone tryed to do something like this before?
I was thinking of a lambda function listening to kinesis firehose and sending the data to event hubs but I have no experience on Azure at all and I don't even know if this is possible.
Yes, this is very much possible.
Inter-Cloud environment where data can be streamed among two services can be achieved using AWS Kinesis and Azure Event Hub.
You can stream data from Amazon Kinesis directly to Azure Event Hub in Real-Time. Using ‘serverless’ model and cloud computing to process and transfer events without having the need to manage any native application written on an on-premise server.
You will be required connection string, SharedAccessKeyName, and SharedAccessKey from the Azure Event Hub. This will be needed to send data to Event Hub. Also, make sure the Event hub can receive data from the IP address you are running the program from.
Refer this third-party tutorial to accomplish the same
I am currently working on an IoT device using AWS IoT core. I am new to working with IoT device. What is the standard/best way for determining whether the device is online and connected to the internet?
Thanks you!
Since you have been using AWS IoT Core, I would recommend that you stay in fully managed services provided by AWS IoT suite. No need to reinvent the wheel such as provisioning a separate database for a basic requirement of pretty much every IoT-enabled solution.
What I understand is that you want to monitor your IoT device fleets for state changes or failures in operation, and to trigger actions when such events occur. To address this challenge, I'd suggest using AWS IoT Events. It accepts inputs from many different IoT telemetry data sources including smart sensors, edge devices, management applications, and other AWS IoT services. You can easily push any telemetry data input to AWS IoT Events by using a standard API interface.
In specific to device heartbeat, please take a look at this sample detector model. A detector model simply represents your equipment or process. On the console, you can find some other pre-made detector model templates which you can customize based on your use-case.
One way to know if a device is online is to check for a heartbeat.
A device heartbeat is a small mqtt message to a topic that the device sends every 5 minutes.
In IoT Core, you would configure a rule that would update a Dynamodb table with a timestamp each time a message is sent to the heartbeat topic.
By checking this timestamp in Dynamodb, you can confirm if your device is currently online.
You can follow this Developer Guide to get connect disconnect events. it works on MQTT topics so we can use rules to trigger Lambda or other services.
we have a microservices architecture developed on Google cloud.
Actually the microservices are all running on cloud run and talk each other with rest (sync) or with pub/sub (async).
It is an event-driven pattern so when a service publish something happened (like "user_created") on the right pub/sub topic, many services receive that event with a push subscription on their http endpoint.
Now we are moving to kafka for message ordering and replaying features.
Unfortunately kafka consumers are pull based so we need to change the way services are receiving events.
Since cloud run is a serverless solutions that scale to zero, we cannot make it listen to kafka topic, because the service could shut down during the night because no request arrive.
We have different services which can safely be updated with a scheduled cron, so every one hour as example, we make a get request to service, which download all new kafka messages and update itself accordingly.
But many other services need a near real time update to accomplish their role.
So which product of google cloud platform is best suited to consume kafka topic in this architecture?
Thanks!
I'm an AWS noob, I'm trying to figure out what the difference between Amazon's Kinesis Data Stream and EventBridge products. Can someone explain this for someone not familiar with the AWS tech stack?
Kinesis is a real-time stream processing service. Typically gets used for storing logs or end-user data coming from browser.
Event bridge is typically used to reliably communicate between apps / microservices, so it's quite similar to SQS, but has some added features.
Differences between SQS and Event Bridge are explained in the post below:
https://www.reddit.com/r/aws/comments/cjnw2l/what_makes_eventbridge_different_than_sqs_and/
I'd like to use AWS IoT to manage a grid of devices. Data by device must be sent to a queue service (RabbitMQ) hosted on an EC2 instance that is the starting point for a real time control application. I read how to make a rule to write data to other Service: Here
However there isn't an example for EC2. Using the AWS IoT service, how can I connect to a service on EC2?
Edit:
I have a real time application developed with storm that consume data from RabbitMQ and puts the result of computation in another RabbitMQ queue. RabbitMQ and storm are on EC2. I have devices producing data and connected to IoT. Data produced by devices must be redirected to the queue on EC2 that is the starting point of my application.
I'm sorry if I was not clear.
The AWS IoT supports pushing the data directly to other AWS services. As you have probably figured out by now publishing to third party APIs isn't directly supported.
From the choices AWS offers Lambda, SQS, SNS and Kinesis would probably work best for you.
With Lambda you could directly forward the incoming message using the one of Rabbit MQs APIs.
With SQS you would put it into an AWS queue first and than poll this queue transfering it to RabbitMQ.
Kinesis would allow more sophisticated processing, but is probably too complex.
I suggest you program a Lamba with the programming language of your choice using one of the numerous RabbitMQ APIs.