We currently use kinesis to receive near real-time event data in json format and load it into s3 and Redshift.
In order to have online triggered complex events based on aggregations, we plan to use wso2 cep, with Storm for scalability.
What is the best practice here? Should storm be the one connected to kinesis?
When you are using WSO2 CEP with Storm scalability you should send events to WSO2 CEP which will publish the events to Storm and get the results back. Basically Storm deployment will be transparent and you will send and receive events via CEP. AFAIU You can send events to WSO2 CEP using desired method via a consumer in Kinesis. CEP supports many receiver types.
You can refer this to understand more about deployment architecture and configurations. This guide explains on how to create an execution plan and deploy in storm. To try out the functionality you can follow this sample.
Regards
Related
We are currently using AWS Lambda for some of the services with the following flow.
A rails application (kubernetes) adds a message to SQS queue
Lambda function is invoked via SQS trigger
Lambda function adds the notification to SNS
SNS calls the configured https endpoint to notify the rails application of the status
This has been working well for us. The function takes about 15 seconds to run (for generating some pdf with headless-chrome)
Due to Geographical data security restrictions for a separate installation of our application, we are unable to use AWS and the only feasible option is to use Oracle Cloud Infrastructure (OCI). OCI has cloud functions and also a Queue service, however unlike AWS, OCI doesn't seem to have inbuilt integration between cloud functions and Queue service.
One of the solutions we have discussed in the team is to deploy a service in kubernetes to consume the messages from the OCI Queue and invoke the cloud function and send the results to Notifications service.
I would appreciate any inputs that can simplify this flow but also maintain the async nature and scalability.
Rather than using OCI Queues you can send the events using OCI Streaming with a single subscriber
then you can link Functions easily and Notification service is available
I guess that when you are talking about service in K8s is 24/24 7/7 service and don't want to manage it through HPA/VPA.
If so, you can use https://knative.dev or alternatives https://ramitsurana.github.io/awesome-kubernetes/projects/projects/#serverless-implementations
I am new to Google Clout Iot core. I just tried the iot core, registered the device and published some data on a topic.
Now i am wondering is only Google functions are used to get/subscribe data from it? or can i also used the data published from a device directly into my applications e.g. node.js or python? like it is usually subscribed through a normal MQTT topic? I dont need a serverless architecture i want to use the data from google iot into my server and store/use it from there..
thanks a lot!
When a message arrives over MQTT or HTTP from a device over IoT Core, that message is published on a GCP Pub/Sub topic using the GCP Pub/Sub infrastructure. What this means is that you can develop a consuming application however you like as long as that application consumes the message from Pub/Sub.
The diagram shown here from the documentation is a good reference.
GCP exposes the API for Pub/Sub to be a subscriber. There are language bindings for a variety of languages.
Cloud Functions is typically seen as a serverless consumer of Pub/Sub messages as it scales to zero and can scale up if there is bursty load. However, there is nothing to prevent you from writing Compute Engine hosted applications or Container hosted applications which can act as subscribers.
References:
Cloud Pub/Sub documentation
Cloud Pub/Sub Client Libraries
I think your question is more on the architectural side of things. You can use IoT Core with something like Pub/Sub and write the data to Cloud Storage in order to later consume it with an application.
I would recommend you to check some of the diagrams from this page if you are interested in knowing which products you could use to best fit your needs.
I hope you find this useful!
AFAIK, one key difference between the two is that Appsync is implemented using MQTT and has an emphasis on real-time notification, while SNS is more like a general pub/sub service.
The real-time notification part of Appsync seems to be the right tool for a chat application. However with Appsync, I will not be able to push the notification to users if a connection is not established(?)
I would like to have real-time notification for users regardless the application is opened or not. What is the right tool for this purpose?
Generally, you'll probably have a more "ready-to-use" solution for chat using AppSync, e.g. https://aws.amazon.com/blogs/mobile/building-a-serverless-real-time-chat-application-with-aws-appsync/
Using SNS will probably provide more flexibility at the cost of more of your time wiring up custom code.
You are correct that AppSync doesn't natively support push notifications for background applications; for that, you'd need to wire in something like SNS. You could probably couple AppSync with SNS using an AppSync Lambda resolver.
Depending on your requirements, i.e. if it's not essential that a system notification appears for new messages, you can also query AppSync at app startup time for chat messages that were received while the client was disconnected.
Say I have 1000 IoT devices. Each device sends telemetry data to the Azure cloud each second. There is mobile app which displays that telemetry data. Mobile user can see data from specific IoT device only, based on his login. It's not an issue to receive all those d2c messages in the cloud, for example with Event Hubs (initially i wanted to use IoT Hub, but it's pretty expensive with huge amount of messages). It's also clear how to get general event stream (for example, by listening to events endpoint in IoT hub). However, this stream gives all messages, from all devices. Receiving all messages in mobile app and filtering them there does not look like right choice. How can I subscribe to events from specific IoT device only? Can I do this with Event Hubs and/or IoT Hub only, or do I need to utilize other Azure services? If yes, what will be the most cheap solution.
Update:
Original question was just about Azure. But is it possible to achieve what I need with AWS without requiring adding multiple parts to the system. Is Message Broker for AWS IoT what I need here (as I understood, this is built in functionality of AWS IoT Core)?
I am building real-time dashboard using aws services, currently my application using MySQL database(RDS), which service and how would be designed real-time dashboard using Amazon web service, currently my approach is to use kinesis with redshift and connect my application via JDBC connector or use kinesis with s3 and use Athena to show real-time aggregators.
please help.
Thanks in advance.
Although the Amazon Kinesis Docs mention that Kinesis streams can be used to send updates to dashboards, Kinesis has no native mechanism to do this alone. For some very good security reasons it's unwise to allow clients (i.e. from a webpage you serve) to access backend services like Kinesis directly.
Instead, you'll want to set up your application layer (Java, in this case) to listen to the Kinesis streams and expose any relevant events to your client, storing any changes you need to keep track of in your database (RDS, in this case). For a real-time dashboard I'd recommend using something like WebSockets to send events in real time to your webpage from your server, as they're widely supported and easy to use. Heres a tutorial on how to implement WebSockets on GlassFish