Amazon EventBridge enables developers to connect 3rd party event-driven applications with Amazon services. Amazon AppFlow offers event-driven integration with 3rd party apps as well.
What is the difference between the two services for the event-driven scenarios and when to use one over the other?
The AppFlow FAQ contains this info:
Amazon EventBridge enables developers to build event driven
applications that interact with SaaS applications and AWS services.
SaaS applications that have integrated with EventBridge emit events to
the customer’s event bus, which can then be routed to targets such as
Amazon EC2 instances or Lambda functions for processing. AppFlow
supports bi-directional transfer of data between SaaS applications and
AWS services that may be initiated by humans using a UI, a schedule,
or events - all with a point and click interface.
AppFlow is great for anyone who wants to connect together their applications without writing any code. EventBridge works with Step Functions and Lambda (and various other AWS services), and so is well suited for developers who need the extra flexibility that those services provide. Right now EventBridge is only one way, whereas AppFlow allows you to send data back to SaaS apps. The list of supported partners for each service is also different.
For some of AppFlow's partners, data is received via API polling (from the FAQ: "AppFlow is a fully managed API integration service that replaces custom connectors"), which is slightly different from EventBridge which receives data in an event-driven manner, where an event is sent via an HTTP call from the partner as soon as a change occurs. The pricing of the two services also differs: EventBridge charges $1/million events, and AppFlow charges $0.001 per flow (with an additional charge per GB of data processed).
Related
We are currently using AWS Lambda for some of the services with the following flow.
A rails application (kubernetes) adds a message to SQS queue
Lambda function is invoked via SQS trigger
Lambda function adds the notification to SNS
SNS calls the configured https endpoint to notify the rails application of the status
This has been working well for us. The function takes about 15 seconds to run (for generating some pdf with headless-chrome)
Due to Geographical data security restrictions for a separate installation of our application, we are unable to use AWS and the only feasible option is to use Oracle Cloud Infrastructure (OCI). OCI has cloud functions and also a Queue service, however unlike AWS, OCI doesn't seem to have inbuilt integration between cloud functions and Queue service.
One of the solutions we have discussed in the team is to deploy a service in kubernetes to consume the messages from the OCI Queue and invoke the cloud function and send the results to Notifications service.
I would appreciate any inputs that can simplify this flow but also maintain the async nature and scalability.
Rather than using OCI Queues you can send the events using OCI Streaming with a single subscriber
then you can link Functions easily and Notification service is available
I guess that when you are talking about service in K8s is 24/24 7/7 service and don't want to manage it through HPA/VPA.
If so, you can use https://knative.dev or alternatives https://ramitsurana.github.io/awesome-kubernetes/projects/projects/#serverless-implementations
I am a newbie to AWS. As I understand, both AWS MQ and AWS SQS are Message Queue tools. The only noted difference is that SQS is fully managed.
When should we use SQS or MQ?
SQS is a simple queueing service. It doesn't support many higher level abstractions like message routing, fanouts, distribution lists etc. It is a queue - a message is produced, and a message is delivered. It is useful when you need a Queue with limited backing logic.
AWS MQ is a managed Apache ActiveMQ(or RabbitMQ) broker service.
This provides you a fully managed Apache ActiveMQ system in the cloud, with support for a variety of industry-standard queue and broadcast protocols like AMQP, JMS etc. It is useful when you have complicated delivery rules - or when you're migrating an existing system from outside AWS into AWS, and your systems happen to talk to one another with a standard queueing protocol.
From AWS Documentation, you can search "Q: When should I use Amazon MQ vs. Amazon SQS and SNS?":
Amazon MQ, Amazon SQS, and Amazon SNS are messaging services that are
suitable for anyone from startups to enterprises. If you're using
messaging with existing applications, and want to move your messaging
to the cloud quickly and easily, we recommend you consider Amazon MQ.
It supports industry-standard APIs and protocols so you can switch
from any standards-based message broker to Amazon MQ without rewriting
the messaging code in your applications. If you are building brand new
applications in the cloud, we recommend you consider Amazon SQS and
Amazon SNS. Amazon SQS and SNS are lightweight, fully managed message
queue and topic services that scale almost infinitely and provide
simple, easy-to-use APIs. You can use Amazon SQS and SNS to decouple
and scale microservices, distributed systems, and serverless
applications, and improve reliability.
Also, in this doc, you can check comparisons with other services.
I am new to Google Clout Iot core. I just tried the iot core, registered the device and published some data on a topic.
Now i am wondering is only Google functions are used to get/subscribe data from it? or can i also used the data published from a device directly into my applications e.g. node.js or python? like it is usually subscribed through a normal MQTT topic? I dont need a serverless architecture i want to use the data from google iot into my server and store/use it from there..
thanks a lot!
When a message arrives over MQTT or HTTP from a device over IoT Core, that message is published on a GCP Pub/Sub topic using the GCP Pub/Sub infrastructure. What this means is that you can develop a consuming application however you like as long as that application consumes the message from Pub/Sub.
The diagram shown here from the documentation is a good reference.
GCP exposes the API for Pub/Sub to be a subscriber. There are language bindings for a variety of languages.
Cloud Functions is typically seen as a serverless consumer of Pub/Sub messages as it scales to zero and can scale up if there is bursty load. However, there is nothing to prevent you from writing Compute Engine hosted applications or Container hosted applications which can act as subscribers.
References:
Cloud Pub/Sub documentation
Cloud Pub/Sub Client Libraries
I think your question is more on the architectural side of things. You can use IoT Core with something like Pub/Sub and write the data to Cloud Storage in order to later consume it with an application.
I would recommend you to check some of the diagrams from this page if you are interested in knowing which products you could use to best fit your needs.
I hope you find this useful!
For the purpose of sending push notifications from the backend, if we need a Push Notification Platform, could you please suggest which of these is intended for that purpose – Amazon SNS or Pinpoint?
Both Amazon SNS and Amazon Pinpoint, supports sending push notification to various devices (e.g Android, iOS etc)
The major difference between Amazon SNS & Amazon Pinpoint is that :
with Amazon SNS you have to set up your application to manage each message's audience, content, and delivery schedule. On the other hand, with Amazon Pinpoint you do not have to code these features, most of them are already built in. With Amazon Pinpoint, you can collect data about your app usage, create highly-targeted segments and send full campaigns(either immediate or scheduled) plus many more features.
I am building real-time dashboard using aws services, currently my application using MySQL database(RDS), which service and how would be designed real-time dashboard using Amazon web service, currently my approach is to use kinesis with redshift and connect my application via JDBC connector or use kinesis with s3 and use Athena to show real-time aggregators.
please help.
Thanks in advance.
Although the Amazon Kinesis Docs mention that Kinesis streams can be used to send updates to dashboards, Kinesis has no native mechanism to do this alone. For some very good security reasons it's unwise to allow clients (i.e. from a webpage you serve) to access backend services like Kinesis directly.
Instead, you'll want to set up your application layer (Java, in this case) to listen to the Kinesis streams and expose any relevant events to your client, storing any changes you need to keep track of in your database (RDS, in this case). For a real-time dashboard I'd recommend using something like WebSockets to send events in real time to your webpage from your server, as they're widely supported and easy to use. Heres a tutorial on how to implement WebSockets on GlassFish