What is a good way to deploy a WebSockets client on AWS?
I'm building an app on AWS which needs to subscribe to several WebSockets and several REST sources and process incoming messages (WebSockets) or make periodic requests (REST). I'm trying to go server-less and maximize use of AWS platform services, to eliminate the need to manage VMs, OS patches, etc. (and hopefully reduce cost).
My idea so far is to trigger a Lambda function every time a message arrives. The function can then transform/normalize the message and push it to an SQS queue for further processing by other subsystems.
There would be two types of such Lambda clients, one that subscribes to WebSockets messages and another that makes HTTP request periodically when invoked by a CloudWatch schedule. It would look something like this:
This approach seems reasonable for my REST clients, but I haven't been able to determine if it's possible to subscribe to WebSockets messages using Lambda. Lambdas can be triggered by IoT, and apparently IoT supports WebSockets now, but apparently only as a transport for the MQTT protocol:
AWS IoT Now Supports WebSockets, Custom Keepalive Intervals, and Enhanced Console
What is the best/easiest/cheapest way to deploy a WebSockets client without deploying an entire EC2 or Docker instance?
Related
How to subscribe to AWS Event Bus events from client side applications ex: NodeJS app, Angular client or a Mobile client app ?
In December 2020, an email from AWS Marketing has presented the advantages to use the Event-Driven architecture . Following the documentation and the tutorials, soon I stumble into the wall of not finding a way to subscribe to this events from a client side application.
The email states:
4 Reasons to Care About Event-Driven Architectures
Are you looking to scale and build robust applications without delays and dependencies? We break down the basics of event-driven architectures, how they work, and show you ways to get started. Learn how event-driven architectures can help you:
Scale and fail independently - No more dependencies
Develop with agility - No custom polling code
Audit with ease - Use your event router to define policies
Cut costs - Stop paying for continuous polling
The disappointing part is that there is no example of libraries to be integrated in the client side code to subscribe to those event. Googling does not return any significant result and the only current library for node: #aws-sdk/client-eventbridge-node only expose a send and destroy methods.
There is no way to directly subscribe to an Amazon EventBridge bus, as it doesn't provide publish/subscribe functionality. In order to process events in EventBridge you create event rules that filter and send matching events to targets. You can find all targets available to EventBridge rules on this list: Amazon EventBridge Targets.
One of these targets could be an Amazon SNS topic, which provides pub/sub functionality, i.e. your client application can subscribe to the topic to automatically receive the respective events.
This may sound complicated at first, but the implementation is strictly following the principle of separating concerns. It provides simple building blocks--like Lego pieces--that you can put together in order to create truly loosely coupled architectures.
This diagram shows the functionality in scope of Amazon Event Bridge and how it communicates with other services and applications.
Services that allow you to subscribe as you want it (directly delivering subscribed messages to your code over a tcp connection such as a websocket) are:
AppSync - websocket
IOT Core - websocket, mqtt
SQS - long poll
Kafka
(From the top of my head)
So a straightforward serverless solution for you could be:
Event bridge —> SQS —> your code
I often use AppSync for this purpose. But eventbridge is cool too.
if you want to avoid polling and you cant/want use AWS Lambda then you can reverse the problem and call an api on your application from EventBridge with a rule.
You can create API Targets in EventBridge:
API destination
TLDR: Is there a way to trigger an AWS lambda or step function based on an external system's websocket message?
I'm building a synchronization service which connects to a system which supports websockets. I can use timers in step functions to wake periodically and call lambda functions to perform the sync, but I would prefer to subscribe to the websocket and perform the sync only when a message is received.
There are plenty of ways to expose websockets in AWS, but I haven't found a way to consume them short of something like an EC2 instance with a custom service running on it. I'm trying to stay in the serverless ecosystem.
It seems like consuming a websocket is a fairly common requirement; have I overlooked something?
Lambdas are ephemeral. They can't be sitting there waiting for a websocket message.
However, I think what you can do is use an Activity task. Once the step function gets to that state it will wait. The activity worker will run on an EC2 instance and subscribe to a websocket. When a message is received it will poll the State Machine for an activity token and call SendTaskSuccess. The state machine will then continue execution and call the lambda that performs the sync.
You can use AWS API gateway service and lambda. It supports web sockets and can trigger lambda on request
I'd like to use AWS IoT to manage a grid of devices. Data by device must be sent to a queue service (RabbitMQ) hosted on an EC2 instance that is the starting point for a real time control application. I read how to make a rule to write data to other Service: Here
However there isn't an example for EC2. Using the AWS IoT service, how can I connect to a service on EC2?
Edit:
I have a real time application developed with storm that consume data from RabbitMQ and puts the result of computation in another RabbitMQ queue. RabbitMQ and storm are on EC2. I have devices producing data and connected to IoT. Data produced by devices must be redirected to the queue on EC2 that is the starting point of my application.
I'm sorry if I was not clear.
The AWS IoT supports pushing the data directly to other AWS services. As you have probably figured out by now publishing to third party APIs isn't directly supported.
From the choices AWS offers Lambda, SQS, SNS and Kinesis would probably work best for you.
With Lambda you could directly forward the incoming message using the one of Rabbit MQs APIs.
With SQS you would put it into an AWS queue first and than poll this queue transfering it to RabbitMQ.
Kinesis would allow more sophisticated processing, but is probably too complex.
I suggest you program a Lamba with the programming language of your choice using one of the numerous RabbitMQ APIs.
I have an external data source as an ActiveMQ topic. I can only connect and consume messages. They come pretty rarely, about 1 message per 10-30 seconds.
I want to collect all the messages and put them into the database.
Also I'd like to have an active web page that can receive the new messages over WebSockets and draw a chart.
I have a prototype built with Python/Flask/MongoDB/SocketIO, BUT...
I would like to use Amazon AWS cloud infrastructure to avoid processing the data on servers.
I believe that AWS Lambda can accept the messages and store them into the database (DynamoDB?) and also send a notification (maybe using SQS) being transformed into WebSocket message. (Not everything is clear there yet, maybe simple ajax polling will be enough).
Here is a question: how it would be possible to consume the messages from external ActiveMQ topic and process it with AWS Lambda?
I was looking on Kinesis, but it looks it only supports the data being pushed to it, but not polling for the data by some protocol...
You can use Lambda as a cron-like facility and poll on a schedule. See Lambda Scheduled Events.
I am using firebase to notify web browsers (javascript clients) about changes on specific topics. I am very happy with it. However I would really like to (only) use aws web services.
Unfortunately I am not able to determine whether it is possible to build such a service on aws. I am not talking about having EC2 instances running some firebase / fanout.io alternatives. I am talking about utilizing services such as lambda, dynamodb streams, SNS & SQS.
Are there any socket notification services available or is it possible to achieve something similar by using the provided services?
I looked into this very recently with the same idea, but eventually I came down on just using fanout. AWS does not provide server-push HTTP notification services out of the box.
Lambda functions are billed per 100 ms, so any long-polling against lambda will end up billing for the entirety of the time the client is connected.
SNS does not provide long polling to browsers; the available clients are geared towards mobile, email, HTTP/S, and other Amazon products like Lambda and SQS.
SQS would require a dedicated queue per client as it does not support broadcast.
Now, if the lambda pricing doesn't bother you, you could possibly do this:
Write a lambda function that is called via the API service that opens up a connection to SQS and waits for a message. The key is to start the lambda call from HTTP, but within the function wait on the queue (using Boto, for example, if you are writing this in Python). This code would need to create a queue dedicated to servicing one particular client, uniquely keyed by something like a GUID that is passed in by the client.
Link to the lambda function using the Amazon API service.
Call the lambda function via the API from the browser and wait for it to either receive a message on the dedicated SQS queue or timeout, probably using long-polling both in the API connection and the SQS connection. Fully draining the queue (or at least taking as many messages in a batch up to some limit) would be advisable here as well in order to reduce the number of calls to the API.
Publish your event to the dedicated SQS queue associated with the client. This will require the publisher to know the client's unique key.
Return the event read from SQS as the result of the lambda call.
Some problems with this approach:
Lambda pricing - not terribly expensive, but something like fanout is basically free
You would need a dedicated SQS queue per client; cleanup might become a problem
SQS bills on number of calls, which includes checking for a message. Long-polling SQS will alleviate some of this
You would need to write the JavaScript client to call the lambda API endpoint repeatedly in a long-polling fashion
Lambda is currently limited as to the number of concurrently running functions it supports (100 right now but you can contact support to bump that up)
Some benefits with this approach:
SQS queues are persistent, so unless a message is processed successfully it will go back on the queue after the visibility timeout
You can set up CloudWatch to monitor all of the API, Lambda, and SQS events
Other Notes
You could call the SQS APIs directly from the browser by using Lambda to issue temporary security credentials via STS. Receiving a message in JavaScript is documented here: http://docs.aws.amazon.com/AWSJavaScriptSDK/guide/browser-examples.html#Receiving_a_message I do not, however, know off the top of my head if you would run into cross-domain issues.
Your only other option, if it must be all AWS, is to use load-balanced EC2 instances running something like fanout as you mentioned.
Using fanout is very little work: it's both extremely affordable and already built and tested.