I am creating a serverless infrastructure with multiple functions. So far I have managed to publish a new function on AWS lambda using the aws-sam-cli.
One of the last functions is my firebase listener which is supposed to trigger certain aws lambda functions.
Intially, I thought to create a new function and add the listener as follows:
import firebase_admin
cred = firebase_admin.credentials.Certificate(cert_json)
app = firebase_admin.initialize_app(cred, config)
bucket = storage.bucket(app=app)
node_to_listen = '/alerts/'
firebase_admin.db.reference(node_to_listen).listen(listener)
However, the issue is that AWS lambda seems to be designed not to run functions continuously but only be triggered by events. This is true as well for the Firebase listen() function, which means that we get a chicken or egg problem, who triggers who?
How can I therefore publish the firebase listener function and where? Should it be deployed somewhere else (e.g. Heroku?) in order to continuously listen and send the event requests to aws lambda? Or is there a way to connect those two?
There's no way to keep an active listener in any Functions-as-a-Service environment that I know of. The whole purpose of such environments is to run (short) workloads in response to events. You are trying to actually to trigger an event by keeping a listener, which simply doesn't fit the FaaS model.
The two solutions I can see:
Implement your listener on an environment that keeps an active process.
Implement your listener on a FaaS environment that can itself listen to Firebase Realtime Database events. The only environment that can current do so it Cloud Functions, which has Firebase Realtime Database as an event source. So you'd then trigger your Lambda function from Cloud Functions.
The second solution is the only one that really feels fully serverless, but it seems a bit weird to trigger Amazon Lambda from Google Cloud Functions.
There is work under way to allow interop between FaaS providers. But I'm not sure of the current status (link to spec/working group welcome), nor if your scenario would be covered in there.
Related
I'm using a service that runs outside of my premises that uses a websocket API to emit events that occur inside it. Let's call this wss://X.
I have a web app that currently runs a websocket client that connects to wss://X and if it receives an event fires a call back that does some book keeping (update a database, queue a notification).
I'd like to keep this separate from my web app (different developer, separate logic etc.). I'd like to have something I'm called a listener for wss://X which will just sleep till something comes up on this connection and if so, do something and then shut down.
This seems like a good candidate for AWS lambda. Short running functions that do book keeping based on events but I'm not sure how I can trigger them. Someone suggested AWS API gateway but I don't know how to make it listen (rather than bind) to a certain URL and then trigger lambda functions based on events that it receives. If this is not possible, what can I use on AWS that would solve this problem for me?
You can't use Lambda for long-running tasks (refer to Is it possible to subscribe to a WebSocket in a serverless fashion using AWS?).
Considering that the listener needs to be long-running as it must listen to events from wss://X, an alternative that I can think of is to start an EC2 instance that runs code which continuously listens to wss://X and does the operations that you want.
Currently, I have a working modded Minecraft server working running on a C5 EC2 instance. The problem is that I have to manually start and stop the server which can be annoying for my friends. I was wondering if it would be possible to automate the EC2 state so that it runs as soon as a player attempts to join the sever. This would be similar to how Minecraft realms behaves which I heard Mojang was using AWS for:
https://aws.amazon.com/blogs/aws/hosting-minecraft-realms-on-aws/
I have looked up tutorials for this and this is the best I could come across:
https://github.com/trevor-laher/OnDemandMinecraft
The problem with this solution is that it requires to make a separate website to log users in and start the EC2 instance while I want the startup and shutdown to be completely automatic.
I would appreciate any guidance.
If the server is off, it would not be possible to "connect" to the server. Therefore, another mechanism is required that can be used to start the server.
Combine that with your desire to minimise costs and the only real solution is to somehow trigger an AWS Lambda function, which could start the server.
There are a few ways you could have users trigger the AWS Lambda function:
Make a call to API Gateway
Upload an object to Amazon S3
Somehow put a message in an SNS topic or an SQS queue
Trigger an Amazon CloudWatch Alarm (which calls Lambda via SNS)
...and probably other ways
When considering a method to use, you should consider security implications such as:
Whether only authorized users should be able to trigger the Lambda function, or is it okay that anybody (eg a web crawler) might trigger it.
Whether you willing to give your friends AWS credentials (not a good idea) that they could use to start the server directly, or whether it should be an indirect method.
Frankly, I would recommend the following architecture:
Create an AWS Lambda function that turns on the server
Create an API Gateway that triggers the Lambda function
Give a URL to your friends that calls the API Gateway and passes a 'secret' (effectively a password)
The API Gateway will call the Lambda function, passing the secret
The Lambda function confirms that the secret is correct and starts the Amazon EC2 instance with Minecraft installed
Here is a tutorial that shows many of these concepts: Build an API Gateway API with Lambda Integration
The purpose of the secret is to avoid the server from starting if an unauthorized person (or a bot) happens to hit the API Gateway endpoint. They will not provide the secret, so the server will not be started.
Stopping the server after a period of non-use is a different matter. The library you referenced might be able to assist with finding a way to do this. You could have a script running on the Minecraft server that monitors the game and, after a period of inactivity, simply calls the operating system to perform a Shutdown.
You could use a BungeeCord hub server that then allows user to begin a connection to the main server and spin it up via. AWS.
This would require the bungee server to be always up however, but the cost of hosting a small bungee server should be relatively cheap.
I don't think there's any way you could do this without having a small server that receives the initial request to spin up the AWS machine.
I am looking for a service or framework in Native AWS which given, a csv file, creates a task and process that task asynchronously and returns a task id or job id to the client and notifies the client when the task is completed. Some requirements for this:
Client should be able to check the progress of the task by job id at any time.
Processing of entire task can take more than 15 mins.
There should be a way for clients to see the reasons of failures.
All the business logic would be at line item level. (this is the only thing developer should care about)
Is there any in-built service or framework for that in Native AWS? I know one can build this kind of service using some SQS, Lambda, SNS, Dynamodb but I am just looking if there is a already available AWS offering for it, which can do all of these?
The closest service to this concept is AWS Step Functions.
However, it would just be one component of a solution. You would still need to create the compute component by using Amazon EC2 or AWS Lambda. You would need to build the interface for users, add authentication, notifications, etc.
Bottom line: There is no AWS service that does what you describe. However, there are the building blocks if you wish to create one yourself.
I'm currently setting up a small Lambda to take snapshots of all the important volumes of our EC2 instances. To guarantee application consistency I need to trigger actions inside the instances: One to quiesce the application before the snapshot and one to wake it up again after the snapshot is done. So far I have no clue how to do this.
I've thought about using SNS or SQS to notify the instances about start and stop of the snapshot, but that has several problems:
I'll need to install (and develop) a custom listener inside the instances.
I'll not get feedback if the quiescing/wake-up is done.
So here's my question: How can I trigger an action inside an instance from an Lambda?
But maybe I'm approaching this from the wrong direction. Is there really no simple backup solution? I know azure has a snapshot based backup service that can do application consitent backups. Did I just miss an equivalent AWS service?
Edit 1:
Ok, it looks like the feature 'Run Command' of AWS Systems Manager is what I really need. It allows me to run scripts, Ansible playbooks and more inside an EC2 instance. When I've got a working solution I'll post the necessary steps.
You can trigger a Lambda function on demand:
Using AWS Lambda with Amazon API Gateway (On-Demand Over HTTPS)
You can invoke AWS Lambda functions over HTTPS. You can do this by
defining a custom REST API and endpoint using Amazon API Gateway, and
then mapping individual methods, such as GET and PUT, to specific
Lambda functions. Alternatively, you could add a special method named
ANY to map all supported methods (GET, POST, PATCH, DELETE) to your
Lambda function. When you send an HTTPS request to the API endpoint,
the Amazon API Gateway service invokes the corresponding Lambda
function. For more information about the ANY method, see Step 3:
Create a Simple Microservice using Lambda and API Gateway.
I'm hoping to use Amazon Lambda to run some background tasks for my web app. These particular tasks will only need to run once for the app (not once per user), so I'd like any user to see in the UI if a task is already running, and I'd like to disable the UI that allows them to start that task again.
Does Lambda offer a way to check the status of a function to see if it is running? If not, what is the best way to persist this info to my web app? Am I taking the wrong approach here altogether?
Lambda functions are supposed to be stateless and keeping functions stateless enables AWS Lambda to rapidly launch as many copies of the function as needed to scale to the rate of incoming events. While AWS Lambda’s programming model is stateless, your code can access stateful data by calling other web services, such as Amazon S3 or Amazon DynamoDB.