We are currently using AWS Lambda for some of the services with the following flow.
A rails application (kubernetes) adds a message to SQS queue
Lambda function is invoked via SQS trigger
Lambda function adds the notification to SNS
SNS calls the configured https endpoint to notify the rails application of the status
This has been working well for us. The function takes about 15 seconds to run (for generating some pdf with headless-chrome)
Due to Geographical data security restrictions for a separate installation of our application, we are unable to use AWS and the only feasible option is to use Oracle Cloud Infrastructure (OCI). OCI has cloud functions and also a Queue service, however unlike AWS, OCI doesn't seem to have inbuilt integration between cloud functions and Queue service.
One of the solutions we have discussed in the team is to deploy a service in kubernetes to consume the messages from the OCI Queue and invoke the cloud function and send the results to Notifications service.
I would appreciate any inputs that can simplify this flow but also maintain the async nature and scalability.
Rather than using OCI Queues you can send the events using OCI Streaming with a single subscriber
then you can link Functions easily and Notification service is available
I guess that when you are talking about service in K8s is 24/24 7/7 service and don't want to manage it through HPA/VPA.
If so, you can use https://knative.dev or alternatives https://ramitsurana.github.io/awesome-kubernetes/projects/projects/#serverless-implementations
Related
I have a template that creates a few resources in GCP, and I want it to either call an HTTP endpoint or publish a message to a topic whenever the deployment completes. I've been checking different services all day, and couldn't find anything about it.
In AWS, it is quite easy to deploy an SNS message that is published to an SNS topic, which is subscribed to SQS Queue, and that triggers a lambda function. In Azure ARM templates, we can use az CLI to invoke a web request and call an endpoint directly.
I couldn't find any similar in GCP. Is there any way of either calling an HTTP endpoint, Cloud Function or perhaps publishing a message to a topic whenever a deployment is finished?
I really appreciate any help.
The best approach in GCP is to Create a Logging Sink using a filter and the Logging query language to only send the Deployment Manager logs to a PubSub topic.
Here is an example of a sink sending deployment manager logs to a PubSub topic previously created:
gcloud logging sinks create $SINK_NAME pubsub.googleapis.com/projects/$YOUR_PROJECT/topics/$TOPIC \
--log-filter='resource.type="deployment"' --description="my sink"
Be careful to Set the destination permissions or will not see the logs in the PubSub topic.
Once you are getting the logs in the PubSub topic, you can configure Cloud Pub/Sub Triggers to fire up an HTTP call based on content of the log.
I am new to AWS and I am trying to build a web application where every time I type in my name and address and click the submit button, I want this information to be sent to SNS to notify a device that a new entry has been received along with the inputs given. I am wondering if I should first create a lambda function to get my name and address before triggering the SNS. I am still not clear what would be a good workflow here.
If you want to interact with Amazon SNS from a web app, you do not need to write a lot of Lambda functions to interact with the service. Instead, you can directly invoke the SNS Service by using the Amazon SNS API.
For example, assume you want to write a Web app using Java. To solve this use case, you can write the Web App using Spring BOOT and then use the SNS Java API to invoke SNS Service Operations, as discussed in this development article.
Creating a Publish/Subscription Spring Boot Application
Likewise, if you are using another programming language, you can use that given API to build the web app and invoke SNS operations.
Can I publish an event into Cloud Pub/Sub outside from GCP?
Let me clarify my query a bit. In AWS as we are able to publish events into SNS topics directly by invoking REST API via API Gateway from the non-cloud client (https://github.com/cdk-patterns/serverless/blob/master/the-big-fan/README.md), is there any such method in GCP to publish an event into Pub/Sub?
I can see there is a similar question in SO (Acces Google Pub/Sub from outside of GCP), but it is not fully answered my question I believe. Yes authentication is required and it is a cross-cutting functionality, but what is the basic technic to publish an event in Pub/Sub outside from GCP
Yes, you can publish a message into a topic, and then pull it, or even have PubSub deliver it to you through a Push subscription.
When publishing a message, or pulling it from a subscriber, you can access PubSub through the REST or RPC API. In addition, you can use one of the client libraries.
Here you can find an example of how to publish a message using the gcloud CLI tool, an example with the REST API, python and java among other programming languages.
As mentioned in the question referenced, you will need to authenticate in order to either publish a message, or pull it. You can use the quickstart as a reference on how to do so. Notice that you can follow the quickstart from any computer or VM outside GCP.
Finally, if you're using Push subscriptions to receive your messages, your endpoint will need to be a publicly accessible HTTPS address and have a valid SSL certificate signed by a certificate authority. Again, this endpoint can live outside GCP.
Amazon EventBridge enables developers to connect 3rd party event-driven applications with Amazon services. Amazon AppFlow offers event-driven integration with 3rd party apps as well.
What is the difference between the two services for the event-driven scenarios and when to use one over the other?
The AppFlow FAQ contains this info:
Amazon EventBridge enables developers to build event driven
applications that interact with SaaS applications and AWS services.
SaaS applications that have integrated with EventBridge emit events to
the customer’s event bus, which can then be routed to targets such as
Amazon EC2 instances or Lambda functions for processing. AppFlow
supports bi-directional transfer of data between SaaS applications and
AWS services that may be initiated by humans using a UI, a schedule,
or events - all with a point and click interface.
AppFlow is great for anyone who wants to connect together their applications without writing any code. EventBridge works with Step Functions and Lambda (and various other AWS services), and so is well suited for developers who need the extra flexibility that those services provide. Right now EventBridge is only one way, whereas AppFlow allows you to send data back to SaaS apps. The list of supported partners for each service is also different.
For some of AppFlow's partners, data is received via API polling (from the FAQ: "AppFlow is a fully managed API integration service that replaces custom connectors"), which is slightly different from EventBridge which receives data in an event-driven manner, where an event is sent via an HTTP call from the partner as soon as a change occurs. The pricing of the two services also differs: EventBridge charges $1/million events, and AppFlow charges $0.001 per flow (with an additional charge per GB of data processed).
I have developed a simple microservice, REST based using Java 8 and Spring Boot2.0. It has its own REST end points which I can call using Postman and I get the response very well. Now I have doubt in understanding the design & architecture if I want to deploy the same application on AWS cloud. I want my application to behave as serverless so I want to deploy on AWS using its Lambda service.
Please assist to clear my following doubts :-
1) First, can I upload my whole application code to AWS Lambda in order to make it serverless?
2) If yes, then do I need to use AWS API Gateway (compulsorily) to invoke my Lambda function when the request passes through it?
3) If yes (point 2), then end points which are there in my original microservice code will become ineffective and will be overridden by new API Gateway end points?
My whole doubt is about end points, which end point will be used to invoke the Lambda functions?
Please assist to clarify my doubt. If there is any sample reference material then it will be really great.
Cheers
Spring Boot and AWS Lambda don't naturally go together IMO.
Lambda is pure code, it does not present itself as a HTTP Server, it is just triggered by one of the other AWS services (API Gateway, CloudWatch, S3, DynamoDB, Kinesis, SDK, etc.). The handler receives a JSON request from the calling service, and processes the request. Here is an example.
API Gateway does much of what Spring Boot provides for you. API Gateway is always online waiting for HTTP requests to come in, for which you only pay for incoming requests, you do not pay for idle (the definition of serverless IMO).
Once a request comes in, API Gateway wraps the request payload with some additional environmental data and sends it to your Lambda handler, which processes the request and returns some response to the client.
Saying that, if you don't want to restructure your service, there are a couple of options open to you:
Wrap into a Docker image and use an AWS Container Service, either using ECS or ElasticBeanstalk, neither of these are considered to be serverless.
I have not tried this, but according to AWS:
You can use the aws-serverless-java-container library to run a Spring Boot application in AWS Lambda. You can use the library within your Lambda handler to load your Spring Boot application and proxy events to it.
See links to Convert your SpringBoot project and Deploy it to AWS Lambda.
Hope this helps.