We plan to run a java application on Elastic BeanStalk. It is noting but a file retriever, processor, transformer and a mapper. It would retrieve the file from S3 and map it to an RDS DB. Question is how do I trigger this application running on beanstalk on file arrival on S3 bucket and also on demand ?
Thanks and regards,
Kunal
You can send an event to SNS (notification topic) on S3 file upload.
Then I see two options:
To the SNS you can hook up a lambda or http invocation, however in that case you will need to handle failures or availability issues.
If your application is running on EC2 I'd suggest to send the upload event to SQS (queue service) and your application can poll for queue messages
also on demand
for that you need to expose a an interfate or service from your application. You did not specify what your application is, so it's really up to you to define what 'on demand' is
Related
I have developed a SAM app that takes incoming emails submissions, listens to specific SQS queues for the incoming data type, parses the payload via the correct python Lambda worker function and submits data down the line into another SQS queue for further action. I have deployed this stack from PyCharm using a SAM YAML template.
I need to improve the app with better disaster recovery options for fault tolerance. I don't believe I can improve the inbound email receiving function through SES, so that will be a single point of failure (if SES fails in us-east-1, my app fails). However, I'd like for the SQS messages to be delivered to the Parser Worker functions in multiple regions.
Scenario:
email1 comes into SES and is stored in the correct S3 key
S3 triggers event notification to Parser1 SQS queue
Parser1 SQS message is picked up in a load-balanced manner by one of multiple Parser1 Worker functions deployed in multiple Regions (us-east1a, us-east-2a, etc) thus providing disaster recovery and fault tolerance
Is there a way to distribute the SQS message in a load-balanced type of scenario to multiple Lambdas deployed in multiple regions so that I don't have the single point of failure in one stack in one region?
I have the app running in production in its single-stack deployment. I am concerned about redundancy and single points of failure. I am not sure how to improve this situation, so have not tried anything yet.
I've got rather rare requirement to deliver SNS topic message to all micro service instances.
Basically it's kind of notification that related data had changed
and all micro service instances should reload their internals from data source.
We are using TerraForm to create our infrastructure, with Kong api gateway.
Micro Service instances could be created 'on the fly' as system load is increased,
so subscriptions to topic could not be created in TerraForm stage.
Micro Service is standard SpringBoot app.
My first approach is:
micro service is exposing http endpoint that can be subscribed to SNS topic
micro service on start will subscribe itself (above endpoint) to required SNS topic, unsubscribe on service shutdown.
My problem is to determine individual micro service instances urls, that can be used in subscription process.
Alternative approach would be to use SQS, create SQS queue per micro srv instance (subscribe it to sns).
Maybe I'm doing it wrong on conceptual level ?
Maybe different architecture approach is required ?
It might be easier for the microservices to check an object in Amazon S3 to "pull" the configuration updates (or at least call HeadObject to check if the configuration has changed) rather than trying to "push" the configuration update to all servers.
Or, use AWS Systems Manager Parameter Store and have the servers cache the credentials for a period (eg 5 minutes) so they aren't always checking the configuration.
Kinda old right now but here is my solution:
create SNS, subscribe with SQS, publish the SQS to redis pub/sub, subscribe to pub/sub
now all your instances will get the event.
I have a general AWS question. I have started using AWS sdk, but looks like if I want to receive events asynchronously from AWS(ex: cloudwatch events), lambda functions is the only way. I want to write a simple application that registers a callback to AWS for events, but i couldn't find a way to do that till now, since i don't want to use lambda, i have been polling from my application. Please, let me know if polling is the only option or if there is a better way to resolve it without polling.
From the documentation:
You can configure the following AWS services as targets for CloudWatch Events:
Amazon EC2 instances
AWS Lambda functions
Streams in Amazon Kinesis Streams
Delivery streams in Amazon Kinesis Firehose
Amazon ECS tasks
SSM Run Command
SSM Automation
Step Functions state machines
Pipelines in AWS CodePipeline
Amazon Inspector assessment templates
Amazon SNS topics
Amazon SQS queues
Built-in targets
The default event bus of another AWS account
That's a lot more than just Lambda, so I'm not sure why you state in your question that Lambda is the only option. The options of Amazon EC2 instances and Amazon SNS topics both provide a method for Amazon to "push" the events to your services, instead of requiring your services to poll.
With cloudwatch events, you can set rules and trigger a number of different targets, including SQS queues which you can poll from your EC2 Instances.
Lambda is certainly a popular endpoint, but based on the docs, there are other targets you can send the events to
Already above answers might also be helpful, but one of the possible options to address your problem could be one of this as well.
You can make use of AWS SNS service to subscribe for the events on AWS resources. And the SNS can publish the events to your application end point. Which is nothing but pub/sub model.
Refer this link http://docs.aws.amazon.com/sns/latest/api/API_Subscribe.html
The end-point could be your http or https based application.
For my web app I will need a separate instance of EC2 to process CPU-intensive things, and things that can be queued so they don't encumber the web serving instance, like image resizing, sending email....
When you create an AWS Elastic Beanstalk instance it asks you to choose between a "web" or "worker" environment. From my understanding it's in the worker environment that I will process those kind of tasks.
What's the role of SQS in this context? I read it is only about sending "messages" and this kind of stuff, but how will I get my image resized with a "message"?
Should I create specific, distinct code for the worker instance, to handle image resizing, and then use SQS to order it to process the image? Then can SQS pass image from web instance, to worker instance? I completely miss the main concept.
A queuing service (such as Amazon SQS) is used to store messages for later retrieval. Think of it like a TODO list -- you add items to the queue, and later you retrieve an item from the queue and take action on the item.
For example, let's say users upload images to a website and you wish to generate thumbnails from those images. Your website will store the image in Amazon S3, then push a message into an SQS queue. The message would include a reference to the image in S3 and details about the user.
Then, your Elastic Beanstalk worker will request a message from the queue and process the image. It would retrieve the image from S3, resize it, store it in an S3 bucket, then perhaps email the user to say that the job is finished. The worker then exits, and Elastic Beanstalk will trigger a new worker to read the next message from the queue and do it all again.
So, yes -- you will create the worker code. Elastic Beanstalk will trigger the worker with the SQS message. SQS itself doesn't trigger anything -- it is actually Elastic Beanstalk that retrieves the message and runs the worker.
See: Elastic Beanstalk worker environments
I need to receive notifications whenver my instance in terminated. I know it can be done by cloudtrail and then using sns and sqs to get email for it, if you receive event of termination.
Is there a simpler way to do that ?
Any solution will is appreciated, but I prefer is doing using boto.
While it is not possible to receive a notification directly from Amazon EC2 when an instance is terminated, there are a couple of ways this could be accomplished:
Auto Scaling can send a notification when an instance managed by Auto Scaling is terminated. See: Configure Your Auto Scaling Group to Send Notifications
AWS Config can also be configured to send a Simple Notification Service (SNS) notification when resources change. This would send many notifications, so you would need to inspect and filter the notifications to find the one(s) indicating an instance termination. See the SNS reference in: Set Up AWS Config Using the Console and Example Amazon SNS Notification and Email from AWS Config.
Amazon Simple Notification Service (SNS) can also push a message to Amazon Queueing Service (SQS), which can be easily polled with the boto python SDK.
Receiving notifications via CloudTrail and CloudWatch Logs is somewhat messier, so I'd recommend the AWS Config method.
Now AWS introduced "rules" Under "Events" in AWS CloudWatch. In your case, you can select EC2 as Event Selector and SNS or SQS as Targets.
https://aws.amazon.com/blogs/aws/new-cloudwatch-events-track-and-respond-to-changes-to-your-aws-resources/
According to the AWS doc: Spot Instance Interruptions, it is possible to pool the instance-metadata in order to get an approximation of the termination time. You can build any custom monitoring solution around that.
> curl http://169.254.169.254/latest/meta-data/spot/instance-action
{"action": "stop", "time": "2017-09-18T08:22:00Z"}
If the instance is not scheduled for termination a http:400 will be returned.