I have a general AWS question. I have started using AWS sdk, but looks like if I want to receive events asynchronously from AWS(ex: cloudwatch events), lambda functions is the only way. I want to write a simple application that registers a callback to AWS for events, but i couldn't find a way to do that till now, since i don't want to use lambda, i have been polling from my application. Please, let me know if polling is the only option or if there is a better way to resolve it without polling.
From the documentation:
You can configure the following AWS services as targets for CloudWatch Events:
Amazon EC2 instances
AWS Lambda functions
Streams in Amazon Kinesis Streams
Delivery streams in Amazon Kinesis Firehose
Amazon ECS tasks
SSM Run Command
SSM Automation
Step Functions state machines
Pipelines in AWS CodePipeline
Amazon Inspector assessment templates
Amazon SNS topics
Amazon SQS queues
Built-in targets
The default event bus of another AWS account
That's a lot more than just Lambda, so I'm not sure why you state in your question that Lambda is the only option. The options of Amazon EC2 instances and Amazon SNS topics both provide a method for Amazon to "push" the events to your services, instead of requiring your services to poll.
With cloudwatch events, you can set rules and trigger a number of different targets, including SQS queues which you can poll from your EC2 Instances.
Lambda is certainly a popular endpoint, but based on the docs, there are other targets you can send the events to
Already above answers might also be helpful, but one of the possible options to address your problem could be one of this as well.
You can make use of AWS SNS service to subscribe for the events on AWS resources. And the SNS can publish the events to your application end point. Which is nothing but pub/sub model.
Refer this link http://docs.aws.amazon.com/sns/latest/api/API_Subscribe.html
The end-point could be your http or https based application.
Related
Here are my numbers from AWS usage even though I am not using SQS at all in my application.
I have a lambda service (it's not even in production yet) and it talks to external API + My frontend via API Gateway.
How come my SQS usage is so high?
I don't even have any aps in sqs [I used to when I did some tutorials with sqs, but that was a simple feature]. And my guess that S3 is si high because of the SAM deploy.
How can I check what's causing this traffic?
You presumably do have, or did have, some SQS queues. See Monitoring Amazon SQS queues using CloudWatch.
Amazon SQS and Amazon CloudWatch are integrated so you can use CloudWatch to view and analyze metrics for your Amazon SQS queues. You can view and analyze your queues' metrics from the Amazon SQS console, the CloudWatch console, using the AWS CLI, or using the CloudWatch API. You can also set CloudWatch alarms for Amazon SQS metrics.
Also, see Why are my Amazon SQS charges higher than expected?
I need to send data from 1 ecs container to another. How can I do that? There is AWS EventBridge that allows me to send data from ECS container to EventBridge. But I could not figure out how to send this data to the other ECS container from EventBridge.
P.S. I have node applications running in ECS containers. I am using HTTP API Gateway and Application Load Balancer (ALB)
Answers to questions asked in comments
What kind of data? Text Data
How big is one msg? Small. Just simple objects
Does it have to be real-time or not? No
I need to send data from 1 ecs container to another. How can I do that?
Usually, when you want your microservices to communicate with each other, an SQS is a preferred choice. The use of the SQS allows you to fully de-couple the producer and the consumer of the messages.
In your case, one container would publish messages to the queue, while the second container would pull for the messages on a fixed schedule. For these to work, both containers would need to have permissions in their task executions role to access the SQS and use AWS SDK to publish and receive the message.
There are other choices as well, such as SNS and EventBridge as you noted. However, due to its simplicity, SQS is often the first choice to consider.
I see this question has been asked few times but has not been answered yet. Making another attempt.
What is the basic difference between an S3 event and Cloudwatch events ?
Is one is preferred over the other ?
Appreciate an answer.
Thanks !
S3 Event Notifications are for events that are specific to S3 buckets. S3 Events Notifications can publish events for
New object created
Object removal
Restore object
Reduced Redundancy Storage (RRS) object lost events
Replication events
And it can send notifications to:
SNS topics
SQS queues
Lambda functions
CloudWatch Events, and the associated (preferred, actually) service, Amazon EventBridge, are much broader, and apply to the entire AWS platform. CloudWatch and EventBridge use the same underlying CloudWatch Events API, but EventBridge has more features.
You can use CloudWatch Events/EventBridge to react to any event published by AWS CloudTrail as well as from a very long list of integrated AWS services. These events can also be published on a schedule using a cron-like schedule expression syntax. It can send notifications to more targets as well, including Amazon EC2, Kinesis data streams, ECS tasks, Systems Manager, and much more.
Generally, it's preferable to use EventBridge for anything other than S3. Since EventBridge shares the same underlying API as CloudWatch Events, any change you make to either one will show up in the other. You should use S3 Events for any of the events listed above (see the docs for up an to date list of events).
Is there a service available in AWS that can provide this functionality? This would be used to run recurring backend jobs like sending email blasts, monitoring health, etc.
The question is somewhat unspecific.
For monitoring and scheduling tasks you can use Amazon CloudWatch. Depending on what you want to do this may be or not be a good fit.
There are tutorials on scheduling AWS Lambda and Amazon ECS, but it supports lots of other targets as well. For example you could generate a SNS message and SNS could trigger some HTTP(S)/REST-Call doing your processing or you could write a message to SQS and your application polls that queue for getting triggered.
I need to receive notifications whenver my instance in terminated. I know it can be done by cloudtrail and then using sns and sqs to get email for it, if you receive event of termination.
Is there a simpler way to do that ?
Any solution will is appreciated, but I prefer is doing using boto.
While it is not possible to receive a notification directly from Amazon EC2 when an instance is terminated, there are a couple of ways this could be accomplished:
Auto Scaling can send a notification when an instance managed by Auto Scaling is terminated. See: Configure Your Auto Scaling Group to Send Notifications
AWS Config can also be configured to send a Simple Notification Service (SNS) notification when resources change. This would send many notifications, so you would need to inspect and filter the notifications to find the one(s) indicating an instance termination. See the SNS reference in: Set Up AWS Config Using the Console and Example Amazon SNS Notification and Email from AWS Config.
Amazon Simple Notification Service (SNS) can also push a message to Amazon Queueing Service (SQS), which can be easily polled with the boto python SDK.
Receiving notifications via CloudTrail and CloudWatch Logs is somewhat messier, so I'd recommend the AWS Config method.
Now AWS introduced "rules" Under "Events" in AWS CloudWatch. In your case, you can select EC2 as Event Selector and SNS or SQS as Targets.
https://aws.amazon.com/blogs/aws/new-cloudwatch-events-track-and-respond-to-changes-to-your-aws-resources/
According to the AWS doc: Spot Instance Interruptions, it is possible to pool the instance-metadata in order to get an approximation of the termination time. You can build any custom monitoring solution around that.
> curl http://169.254.169.254/latest/meta-data/spot/instance-action
{"action": "stop", "time": "2017-09-18T08:22:00Z"}
If the instance is not scheduled for termination a http:400 will be returned.