I would like to build metrics in DataDog that are related to CloudFormation stack deployments, like deployment frequency, etc. However, I don't see any way of accessing CloudFormation events other than manually getting them by ways of, for example, using CLI describe-stack-events command, I don't think they are "propagated" anywhere so to be, for example, exported to DataDog via CloudWatch integration.
The only solutions I came up are convoluted in my opinion, like:
Creating a tool which scans the stacks/events periodically and pushes them to DD
Creating a Lambda function within the stack which will emit an event which can be captured by DD
Using an SNS topic as an intermediary
Is there a simpler way to "see" events like CREATE_COMPLETE or UPDATE_COMPLETE stack events in DataDog?
Related
so im trying to run Terraform through CodePipeline. I need to manage a fleet of clusters. It seems CodePipeline is one of the good ways to trigger certain pipelines on some conditions.
I have a very simple requirement - i want to see the terraform execution in real time. i want to expose the CodePipeline run in a way that i can stream this. Is this where EventBridge is used. I tried to look at an EventBridge example here - https://medium.com/hackernoon/monitoring-ci-cd-pipelines-with-amazon-eventbridge-32177e2f2c3e - but it doesnt seem to be streaming run output in real time.
Which event or hook to should i attach to? And is CodePipeline even the right thing to use here ?
Which event or hook to should I attach to?
You're looking at the wrong AWS service. EventBridge is not for streaming log output. It is for discrete events, not a stream.
Your CodePipeline would be using a CodeBuild task to execute Terraform. Your CodeBuild task will be configured to log to AWS CloudWatch Logs. You can view the CloudWatch Logs output in the AWS CloudWatch web console, with the option to poll for new log output.
You can also do the same in a command line console with the aws logs tail command, documented here.
To do the same thing in your own code you would have to write your code to poll the CloudWatch Logs API in an loop.
And is CodePipeline even the right thing to use here?
Yes absolutely
I want to build an end to end automated system which consists of the following steps:
Getting data from source to landing bucket AWS S3 using AWS Lambda
Running some transformation job using AWS Lambda and storing in processed bucket of AWS S3
Running Redshift copy command using AWS Lambda to push the transformed/processed data from AWS S3 to AWS Redshift
From the above points, I've completed pulling data, transforming data and running manual copy command from a Redshift using a SQL query tool.
Doubts:
I've heard AWS CloudWatch can be used to schedule/automate things but never worked on it. So, if I want to achieve the steps above in a streamlined fashion, how to go about it?
Should I use Lambda to trigger copy and insert statements? Or are there better AWS services to do the same?
Any other suggestion on other AWS Services and of the likes are most welcome.
Constraint: Want as many tasks as possible to be serverless (except for semantic layer, Redshift).
CloudWatch:
Your options here are either to use CloudWatch Alarms or Events.
With alarms, you can respond to any metric of your system (eg CPU utilization, Disk IOPS, count of Lambda invocations etc) when it crosses some threshold, and when this alarm is triggered, invoke a lambda function (or send SNS notification etc) to perform a task.
With events you can use either a cron expression or some AWS service event (eg EC2 instance state change, SNS notification etc) to then trigger another service (eg Lambda), so you could for example run some kind of clean-up operation via lambda on a regular schedule, or create a snapshot of an EBS volume when its instance is shut down.
Lambda itself is a very powerful tool, and should allow you to program a decent copy/insert function in a language you are familiar with. AWS has several GitHub repos with lots of examples too, see for example the serverless examples and many samples. There may be other services which could work for you in your specific case, but part of Lambda's power is its flexibility.
I am currently in charge of adding CloudWatch integration to an already made Cloud Formation stack.
We create the stacks through CLI, but at the moment we add CloudWatch manually afterwards.
What i need is to automatically activate CloudWatch for instances and monitor CPU, hdd and so on through the use of CloudFormation templates.
Thanks in advance!
My suggestion is that you don't add new CloudWatch items to the existing CloudFormation stack. Instead, create a CF template with the appropriate metrics and deploy from this template for each instance you want to monitor.
From there, I suggest you create an AWS Lambda function that will receive an Instance Id as input and will deploy a CloudFormation stack against the instance. You should enable CloudTrail on your account and create a Rule to match any RunInstances event on the account and trigger the Lambda function.
Keep in mind the default limit for CloudFormation stacks is 200. You might need to request an increase depending on your use case.
With CloudWatch you can monitor applications running on AWS. Is it also possible to monitor an external service?
For example, I have a REST API and I want to get notified once that API is not accessible anymore. Does AWS offer you a monitoring tool for that purpose?
Not Cloudwatch just by itself, but you can use a combination of Cloudwatch and Lambdas to do what you're asking. You can use cloudwatch events to run lambdas on a schedule, something like once every 5 mins.
CloudwatchEvents -> HealthCheck Lambda -> Cloudwatch Custom Metrics
Your lambda can then ping the API you're monitoring the health of, and either send its status to cloudwatch as a custom metric; or potentially if your lambda throws an error when the API fails, the lambda error metric which is already in cloudwatch becomes your API failure metric
Once the metric exists in cloudwatch, either as a custom metric or the lambda metric by proxy, you're able to do usual cloudwatch things like alarms and notifications.
Now there is a simple way to monitor external resources - CloudWatch Synthetics. Just create a canary to regularly monitor a website, API or even validate a multi-step UI flow.
Read more in the docs: CloudWatch > Using Synthetic Monitoring
Amazon CloudWatch supports custom metrics generated by your applications and services that you do not run on AWS. In this way, CloudWatch can be an integrated storage and aggregation point, allowing you to monitor all of the metrics that you collect, and track on a single platform.
There might be more than one way to reach your goal by using the AWS CLI, an API/SDK, or the CloudWatch collectd plugin etc. I'd recommend you take a look at these links for more details: link-1, link-2, link-3, link-4
Is it possible to send a SNS notification after the CFT completion in AWS ? Is there any way to get the progress of the launching CFT in AWS.
When create resources using a CF template there is an Advanced section of the Options menu. From there you can set Notification options using SNS and Topics.
When you start the CF process you can also view the status and importantly where the template might have failed.
You cannot specify notification ARNs via a CloudFormation template itself. You can specify them if you use the console to create the stack creation. But you cannot use the console to update the ARNs once the stack has been created. You can, however, use aws-cli to update the stack with notifications ARNs once it has been created, eg:
aws cloudformation update-stack --stack-name stack-name --use-previous-template --notification-arns "arn:aws:sns:us-east-1:${ACCOUNT_ID}:${TOPIC_NAME}"
Replace the variable ${VARIABLE} with the literal values from your account.
There's also knowledge center article from AWS where you can replace ROLLBACK_IN_PROGRESS statement with any other state of CloudFormation to get SNS Notification.
You can trick CloudFormation into sending SNS messages from inside the template:
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/template-custom-resources.html
Custom resources enable you to write custom provisioning logic in
templates that AWS CloudFormation runs anytime you create, update (if
you changed the custom resource), or delete stacks. For example, you
might want to include resources that aren't available as AWS
CloudFormation resource types. You can include those resources by
using custom resources. That way you can still manage all your related
resources in a single stack.
Use the AWS::CloudFormation::CustomResource or Custom::String resource
type to define custom resources in your templates. Custom resources
require one property: the service token, which specifies where AWS
CloudFormation sends requests to, such as an Amazon SNS topic.