I'm new to this and I'd like to get some ideas in terms of a code that can dynamically tag AWS resources. I'm confuse as to what will trigger the execution of the code that will tag it. Can someone please point me to right resources and sample codes?
You need to monitor CloudTrail events for creation of resources you would like to tag and invoke a Lambda function for the matching events, which tags
the resources accordingly.
CloudWatch Event Rule is setup to monitor :create* API calls via CloudTrail.
This rule triggers the lambda function whenever a matching event found.
The Lambda function fetches the resource identifier and principal information from the event and tags the resources accordingly.
I've devised a solution to tag EC2 resources for governance. It is developed in CDK Python and uses Boto3 to attach tags.
You can further extend this code to cover other resource types or maintain a DynamoDb table to store additional tags per principal
such as Project, Team, Cost Center. You can then simply fetch the tags of a principal and apply them all at once.
You can write lambda functions and use Cloudwatch events to trigger that function which will assign tag to your resources.
You can use AWS nodejs-sdk or boto3 for Python.
Related
Is there any way to rollback all the changes made by an AWS Lambda by using the SDK? E.g. the lambda created and launched by a CloudFormation template creates a bucket via the AWS SDK. In case something fails it would be great to have a 'stack rollback' for the same stack that deployed the lambda as well (and all the resources created by the lambda reverted as well).
Or alternatively: how can I 'remember' from my lambda which resources were created so that I can rollback them and delete them when the lambda is called afterwards with a 'Delete' event?
I'm assuming you mean custom resources, as that's the only way you can run scripts in cloudformation.
Custom resources have a property called pysicalReourceId. You can use it after your create event to provide info over the resource you've created. When updating or deleting the resource, the id is provided to the lambda event so you can use it. A guide can also be found here: https://advancedweb.hu/how-to-use-the-physicalresourceid-for-cloudformation-custom-resources/
If for some reason it's not possible to use the resource ID I'd use tagging. When creating, tag your resources and when deleting, fetch the resources based on their tag and delete them.
I am trying to more effectively manage the resources we create in our AWS accounts and I would like to start by attaching a lambda or many lambdas whenever a resource is created. At a minimum, I need to tag the resources because we simply can't count on people to do it at all, much less correctly.
For example: I can get an event anytime an object is deleted in S3 but that isn't what I want; I want to know when a bucket it created; either through the console of a CFT or the CLI
The closest thing I can see is CloudFormation events man be monitored. We do a lot of stack creation of resources but not always so this isn't good enough. And, in any event, I would need to know all the resources that were created with that stack which the documentation doesn't make clear if I could even get
Can this be done? If so, how?
CloudTrail tracks user activity and API usage and generates CloudTrail Trails. Trails have Data Events.
CloudTrail Data Events can be passed directly to a Lambda function for processing (and/or S3 and/or CloudWatch).
In my own case we store CloudTrail Events in a S3 bucket but also pass them to a Lambda function that applies a billing tag to all new resources created in the account.
GorillaStack Autotag might be a good starting point as a reference for the function.
I am currently in charge of adding CloudWatch integration to an already made Cloud Formation stack.
We create the stacks through CLI, but at the moment we add CloudWatch manually afterwards.
What i need is to automatically activate CloudWatch for instances and monitor CPU, hdd and so on through the use of CloudFormation templates.
Thanks in advance!
My suggestion is that you don't add new CloudWatch items to the existing CloudFormation stack. Instead, create a CF template with the appropriate metrics and deploy from this template for each instance you want to monitor.
From there, I suggest you create an AWS Lambda function that will receive an Instance Id as input and will deploy a CloudFormation stack against the instance. You should enable CloudTrail on your account and create a Rule to match any RunInstances event on the account and trigger the Lambda function.
Keep in mind the default limit for CloudFormation stacks is 200. You might need to request an increase depending on your use case.
I'd like to run some code using Lambda on the event that I create a new EC2 instance. Looking the blueprint config-rule-change-triggered I have the ability to run code depending on various configuration changes, but not when one is created. Is there a way to do what I want? Or have I misunderstood the use case of Lambda?
We had similar requirements couple of days back(Users were supposed to get emails whenever a new instance gets launched)
1) Go to cloudwatch, then select Rules
2) Select service name (its ec2 for your case) then select "Ec2 instance state-change notification"
3) Then select pending in "Specific state" dropdown
4) Click on Add target option and select your lambda function.
That's it, whenever a new instance gets launched, Cloudwatch will trigger your lambda function.
Hope it helps !!
You could do this by inserting code into your EC2 instance launch userdata and have that code explicitly invoke a Lambda function, but that's not the best way to do it.
A better way is to use a combination of CloudTrail and Lambda. If you enable CloudTrail logging (every a/c should have this enabled, all the time, in all regions) then CloudTrail will log to S3 all of the API calls made in your account. You then connect this to Lambda by configuring S3 to publish events to Lambda. Your Lambda function will receive an S3 event, can then retrieve the API logs, find RunInstances API calls, and then do whatever work you need to as a consequence of the new instance being launched.
Some helpful references here and here.
I don't see a notification trigger for instance startup, however what you can do is write a startup script and pass that in via userdata. That startup script would need to download and install the AWS CLI and then authenticate to SNS and publish a message to a pre-configured topic. The startup script would authenticate to SNS and whatever other AWS services are needed via your IAM Role, so you would need to give the IAM Role permission to do whatever you want the script to do. This can be done in the IAM console.
That topic would then have your Lambda function subscribed to it, which would execute. Similar to the below article (though the author is doing something similar for shutdown, not startup).
http://rogueleaderr.com/post/48795010760/how-to-notifyemail-yourself-when-an-ec2-instance
If you are putting the EC2 instances into an autoscale group, I believe there is a trigger that gets fired when the autoscale group launches a new instance, so you could take advantage of that.
I hope that helps.
If you have CloudTrail enabled, then you can have S3 PutObject/TrailBucket trigger a Lambda function. Lambda function parses the object that is passed to it and if it finds RunInstances event, then run your code.
I do the exact same thing to notify certain users when a new instance is launched. With Lambda/Python, it is ~20 lines of code.
Is it possible to send a SNS notification after the CFT completion in AWS ? Is there any way to get the progress of the launching CFT in AWS.
When create resources using a CF template there is an Advanced section of the Options menu. From there you can set Notification options using SNS and Topics.
When you start the CF process you can also view the status and importantly where the template might have failed.
You cannot specify notification ARNs via a CloudFormation template itself. You can specify them if you use the console to create the stack creation. But you cannot use the console to update the ARNs once the stack has been created. You can, however, use aws-cli to update the stack with notifications ARNs once it has been created, eg:
aws cloudformation update-stack --stack-name stack-name --use-previous-template --notification-arns "arn:aws:sns:us-east-1:${ACCOUNT_ID}:${TOPIC_NAME}"
Replace the variable ${VARIABLE} with the literal values from your account.
There's also knowledge center article from AWS where you can replace ROLLBACK_IN_PROGRESS statement with any other state of CloudFormation to get SNS Notification.
You can trick CloudFormation into sending SNS messages from inside the template:
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/template-custom-resources.html
Custom resources enable you to write custom provisioning logic in
templates that AWS CloudFormation runs anytime you create, update (if
you changed the custom resource), or delete stacks. For example, you
might want to include resources that aren't available as AWS
CloudFormation resource types. You can include those resources by
using custom resources. That way you can still manage all your related
resources in a single stack.
Use the AWS::CloudFormation::CustomResource or Custom::String resource
type to define custom resources in your templates. Custom resources
require one property: the service token, which specifies where AWS
CloudFormation sends requests to, such as an Amazon SNS topic.