I'm trying to create a custom metric using CloudFormation. I've followed the example from https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-logs-metricfilter.html. My existing lambda logs to a CloudWatch log group that shows up in CloudWatch as /aws/lambda/my-function-name. Here's my CloudFormation YAML for the metric:
ErrorsLogMetric:
Type: AWS::Logs::MetricFilter
Properties:
LogGroupName: !Sub "/aws/lambda/${AWS::StackName}"
FilterPattern: "[ERROR]"
MetricTransformations:
- MetricValue: "1"
MetricNamespace: "LogMetrics"
MetricName: "MyCustomMetric"
${AWS::StackName} resolves to "my-function-name" when it runs. The CloudFormation script runs successfully and says the metric was created, but when I go to CloudWatch the log group for my lambda, it shows zero filters. What do I need to do differently to cause this custom metric to show up as a filter for my lambda log group when it is created via CloudFormation?
If I hard code the property as LogGroupName: "/aws/lambda/my-function-name" then it works. But I don't want to hard code it since the value of ${AWS::StackName} is dynamic in different use cases.
It seemed to be for me the solution was to add
DependsOn: LambdaLogGroup
where LambdaLogGroup was the resource defined earlier in my template. It threw me off because the log group had already existed for some time, and I was attempting to update the stack and add the metric. Something about that still required the dependency even though the log group already existed and was not updated by CloudFormation when I added the Metric.
Related
I'm trying to replicate an extremely basic manually configured AWS ECS Fargate deployment of a single container using CloudFormation. Looks like I'm almost there; the resulting stack spins up a container I can access. But there are no logs.
I compared my manual task (created via the UI) and the CloudFormation one, and added an identical log configuration to the container definition, but simply changing the log group from /ecs/foo to /ecs/bar:
LogConfiguration:
LogDriver: awslogs
Options:
awslogs-create-group: true
awslogs-group: '/ecs/bar'
awslogs-region: !Ref AWS::Region
awslogs-stream-prefix: 'ecs'
But now the the task fails to start a container. It gives an error like this:
Resourceinitializationerror: failed to validate logger args: create stream has been retried 1 times: failed to create Cloudwatch log group: AccessDeniedException: User: arn:aws:sts::…:assumed-role/ecsTaskExecutionRole/… is not authorized to perform: logs:CreateLogGroup on resource: arn:aws:logs:us-east-1:…:log-group:/ecs/bar:log-stream: because no identity-based policy allows the logs:CreateLogGroup action status code: 400, request id: … : exit status 1
One documentation page mentions this logs:CreateLogGroup permission, and says:
To use the awslogs-create-group option, add logs:CreateLogGroup as an inline IAM policy.
But what I don't understand is how my CloudFormation template differs from the stack manually created via the UI. By looking at the generated template for the manually-created stack, it appears both task definitions indicate the ecsTaskExecutionRole. My CloudFormation template task definition has this:
ExecutionRoleArn: 'arn:aws:iam::…:role/ecsTaskExecutionRole'
How was the manually-created stack able to create the log group, but my standalone from-scratch CloudFormation template could not? Where would I indicate the logs:CreateLogGroup permission? The manually-created stack doesn't seem to indicate any inline policy. (Admittedly for some reason the manually-created task definition doesn't seem to use a CloudFormation stack, so maybe it has some hidden settings I'm not seeing in the UI.)
ecsTaskExecutionRole should be assigned to TaskRoleArn, not to ExecutionRoleArn.
If I want the task to automatically create a log group dynamically using awslogs-create-group, it appears that the correct approach is to have an IAM policy that includes the logs:CreateLogGroup permission, as mentioned at Using the awslogs log driver. (I still don't understand how creating the task definition manually in the UI resulted in the log group getting created.) Another page related to ECS resource initialization errors says I need to "add logs:CreateLogGroup as an inline IAM policy", but no one has been able to provide an example of how to do that in CloudFormation. I'm sure I could figure it out …
However rather than having the service dynamically create a log group, it seems to me better practice to declare and configure the AWS::Logs::LogGroup resource in the CloudFormation template itself. (Thanks to the AWS CloudFormation - Beginner to Advanced (Hands-On Guide) Udemy course for inspiring this approach.) Thus I would declare the log group like this in CloudFormation:
BarLogGroup:
Type: AWS::Logs::LogGroup
DeletionPolicy: Retain
Properties:
LogGroupName: '/ecs/bar'
Then to avoid duplication, reference the log group in the log configuration:
LogConfiguration:
LogDriver: awslogs
Options:
awslogs-group: !Ref BarLogGroup
awslogs-region: !Ref AWS::Region
awslogs-stream-prefix: 'ecs'
This gets around the problem of runtime permissions and dynamic creation altogether. The log group is a resource, too, and since the task depends on it we might as well describe it declaratively with the other resources. This way log group creation depends on permissions of the role creating the stack, not the task itself, which seems more appropriate.
(I've added a deletion policy, assuming you want to keep the logs even if you delete/recreate the stack.)
After making the above changes, my CloudFormation EC Fargate stack now runs and produces logs just like the manually created stack, so this approach is a success.
I'm creating a rule that should fire every time there is a change in status in a SageMaker batch transform job.
I'm using Serverless Framework but to simplify it even further, here's what I did:
The rule, exported from AWS console:
AWSTemplateFormatVersion: '2010-09-09'
Description: >-
CloudFormation template for EventBridge rule
'sagemaker-transform-status-to-CWL'
Resources:
EventRule0:
Type: AWS::Events::Rule
Properties:
EventBusName: default
EventPattern:
source:
- aws.sagemaker
detail-type:
- SageMaker Training Job State Change
Name: sagemaker-transform-status-to-CWL
State: ENABLED
Targets:
- Id: XXX
Arn: >-
arn:aws:logs:us-east-1:XXX:log-group:/aws/events/sagemaker-notifications
Eventually I want this to trigger a step function or a lambda function, but for now I am configuring the target to be CloudWatch with log group 'sagemaker-notifications'
I expect that everytime I run a batch transform job in SageMaker, this will get notified and the log would show up on cloudwatch.
But I'm not getting any logs, so when I tried to PutEvents manually to test it, I was getting this:
Error. NotAuthorizedForSourceException. Not authorized for the source.
It's probably an issue with roles, but I'm not sure which kind of role to configure, where and who should assume it.
Tried going through AWS tutorials, adding permissions to the default event bus, using serverless framework
See some sample event patterns here - https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-events-rule.html#aws-resource-events-rule--examples
Your source should be a custom source, and cannot contain aws. (Reference -https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html)
I would like to perform the following operations in order with CloudFormation.
Start up an EC2 instance.
Give it privileges to access the full internet using security group A.
Download particular versions of Java and Python
Remove its internet privileges by removing security group A and adding a security group B.
I observe that there is a DependsOn attribute for specifying the order in which to create resources, but I was unable to find a feature that would allow me to update the security groups on the same EC2 instance twice over the course of creating a stack.
Is this possible with CloudFormation?
Not in CloudFormation natively, but you could launch the EC2 instance with a configured userdata script that itself downloads Java/Python and the awscli, as necessary, and then uses the awscli to switch security groups for the current EC2 instance.
However, if all you need is Java and Python pre-loaded then why not simply create an AMI with them already installed and launch from that AMI?
The best way out is to utilise a Cloudformation custom resource here. You can create a lambda function that does exactly what you need. This lambda function can then be called as a custom resource function in the cloud formation template.
You can pass your new security group ID and instance ID to the lambda function and code the lambda function to use AWS SDK and do the modifications that you need.
I have leveraged it to post an update to my web server about the progress of the cloud formation template. Below is the sample code of the template.
EC2InstanceProfile:
Type: AWS::IAM::InstanceProfile
Properties:
Path: /
Roles: [!Ref 'EC2Role']
MarkInstanceProfileComplete:
Type: 'Custom::EC2InstanceProfileDone'
Version: '1.0'
DependsOn: EC2InstanceProfile
Properties:
ServiceToken: !Ref CustomResourceArn
HostURL: !Ref Host
LoginType: !Ref LoginType
SecretId: !Ref SecretId
WorkspaceId: !Ref WorkspaceId
Event: 2
Total: 3
Here the resource MarkInstanceProfileComplete is a custom resource that calls a Lambda function. It takes the event count and total count as input and processes them to calculate percentage progress. Based on that it sends out a request to my web server. For all we care, this Lambda function can do potentially anything you want it to do.
I have a lambda which has a log group, say LG-1, for which retention is set to Never Expire (default). I need to change this Never Expire to 1 month. I am doing this using CloudFormation. As the log group already exists, when I am trying to deploy my lambda again with the changes in template as :
LambdaFunctionLogGroup:
Type: 'AWS::Logs::LogGroup'
DependsOn: MyLambda
Properties:
RetentionInDays: 30
LogGroupName: !Join
- ''
- - /aws/lambda/
- !Ref MyLambda
the update is failing with error :
[LogGroup Name] already exists.
One possible solution is to delete the log group and then again create it with new changes as shown above which works perfectly well.
But I need to do it without deleting the log group as it will result in the deletion of all the previous logs that I have.
Is there any workaround which is possible ?
#ttulka answered:
".. it is impossible to manipulate resources from CF which already exist out of the stack."
But actually the problem is more general than that and applies to resources created inside of the stack. It has to do with AWS CloudFormation resource "Replacement policy". For some resources the way CloudFormation "updates" the resource is to create a new resource, then delete the old resource (this is called the "Replacement" update policy). This means there is a period of time where you've got two resources of the same type with many of the same properties existing at the same time. But if a certain resource property has to be unique, the two resource can't exist at the same time if they have the same value for this property, so ... CloudFormation blows up.
AWS::Logs::LogGroup.LogGroupName property is one such property. AWS::CloudWatch::Alarm.AlarmName is another example.
A work around is to unset the name so that a random name is used, perform an update, then set the name back to it's predictable fixed value and update again.
Rant: It's an annoying problem that really shouldn't exist. I.e. AWS CF should be smart enough to not have to use this weird clunky resource replacement implementation. But ... that's AWS CF for you ...
I think it is impossible to manipulate resources from CF which already exist out of the stack.
One workaround would be to change the name of the Lambda like my-lambda-v2 to keep the old log group together with the new one.
After one month you can delete the old one.
Use customresource Backed lambda within your cloudformation template. The custom resource would be triggered automatically the first time and update your retention policy of the existing log group. If you need it you custom resource lambda to be triggered every time, then use a templating engine like jinja2.
import boto3
client = boto3.client('logs')
response = client.put_retention_policy(
logGroupName='string',
retentionInDays=123
)
You can basically make your CF template do (almost) anything you want using Custom Resource
More information (Boto3, you can find corresponding SDK for the language you use) - https://boto3.amazonaws.com/v1/documentation/api/1.9.42/reference/services/logs.html#CloudWatchLogs.Client.put_retention_policy
EDIT: Within the CloudFormation Template, it would look something like the following:
LogRetentionSetFunction:
Type: AWS::Serverless::Function
Properties:
CodeUri: src
Handler: set_retention_period.handler
Role: !GetAtt LambdaRole.Arn
DeploymentPreference:
Type: AllAtOnce
PermissionForLogRetentionSetup:
Type: AWS::Lambda::Permission
Properties:
Action: lambda:invokeFunction
FunctionName:
Fn::GetAtt: [ LogRetentionSetFunction, Arn ]
Principal: lambda.amazonaws.com
InvokeLambdaFunctionToSetLogRetention:
DependsOn: [PermissionForLogRetentionSetup]
Type: Custom::SetLogRetention
Properties:
ServiceToken: !GetAtt LogRetentionSetFunction.Arn
StackName: !Ref AWS::StackName
AnyVariable: "Choose whatever you want to send"
Tags:
'owner': !Ref owner
'task': !Ref task
The lambda function would have the code which sets up the log retention as per the code which I already specified before.
For more information, please google "custom resource backed lambda". Also to get you a head start I have added the ink below:
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/template-custom-resources.html
In the console it's like super easy. But I just couldn't find how to do in the YAML template for CloudFormation?
The AWS::Logs::LogGroup - AWS CloudFormation documentation shows:
myLogGroup:
Type: "AWS::Logs::LogGroup"
Properties:
RetentionInDays: 7
RetentionInDays
The number of days log events are kept in CloudWatch Logs. When a log event expires, CloudWatch Logs automatically deletes it.