AWS CodeBuild has an option in the UI to upload build output logs to CloudWatch and/or publish the logs to S3 but I do not see an option in Terraform to enable this feature through the "aws_codebuild_project" resource.
Here is a link to the Terraform docs:
https://www.terraform.io/docs/providers/aws/r/codebuild_project.html
Is there another way to possibly enable this feature via Terraform?
I can create a CloudWatch Event to capture CodeBuild events but that doesn't give me the full logs.
So even thought the Terraform documentation does not specify support for enabling CloudWatch logs for CodeBuild jobs, if you create a CodeBuild job with the AWS CodeBuild Project resource it will by default enable that checkbox for you, at least as of the date of this answer.
The Cloudwatch log Log Group will be created with the following format: aws/codebuild/codebuild_project_name
There is an open PR for a Github issue to enable CodeBuild CloudWatch logs, so hopefully we will be able to use the following config soon:
logs_config {
cloudwatch_logs {
status = "ENABLED|DISABLED"
group_name = "..."
stream_name = "..."
}
}
Related
I am using an AWS Managed Airflow to orchestrate a data pipeline. The problem that I am facing now is that logs are not being printed in the airflow log not on cloud watch.
I am getting the following error in the log of Air flow:
*** Reading remote log from Cloudwatch log_group: airflow-Task log_stream: 2021-08-26T08_08_24+00_00/1.log.
Could not read remote logs from log_group: airflow-Task log_stream: 2021-08-26T08_08_24+00_00/1.log.
AWS has a troubleshooting guide for this specific case.
*** Reading remote log from Cloudwatch log_group: airflow-{environmentName}-Task log_stream: {DAG_ID}/{TASK_ID}/{time}/{n}.log.Could not read remote logs from log_group: airflow-{environmentName}-Task log_stream: {DAG_ID}/{TASK_ID}/{time}/{n}.log.
https://docs.aws.amazon.com/mwaa/latest/userguide/t-cloudwatch-cloudtrail-logs.html#t-task-logs
I would check the following:
Verify that you enabled task logs at the INFO level for your environment.
Check if the MWAA Execution role has CloudWatch read access policy attached.
Eventually try adding apache-airflow[amazon] provider to the MWAA Requirements file.
It was an issue related to role and policies.
MWAA Execution role didn't have the required CloudWatch access policy attached to it.
In my case the issue was because of encryption.
Check if your MWAA uses the same KMS key as your CloudWatch Log Groups.
If MWAA is created with custom KMS key this key is also used for CloudWatch Log Groups encryption.
When you delete and re-create MWAA (with new KMS key) Log Groups are not touched and remain encrypted with old key. And they are not accessible for MWAA any more.
Two ways to fix it:
Delete Log Groups, create new Log Groups with the same name and using KMS key which is used by MWAA.
Don't use KMS key when you create MWAA. In this case, AWS managed key aws/airflow will be used automatically. It should help to avoid dependency on particular KMS key.
In My case, this issue happened because the KMS used for encrypting the S3 bucket and MWAA was missing following in the Key policy
"kms:GenerateDataKey*",
for principal
AWS": "arn:aws:iam:::root"
I am not able to delete these subscriptions attached to the CloudWatch Logs Groups.
These subscriptions are created by CloudFormation stack via Serverless Framework. However, when I finished testing and deployed to the template, there was a permission error during the cleanup. Hence, these subscriptions became dangled and I am not able to locate it.
Tried with CLI and seems no relevant info regarding that.
$ aws logs describe-log-groups --log-group-name-prefix yyy
{
"logGroups": [
{
"logGroupName": "yyy",
"creationTime": 1555604143719,
"retentionInDays": 1,
"metricFilterCount": 0,
"arn": "arn:aws:logs:us-east-1:xxx:log-group:yyy:*",
"storedBytes": 167385869
}
]
}
Select the Log Group using the radio button on the left of the Log Group name. Then click Actions, Remove Subscription Filter.
Via CLI is listed in AWS document => This link
Via Console UI -> This capture
As you created the subscription with cloudformation stack via serverless, manually removing the subscription filter as jarmod is not a best practice.
What you should do is remove the cloudwatchLog event from the lambda functions and deploy, it should remove the subscriptions.
I am exploring CodeStar using a basic project created with the Python 3.7 Lambda template following the Serverless Project Tutorial in the AWS CodeStar documentation:
https://docs.aws.amazon.com/codestar/latest/userguide/sam-tutorial.html
My build and deploy are successful. However a see a warning in my CloudFormation event log:
The IAM user doesn't allow CloudFormation to call lambda:GetAlias, this could result in formulating a appspec file with stale CurrentVersion for CodeDeploy deployment. Please fix it to avoid any possible CodeDeploy deployment failures.
I am just using the AWS resources created automatically by the CodeStar console.
What do I do to fix this warning?
Details
The CodeDeploy step in the CodePipeline deploys the lambda function by updating a CloudFormation stack named: awscodestar-<codestar project name>-lambda.
When I looked in the event log for this stack, I noticed the above warning for the resource named HelloWorldAliaslive
To fix this, add the lambda:GetAlias permission to the inline policy associated with the IAM role named CodeStarWorker-<project>-CloudFormation
Open the AWS Console for CodeStar
Click Project in the left navbar
Find the Project Resources section. One of the AWS IAM resources will have a name CodeStarWorker-<project>-CloudFormation. Click the link in the ARN column of the table to open the role in IAM.
Locate the inline policy named CodeStarWorkerCloudFormationRolePolicy and click the Edit button.
Add the "lambda:GetAlias" action to this policy.
This policy is created automatically by CodeStar. In my account, the policy included several Statements. I chose to add the "lambda:GetAlias" action to the statement which already had "lambda:CreateAlias" action.
After making this change, the warning no longer appeared in my CloudFormation event logs.
I have a CodePipeline setup. But now I want to only have my Pipeline run when I trigger it manually, is that possible?
This worked for me:
Set up a regular pipeline with Github webhook set.
This will create and register an internal aws webhook.
Then simply deregister and remove the webhooks like it is written in the aws docs via aws-cli.
Adjust the following to your region, mine is us-east-2:
Find out the name of the webhook:
aws codepipeline list-webhooks
De-register:
aws codepipeline deregister-webhook-with-third-party --webhook-name <webhook-name>
Delete:
aws codepipeline delete-webhook --name <webhook-name>
Note:
It's necessary to have the source step configured to use the Github webhook.
Now you can trigger your pipeline manually via AWS Console or via aws-cli with:
aws codepipeline start-pipeline-execution --name <pipeline-name>
You need configure your CodePipeline to be triggered by CloudWatch event. Once you've done that, an event rule under CloudWatch named "codepipeline-{repository_name}" will be created.
Disabling this rule will stop CodePipeline from been triggered by CodeCommit. When you need a build, just click 'Release Change' button.
You can use CloudWatch Events, to never trigger your pipeline:
https://docs.aws.amazon.com/codepipeline/latest/userguide/pipelines-trigger-source-schedule-console.html
Set up a cronjob that will never execute, and then you can manually trigger the pipeline whenever you want by clicking on the 'Release change' button.
I haven't tested, but I think it should work.
I'm creating a logs aggregator lambda to send Cloudwatch logs to a private log analysis service. Given the number of resources used by my employer, it was decided to create a subscription lambda that handles log group subscription to the aggregator.
The solution works fine, but it requires to manually search a resource's log group via amazon console and then invoke the subscription lambda with it.
My question:
Is there a way to, given a resource arn, find which log group is mapped to it? Since I'm using Cloudformation to create resources it is easy to export a resource's arn.
UPDATE
To present an example:
Let's say I have the following arn:
arn:aws:appsync:<REGION>:<ACCOUNTID>apis/z3pihpr4gfbzhflthkyjjh6yvu
which is an Appsync GraphQL API.
What I want it a method (using te API or some automated solution) to get the Cloudwatch log group of that resource.
You can try the describe-log-groups command. It is available on the cli, must also be there on the API.
To get the names of the log groups you can go with:
aws logs describe-log-groups --query 'logGroups[*].logGroupName' --log-group-name-prefix '/aws/appsync/[name-of-the-resource]'
Output will look like this:
[
"/aws/appsync/[name-of-your-resource]"
]