I am using an AWS Managed Airflow to orchestrate a data pipeline. The problem that I am facing now is that logs are not being printed in the airflow log not on cloud watch.
I am getting the following error in the log of Air flow:
*** Reading remote log from Cloudwatch log_group: airflow-Task log_stream: 2021-08-26T08_08_24+00_00/1.log.
Could not read remote logs from log_group: airflow-Task log_stream: 2021-08-26T08_08_24+00_00/1.log.
AWS has a troubleshooting guide for this specific case.
*** Reading remote log from Cloudwatch log_group: airflow-{environmentName}-Task log_stream: {DAG_ID}/{TASK_ID}/{time}/{n}.log.Could not read remote logs from log_group: airflow-{environmentName}-Task log_stream: {DAG_ID}/{TASK_ID}/{time}/{n}.log.
https://docs.aws.amazon.com/mwaa/latest/userguide/t-cloudwatch-cloudtrail-logs.html#t-task-logs
I would check the following:
Verify that you enabled task logs at the INFO level for your environment.
Check if the MWAA Execution role has CloudWatch read access policy attached.
Eventually try adding apache-airflow[amazon] provider to the MWAA Requirements file.
It was an issue related to role and policies.
MWAA Execution role didn't have the required CloudWatch access policy attached to it.
In my case the issue was because of encryption.
Check if your MWAA uses the same KMS key as your CloudWatch Log Groups.
If MWAA is created with custom KMS key this key is also used for CloudWatch Log Groups encryption.
When you delete and re-create MWAA (with new KMS key) Log Groups are not touched and remain encrypted with old key. And they are not accessible for MWAA any more.
Two ways to fix it:
Delete Log Groups, create new Log Groups with the same name and using KMS key which is used by MWAA.
Don't use KMS key when you create MWAA. In this case, AWS managed key aws/airflow will be used automatically. It should help to avoid dependency on particular KMS key.
In My case, this issue happened because the KMS used for encrypting the S3 bucket and MWAA was missing following in the Key policy
"kms:GenerateDataKey*",
for principal
AWS": "arn:aws:iam:::root"
Related
I am trying to register a respository on AWS S3 to store ElasticSearch snapshots.
I am following guide and ran the very first command listed in the doc.
But I am getting the error Access Denied while executing that command.
The role that is being used to perform operations on S3 is the AmazonEKSNodeRole.
I have assigned the appropriate permissions to the role to perform operations on the S3 bucket.
Also, here is another doc which suggests to use kibana for ElasticSearch version > 7.2 but I am doing the same via cURL requests.
Below is trust Policy of the role through which I am making the request to register repository in the S3 bucket.
Also, below are the screenshots of the permissions of the trusting and trusted accounts respectively -
I am trying to sync two S3 buckets in different accounts. I have successfully configured the locations and created a task. However, when I run the task I get a Unable to connect to S3 endpoint error. Can anyone help?
This could have been related to the datasync's IAM role's policy (datasync IAM role) not having permission to the target S3 bucket
verify your policy and trust relationship using the below documentation
https://docs.aws.amazon.com/datasync/latest/userguide/using-identity-based-policies.html
Also turn on cloudwatch logs (like shown in the image) and view detailed log in cloudwatch. If it is permission related, add the missing policy in the Datasync role.
While creating AWS EMR cluster, always i get the issue- Service role EMR_DefaultRole has insufficient EC2 permissions
And the cluster terminates automatically, have even done steps as per aws documentation of recreating emr specific roles, but no progress please guide how to resolve the issue- Service role EMR_DefaultRole has insufficient EC2 permissions
EMR needs two roles to start the cluster 1) EC2 Instance profile role 2)EMR Service role. The service role should have enough permissions to provision new resources to start the cluster, EC2 instances, their network etc. There could be many reasons for this common error:
Verify the resources and their actions. Refer https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-iam-role.html.
Check if you are passing the tag that signifies if cluster needs to use emr managed policy.
{
"Key": "for-use-with-amazon-emr-managed-policies",
"Value": "true"
}
At last try to find out the exact reason from cloud trail. Go to aws>cloud trail. From the event history configuration enable the error code so that you can see the exact error. If you find the error code something like 'You are not authorized to perform this operation. Encoded authorization failure message'. Then open the event history details, pick up the encrypted error message and decrypt using aws cli
aws sts decode-authorization-message message. This will show you the complete role details, event, resources, action. Compare it with AWS IAM permissions and you can find out the missing permission or parameter that you need to pass while creating the job flow.
I am unable to read or write into a Default AWS KMS encrypted bucket without using the following configuration on my Qubole cluster
fs.s3a.server-side-encryption-algorithm=SSE-KMS
fs.s3a.server-side-encryption.key=<key>
But if I enable this configuration, it's only writing encrypted data into all the other buckets (which do not have encryption enabled) as well.
Is there a work around for this?
You can try the followings:
Remove the following from cluster overrides:
fs.s3a.server-side-encryption-algorithm=SSE-KMS
fs.s3a.server-side-encryption.key=<key>
Reference:
https://docs.qubole.com/en/latest/admin-guide/enabling-encryption/enabling-sse/enabling-sse-kms.html
Use default encryption as mentioned here:
https://docs.aws.amazon.com/AmazonS3/latest/user-guide/default-bucket-encryption.html
Restart the cluster
Ideally after following the above steps, you should be able to achieve your goal.
I recently enabled default ebs encryption as mentioned here: https://aws.amazon.com/blogs/aws/new-opt-in-to-default-encryption-for-new-ebs-volumes/. Afterwards, when attempting to launch a beanstalk instance, I get a generic 'ClientError' and the instance immediately terminates. If I disabled default encryption it works fine.
Does anyone know what changes are required to get beanstalk to work with a customer managed encryption key? I suspected it was a permissions issue so I temporarily gave the beanstalk roles full admin access but that did not solve the issue. Is there something else I am missing?
I saw this related question but it was before default EBS encryption was released and I was hoping to avoid having to copy and encrypt the AMI manually...
If you are using a custom CMK, you have to update the key policy and assign permissions explicitly. For EBS encryption, a principal usually requires the following permissions:
kms:CreateGrant
kms:Encrypt
kms:Decrypt
kms:ReEncrypt*
kms:GenerateDataKey*
kms:DescribeKey
The best way to troubleshoot key permission issues is to check the Cloudtrail event history. Filter the events by event source and check if there is any "access denied" error.
Filter: Event source: kms.amazonaws.com
You can see which action is denied here and adjust the key policy accordingly. "User name" field in the event gives you a hint to determine the ARN of the principal to use in the policy.
In your case, it is very likely that one of the service-linked roles requires permissions to access the KMS key. There is a good explanation for key permissions here for auto-scaling service-linked role.