Getting this error while trying to connect grafana to cloudwatch in opt-in ap-east-1 region. I'm using access & secret key with Assume Role ARN witch work fine in other regions.
Found similar issue https://github.com/grafana/grafana/issues/48267#issuecomment-1119592852
but setting AWS_STS_REGIONAL_ENDPOINTS=regional env var on grafana server does not help.
May be i'm doing something wrong, please any suggestions.
Related
I am receiving the following errors in the EC2 CloudWatch Agent logs, /var/logs/awslogs.log:
I verified the EC2 has a role:
And the role has the correct policies:
I have set the correct region in /etc/awslogs/awscli.conf:
I noticed that running aws configure list in the EC2 gives this:
Is this incorrect? Should it list the profile (EC2_Cloudwatch_Profile) there?
I was using terraform and reprovisioning by doing:
terraform destroy && terraform apply
Looks like due to IAM being a global service it is "eventually consistent" and not "immediately consistent", when the profile instance was destroyed, the terraform apply began too quickly. Despite the "destroy" being complete, the arn for the previous profile instance was still there, and was re-used. However, the ID changed to a new ID.
Replacing the EC2 would bring it up to speed with the correct ID. However, my solution is to just wait longer between terraform destroy and apply.
I am using an AWS Managed Airflow to orchestrate a data pipeline. The problem that I am facing now is that logs are not being printed in the airflow log not on cloud watch.
I am getting the following error in the log of Air flow:
*** Reading remote log from Cloudwatch log_group: airflow-Task log_stream: 2021-08-26T08_08_24+00_00/1.log.
Could not read remote logs from log_group: airflow-Task log_stream: 2021-08-26T08_08_24+00_00/1.log.
AWS has a troubleshooting guide for this specific case.
*** Reading remote log from Cloudwatch log_group: airflow-{environmentName}-Task log_stream: {DAG_ID}/{TASK_ID}/{time}/{n}.log.Could not read remote logs from log_group: airflow-{environmentName}-Task log_stream: {DAG_ID}/{TASK_ID}/{time}/{n}.log.
https://docs.aws.amazon.com/mwaa/latest/userguide/t-cloudwatch-cloudtrail-logs.html#t-task-logs
I would check the following:
Verify that you enabled task logs at the INFO level for your environment.
Check if the MWAA Execution role has CloudWatch read access policy attached.
Eventually try adding apache-airflow[amazon] provider to the MWAA Requirements file.
It was an issue related to role and policies.
MWAA Execution role didn't have the required CloudWatch access policy attached to it.
In my case the issue was because of encryption.
Check if your MWAA uses the same KMS key as your CloudWatch Log Groups.
If MWAA is created with custom KMS key this key is also used for CloudWatch Log Groups encryption.
When you delete and re-create MWAA (with new KMS key) Log Groups are not touched and remain encrypted with old key. And they are not accessible for MWAA any more.
Two ways to fix it:
Delete Log Groups, create new Log Groups with the same name and using KMS key which is used by MWAA.
Don't use KMS key when you create MWAA. In this case, AWS managed key aws/airflow will be used automatically. It should help to avoid dependency on particular KMS key.
In My case, this issue happened because the KMS used for encrypting the S3 bucket and MWAA was missing following in the Key policy
"kms:GenerateDataKey*",
for principal
AWS": "arn:aws:iam:::root"
Version: 5.7.mysql_aurora.2.07.2
i am trying to link lambda function when my RDS get inserted using lambda_async. but was not able to reach it with an error
#63996 - Lambda API returned error: Missing IAM Credentials for specified aws_default_lambda_role
i have followed each and every step carefully but getting above error. :(
i have followed the documentation at
https://docs.amazonaws.cn/en_us/AmazonRDS/latest/AuroraUserGuide/AuroraMySQL.Integrating.Lambda.html
and
Call AWS Lambda from Aurora RDS Stored Procedure Permissions Issue.
please help!!
actually i was placing lambda role in parameter group value. replacing with RDS IAM role solve my issue thanks!!
I found this answer: https://www.reddit.com/r/aws/comments/b2b9k4/new_rds_cluster_management_screen_missing_manage/ on which they sugest to add an IAM role for the cluster.
For that go to the RDS Cluster and under Connectivity & security tab, go to Manage IAM roles and add the desired role.
Note: You may need to enable also IAM DB authentication under Configuration tab.
Note2: Reboot your DB instance to apply changes.
I have k8s clusters on AWS working with ECR and pulling images from all regions. This works fine.
But when I try to pull images from a different account they get "no such host". I followed these instructions to set iam permissions (and the docs). I'm not getting permission denied - I'm getting this:
Failed to pull image "<acc id>.dkr.ecr.ap-outheast-2.amazonaws.com/image:tag":
rpc error: code = Unknown desc = Error response from daemon:
Get https://<acc id>.dkr.ecr.ap-outheast-2.amazonaws.com/v1/_ping:
dial tcp: lookup <acc id>.dkr.ecr.ap-outheast-2.amazonaws.com
on 10.71.0.2:53: no such host
My cluster is running in ap-southeast-1 and the IP 10.71.0.2:53 is the default DNS AWS set for the VPC
I'm trying to wok around this by populating this region's ECR as well. But it seems pretty wrong.
Any idea how to allow ECR to pull from another region?
I think you made simple typo in .dkr.ecr.ap-outheast-2.amazonaws.com/image:tag - that's why you receive no such host from DNS server, just try to replace ap-outheast-2 with ap-southeast-2.
Generally if you set ECR IAM right that should work as ECR is accessible/routable as public service in Internet with limitations based on IAM.
When I try to launch an Elastic Beanstalk environment from the command line, I get this error during the process:
Environment health has transitioned from Pending to Warning. Access
denied while accessing Auto Scaling and Elastic Load Balancing using
role "arn:aws:iam::XXXXXXXXXX:role/aws-elasticbeanstalk-service-role".
Verify the role policy.
When I launch it from the console, I do not get this warning. I have tried to replicate the same AWS console configuration from the CLI , but I still get this error.
Any clue what's going on?
If anyone is still getting a similar error and is still unsure how to solve it
Navigate to IAM in the AWS Console.
Navigate to the aws-elasticbeanstalk-service-role (under Access Management > Roles).
Attach the following policies:
AutoScalingFullAccess
ElasticLoadBalancingFullAccess
I then rebuilt the environment, no more errors!
The following article helped me understand this better: https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/concepts-roles-service.html
You can go to your IAM console. You will see the "aws-elasticbeanstalk-service-role", Under the permission tab, you can grant the specific you want to give the policy on your behalf. Refresh and you should have "Ok". Make sure the permission you grant the role can manage this " Auto Scaling and Elastic Load Balancing using role"
Elastic Beanstalk now uses a Service Role to call other AWS services on your behalf. The IAM role is created in your account and you give permissions to launch
Service role is optional but recommended for new environments.
Especially for enhanced health monitoring (which is what your using based on the error message), service role is mandatory. A misconfigured service role can lead to this error message.
The console experience makes it very easy for you to create/use the role with the correct permissions. This is because you just need to select the correct role from the dropdown (if the role already exists) when using the create environment wizard.
When using the CLI you need to pass the service role option setting. (namespace: aws:elasticbeanstalk:environment, option_name: ServiceRole). You can find the required permissions for a role configured in this documentation.
There are some more details about service role in my previous stack overflow answer here.
If anyone can not solve the error whit the previous actions what I had to do is:
Add to aws-elasticbeanstalk-service-role the following policies AutoScalingFullAccess | ElasticLoadBalancingFullAccess
Not re-start but build the environment again
(after a while it kept launching the error)
Then I added:
AdministratorAccess-AWSElasticBeanstalk (again to aws-elasticbeanstalk-service-role)
Re-build the environment again
It is working at the moment.