How to fix AWS Config generating AccessDenied error? - amazon-web-services

I am trying to allow AWS Config to write to a non-public S3 bucket.
Based on the official documentation, I should have two policies assigned to the AWS role. However, It is not possible to add any policy to the service-linked role, neither to create a custom new service-linked role for AWS config.
As such, how can I stop receiving the S3 AccessDenied error without making the bucket public?
edit: here is the error log:
{
"eventVersion": "1.07",
"userIdentity": {
"type": "AssumedRole",
"principalId": "xxxxxxxxxxxxxxxxxxxxx:AWSConfig-BucketConfigCheck",
"arn": "arn:aws:sts::xxxxxxxxxxxx:assumed-role/AWSServiceRoleForConfig/AWSConfig-BucketConfigCheck",
"accountId": "xxxxxxxxxxxx",
"accessKeyId": "xxxxxxxxxxxxxxxxxxxx",
"sessionContext": {
"sessionIssuer": {
"type": "Role",
"principalId": "xxxxxxxxxxxxxxxxxxxxx",
"arn": "arn:aws:iam::xxxxxxxxxxxx:role/aws-service-role/config.amazonaws.com/AWSServiceRoleForConfig",
"accountId": "xxxxxxxxxxxx",
"userName": "AWSServiceRoleForConfig"
},
"attributes": {
"creationDate": "2020-04-30T00:43:57Z",
"mfaAuthenticated": "false"
}
},
"invokedBy": "AWS Internal"
},
"eventTime": "2020-04-30T00:43:57Z",
"eventSource": "s3.amazonaws.com",
"eventName": "PutObject",
"awsRegion": "eu-west-1",
"sourceIPAddress": "xxx.xxx.xxx.xxx",
"userAgent": "[AWSConfig]",
"errorCode": "AccessDenied",
"errorMessage": "Access Denied",
"requestParameters": {
"bucketName": "aws-config-bucket-xxxxxxxxxxxx",
"Host": "aws-config-bucket-xxxxxxxxxxxx.s3.eu-west-1.amazonaws.com",
"x-amz-acl": "bucket-owner-full-control",
"x-amz-server-side-encryption": "AES256",
"key": "AWSLogs/xxxxxxxxxxxx/Config/ConfigWritabilityCheckFile"
},
"responseElements": null,
"additionalEventData": {
"SignatureVersion": "SigV4",
"CipherSuite": "ECDHE-RSA-AES128-SHA",
"bytesTransferredIn": 0,
"AuthenticationMethod": "AuthHeader",
"x-amz-id-2": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx=",
"bytesTransferredOut": 243
},
"requestID": "xxxxxxxxxxxxxxxx",
"eventID": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
"readOnly": false,
"resources": [
{
"type": "AWS::S3::Object",
"ARN": "arn:aws:s3:::aws-config-bucket-xxxxxxxxxxxx/AWSLogs/xxxxxxxxxxxx/Config/ConfigWritabilityCheckFile"
},
{
"accountId": "xxxxxxxxxxxx",
"type": "AWS::S3::Bucket",
"ARN": "arn:aws:s3:::aws-config-bucket-xxxxxxxxxxxx"
}
],
"eventType": "AwsApiCall",
"managementEvent": false,
"recipientAccountId": "xxxxxxxxxxxx",
"vpcEndpointId": "vpce-xxxxxxxx",
"eventCategory": "Data"
}

I found the answer here: https://forums.aws.amazon.com/thread.jspa?threadID=314156
When AWS Config sends configuration information to an Amazon S3
bucket in another account, it first attempts to use the IAM role, but
this attempt fails if the access policy for the bucket does not grant
WRITE access to the IAM role. In this event, AWS Config sends the
information again, this time as the AWS Config service principal.
I checked my logs and there was an AWS Config service principal log, the same second as the AccessDenied, that was being accepted. Therefore, the error can be safely ignored. I have updated my Cloudwatch alarm to ignore it:
{($.errorCode="*UnauthorizedOperation") || (($.errorCode="AccessDenied*") && (($.userIdentity.type!="AssumedRole") || ($.userAgent!="[AWSConfig]")))}

Related

Determine the exact problem point of calling AWS Backup to AWS KMS in order to decrypt S3 bucket

Here is a log in AWS CloudTrail presumably indicating that
the AWS Backup service ("userAgent": "backup.amazonaws.com",)
tries to call the AWS KMS service("eventSource": "kms.amazonaws.com",)
in order to decrypt("eventName": "Decrypt",)
some S3 bucket("principalId": "AROAY6JAXY37VKF726QCA:AWS_BACKUP_S3_71C3048C",) and fails.
But there are no clues as to which particular bucket or KMS key caused the problem, I'd appreciate any advice on how to determine the exact problem point, thanks.
BTW the account contains hundreds of keys and buckets that are being back-up, so to determine the problem point in some manual way like, like researching each bucket/key, is very undesirable:)
{
"eventVersion": "1.08",
"userIdentity": {
"type": "AssumedRole",
"principalId": "AROAY6JAXY37VKF726QCA:AWS_BACKUP_S3_71C3048C",
"arn": "arn:aws:sts::****:assumed-role/AWSBackupDefaultServiceRole/AWS_BACKUP_S3_71C3048C",
"accountId": "****",
"accessKeyId": "ASIAY6JAXY37WHC7VFWN",
"sessionContext": {
"sessionIssuer": {
"type": "Role",
"principalId": "AROAY6JAXY37VKF726QCA",
"arn": "arn:aws:iam::****:role/service-role/AWSBackupDefaultServiceRole",
"accountId": "****",
"userName": "AWSBackupDefaultServiceRole"
},
"webIdFederationData": {},
"attributes": {
"creationDate": "2022-12-18T05:54:35Z",
"mfaAuthenticated": "false"
}
},
"invokedBy": "backup.amazonaws.com"
},
"eventTime": "2022-12-18T05:54:35Z",
"eventSource": "kms.amazonaws.com",
"eventName": "Decrypt",
"awsRegion": "eu-central-1",
"sourceIPAddress": "backup.amazonaws.com",
"userAgent": "backup.amazonaws.com",
"errorCode": "AccessDenied",
"errorMessage": "The ciphertext refers to a customer master key that does not exist, does not exist in this region, or you are not allowed to access.",
"requestParameters": null,
"responseElements": null,
"requestID": "660b1b90-66d9-4a2b-9ab1-4af4e195b71f",
"eventID": "ef6fe50b-67f9-46c6-bbf5-057de88c3c1c",
"readOnly": true,
"eventType": "AwsApiCall",
"managementEvent": true,
"recipientAccountId": "****",
"eventCategory": "Management"
}

Access denied on table access to AWS Lake Formation using AWS EMR

I'm working on a small pet project in which I want to setup AWS Lake Formation, put there some data and then access this data in AWS EMR. I was following instructions in this doc: https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-lake-formation.html. I've already got:
A data lake with some CSVs
Created table schemas for these CSVs
A external IdP (Auth0)
IAM Roles as in the tutorial.
The EMR cluster set up as in tutorial.
The problem is that I cannot access data in data lake. I can list databases and tables but can't perform a SELECT on any table.
I get:
com.amazonaws.emr.recordserver.remote.RecordServerException: com.amazonaws.emr.recordserver.remote.RecordServerException: Error while trying to get temporary table credentials for user: 5000, table: ***, database: ***, catalog: null from Secret Agent.
...
Caused by: SecretAgentClientException{httpResponseCode=Optional[500] message=Access denied for tableArn=arn:aws:glue:eu-west-1:743262912284:TABLE_NAME userId=5000 permissions=SELECT cause.class=null}
I checked the logs in Lake Formation and there I've got a DataAccess event with principal set to the user that I use in Zeppelin notebook.
{
"eventVersion": "1.08",
"userIdentity": {
"type": "AssumedRole",
"principalId": "***",
"arn": "***",
"accountId": "***",
"accessKeyId": "***",
"sessionContext": {
"sessionIssuer": {
"type": "Role",
"principalId": "***",
"arn": "PROPER_ROLE_ARN",
"accountId": "***",
"userName": "PROPER_ROLE"
},
"attributes": {
"creationDate": "2022-01-14T10:26:39Z",
"mfaAuthenticated": "false"
}
}
},
"eventTime": "2022-01-14T10:53:40Z",
"eventSource": "lakeformation.amazonaws.com",
"eventName": "GetDataAccess",
"awsRegion": "eu-west-1",
"sourceIPAddress": "***",
"userAgent": "aws-sdk-java/1.11.970 Linux/4.14.252-195.483.amzn2.x86_64 OpenJDK_64-Bit_Server_VM/25.312-b07 java/1.8.0_312 vendor/Amazon.com_Inc.",
"errorCode": "AccessDenied",
"errorMessage": "Access is not allowed.",
"requestParameters": {
"tableArn": "arn:aws:glue:eu-west-1:***:***",
"permissions": [
"SELECT"
],
"durationSeconds": 3600
},
"responseElements": null,
"additionalEventData": {
"requesterService": "UNKNOWN",
"LakeFormationTrustedCallerInvocation": "true",
"lakeFormationPrincipal": "PROPER_PRINCIPAL_ARN"
},
"requestID": "1e4c02e9-e09e-4906-a96f-c29a882b46de",
"eventID": "b5b26de0-60f4-482c-b670-7fba9d7089e5",
"readOnly": true,
"eventType": "AwsApiCall",
"managementEvent": true,
"recipientAccountId": "***",
"eventCategory": "Management",
"tlsDetails": {
"clientProvidedHostHeader": "lakeformation.eu-west-1.amazonaws.com"
}
}
The assumed role in sessionContext is correct. I've checked policies of created roles multiple times and they are exactly as in the tutorial (they have granted access to the lakeformation and to glue and also to all tables)
Please ensure that you allow each account to run Lake Formation-enabled EMR clusters. You can do so on the Lake Formation console by going to "External Data Filtering" found on the left navigation panel.
https://docs.aws.amazon.com/lake-formation/latest/dg/getting-started-setup.html#emr-switch.

Securely upload secrets to Secret Manager/Parameter Store

I caught a misstake I have made in the way I have been uploading secrets to Secrets Manager. Through using cloudformation I have been sending in the secret as a plain text parameter into the template. The secret itself never gets exposed in the cloudformation yaml file. However, the secret is exposed as a parameter in cloudformation. Hence, being able to read/describe the stack is enough to get the secret.
Did some digging and found this. They suggest creating the parameter store/secret manager using cdk or cloudformation and after which you upload the secret using SDK/CLI.
To my question: does the SDK and CLI give traces themselves? Meaning, have I just moved the problem. Shifted from exposing the secret in cloudformation to exposing it to cloudtrail or any other monitoring in AWS.
How can I securely upload my own secrets in combination with IaC, without manually using the AWS console. Is there a way to turn of logging for certain SDK/CLI calls?
Depending on your use case there are different options:
If you set up new resources and need to create a new secret, you can have the SecretsManager generate the secret for you. See CloudFormation docs for the Secret resource.
If you want to store an existing secret, the option with the separate API-call is a good suggestion. The only place this could in principle be recorded is CloudTrail, which records any API-Call, but I have confirmed, that the secret value is not stored in the PutSecretValue event record.
A CreateSecret event from CloudTrail:
{
"eventVersion": "1.08",
"userIdentity": {
"type": "IAMUser",
"principalId": "AIDA2BFBC5RB4SDFSDQDI",
"arn": "arn:aws:iam::123456789123:user/myself",
"accountId": "123456789123",
"accessKeyId": "ASIA2BFSDFSD5RBR4L2JB7T",
"userName": "myself",
"sessionContext": {
"sessionIssuer": {},
"webIdFederationData": {},
"attributes": {
"mfaAuthenticated": "true",
"creationDate": "2021-07-05T11:38:38Z"
}
}
},
"eventTime": "2021-07-05T11:39:46Z",
"eventSource": "secretsmanager.amazonaws.com",
"eventName": "CreateSecret",
"awsRegion": "eu-central-1",
"sourceIPAddress": "95.48.10.191",
"userAgent": "aws-internal/3 aws-sdk-java/1.11.1030 Linux/5.4.109-57.183.amzn2int.x86_64 OpenJDK_64-Bit_Server_VM/25.292-b10 java/1.8.0_292 vendor/Oracle_Corporation cfg/retry-mode/legacy",
"requestParameters": {
"name": "/demo",
"clientRequestToken": "5c59462b-d05c-4cfa-a224-a8d60f3edeff"
},
"responseElements": null,
"requestID": "6e61267a-ed8a-4383-8729-c33b8c217990",
"eventID": "23facc03-032c-4b24-bc36-d8f4e330445e",
"readOnly": false,
"eventType": "AwsApiCall",
"managementEvent": true,
"eventCategory": "Management",
"recipientAccountId": "123456789123",
"sessionCredentialFromConsole": "true"
}
A PutSecretValue event in CloudTrail:
{
"eventVersion": "1.08",
"userIdentity": {
"type": "IAMUser",
"principalId": "AIDA2BFSASB4SXNVRQDI",
"arn": "arn:aws:iam::123456789123:user/myself",
"accountId": "123456789123",
"accessKeyId": "ASIA2BFBSAWR4L2JB7T",
"userName": "myself",
"sessionContext": {
"sessionIssuer": {},
"webIdFederationData": {},
"attributes": {
"mfaAuthenticated": "true",
"creationDate": "2021-07-05T11:38:38Z"
}
}
},
"eventTime": "2021-07-05T11:40:09Z",
"eventSource": "secretsmanager.amazonaws.com",
"eventName": "PutSecretValue",
"awsRegion": "eu-central-1",
"sourceIPAddress": "11.11.190.191",
"userAgent": "aws-internal/3 aws-sdk-java/1.11.1030 Linux/5.4.109-57.183.amzn2int.x86_64 OpenJDK_64-Bit_Server_VM/25.292-b10 java/1.8.0_292 vendor/Oracle_Corporation cfg/retry-mode/legacy",
"requestParameters": {
"clientRequestToken": "61297703-b519-4e9e-8984-aacd40db826b",
"secretId": "/demo"
},
"responseElements": null,
"requestID": "97693f1b-f586-4641-af4c-b46d66fd27c1",
"eventID": "192f8959-3c51-40f5-8ca6-88f9075dc2a3",
"readOnly": false,
"eventType": "AwsApiCall",
"managementEvent": true,
"eventCategory": "Management",
"recipientAccountId": "123456789123",
"sessionCredentialFromConsole": "true"
}

Instace Type Change log

How do I find out when the Instance size was changed using Cloudtrail? e.g. large -xlarge date, user and so on. One of the instance size have been changed and I'd like to find out which user has changed it
The Logging Amazon EC2, Amazon EBS, and Amazon VPC API Calls with AWS CloudTrail documentation states that:
All Amazon EC2, Amazon EBS, and Amazon VPC actions are logged by CloudTrail and are documented in the Amazon EC2 API Reference.
The Amazon EC2 API Reference shows that the action you're looking for is the one called ModifyInstanceAttribute; in your case you should target events that have "eventName": "ModifyInstanceAttribute" and have the instanceType key in the requestParameters object. The identity of the user (or role) that initiated the action is in userIdentity.
An example of such event in CloudTrail is:
{
"eventVersion": "1.05",
"userIdentity": {
"type": "IAMUser",
"principalId": "XXXXXXXXXXX",
"arn": "arn:aws:iam::XXXXXXXXXX:user/my_user",
"accountId": "XXXXXXXXXX",
"accessKeyId": "XXXXXXXXXX",
"userName": "my_user",
"sessionContext": {
"sessionIssuer": {},
"webIdFederationData": {},
"attributes": {
"mfaAuthenticated": "true",
"creationDate": "2020-11-26T15:49:37Z"
}
}
},
"eventTime": "2020-11-26T16:54:18Z",
"eventSource": "ec2.amazonaws.com",
"eventName": "ModifyInstanceAttribute",
"awsRegion": "eu-west-1",
"sourceIPAddress": "111.22.33.444",
"userAgent": "console.ec2.amazonaws.com",
"requestParameters": {
"instanceId": "i-08999dedafc4xxyyz",
"instanceType": {
"value": "t3.nano"
}
},
"responseElements": {
"requestId": "11111111-2222-472f-ad77-bbeb506b242d",
"_return": true
},
"requestID": "11111111-2222-472f-ad77-bbeb506b242d",
"eventID": "aaaaaaa-c757-4501-8889-4f9d90720c0c",
"eventType": "AwsApiCall",
"recipientAccountId": "XXXXXXXXXX"
}

Create CloudWatch Alarm to notify about setting a S3 object to public

I want to create on CloudWatch a metric filter and an alarm based on it to notify me about S3 events, specially when a file or a bucket is set to public. This is the metric filter I used to create the metric:
{ ($.eventSource = s3.amazonaws.com) && (($.eventName = PutBucketAcl)
|| ($.eventName = PutObjectAcl)) &&
(($.requestParameters.AccessControlPolicy.AccessControlList.Grant.Grantee.type
= Group ))}
I tested this pattern by putting the following Custom log data :
{
"Records": [
{
"eventVersion": "1.03",
"userIdentity": {
"type": "IAMUser",
"principalId": "111122223333",
"arn": "arn:aws:iam::111122223333:user/myUserName",
"accountId": "111122223333",
"accessKeyId": "AKIAIOSFODNN7EXAMPLE",
"userName": "myUserName"
},
"eventTime": "2015-08-26T20:46:31Z",
"eventSource": "s3.amazonaws.com",
"eventName": "DeleteBucketPolicy",
"awsRegion": "us-west-2",
"sourceIPAddress": "127.0.0.1",
"userAgent": "[]",
"requestParameters": {
"bucketName": "myawsbucket"
},
"responseElements": null,
"requestID": "47B8E8D397DCE7A6",
"eventID": "cdc4b7ed-e171-4cef-975a-ad829d4123e8",
"eventType": "AwsApiCall",
"recipientAccountId": "111122223333"
},
{
"eventVersion": "1.03",
"userIdentity": {
"type": "IAMUser",
"principalId": "111122223333",
"arn": "arn:aws:iam::111122223333:user/myUserName",
"accountId": "111122223333",
"accessKeyId": "AKIAIOSFODNN7EXAMPLE",
"userName": "myUserName"
},
"eventTime": "2015-08-26T20:46:31Z",
"eventSource": "s3.amazonaws.com",
"eventName": "PutBucketAcl",
"awsRegion": "us-west-2",
"sourceIPAddress": "",
"userAgent": "[]",
"requestParameters": {
"bucketName": "",
"AccessControlPolicy": {
"AccessControlList": {
"Grant": {
"Grantee": {
"xsi:type": "Group",
"xmlns:xsi": "http://www.w3.org/2001/XMLSchema-instance",
"ID": "d25639fbe9c19cd30a4c0f43fbf00e2d3f96400a9aa8dabfbbebe1906Example"
},
"Permission": "FULL_CONTROL"
}
},
"xmlns": "http://s3.amazonaws.com/doc/2006-03-01/",
"Owner": {
"ID": "d25639fbe9c19cd30a4c0f43fbf00e2d3f96400a9aa8dabfbbebe1906Example"
}
}
},
"responseElements": null,
"requestID": "BD8798EACDD16751",
"eventID": "607b9532-1423-41c7-b048-ec2641693c47",
"eventType": "AwsApiCall",
"recipientAccountId": "111122223333"
},
{
"eventVersion": "1.03",
"userIdentity": {
"type": "IAMUser",
"principalId": "111122223333",
"arn": "arn:aws:iam::111122223333:user/myUserName",
"accountId": "111122223333",
"accessKeyId": "AKIAIOSFODNN7EXAMPLE",
"userName": "myUserName"
},
"eventTime": "2015-08-26T20:46:31Z",
"eventSource": "s3.amazonaws.com",
"eventName": "GetBucketVersioning",
"awsRegion": "us-west-2",
"sourceIPAddress": "",
"userAgent": "[]",
"requestParameters": {
"bucketName": "myawsbucket"
},
"responseElements": null,
"requestID": "07D681279BD94AED",
"eventID": "f2b287f3-0df1-4961-a2f4-c4bdfed47657",
"eventType": "AwsApiCall",
"recipientAccountId": "111122223333"
}
]
}
I clicked Test Pattern and I get this message:
Results Found 0 matches out of 50 event(s) in the sample log.
Is the metric filter proper and correct ? I'm supposed to have one result but it is not coming up.
Calculating whether a policy is providing open access is quite complex, due to the many ways that rules can be specified in the Bucket Policy (for example, wildcards can provide access).
An easier approach would be to use the Amazon S3 Bucket Permissions check in Trusted Advisor:
Checks buckets in Amazon Simple Storage Service (Amazon S3) that have open access permissions or allow access to any authenticated AWS user.
You can then Monitor Trusted Advisor Check Results with Amazon CloudWatch Events.
However, that particular check is not included in the Free Tier for Trusted Advisor. You would need to be on a Support Plan for that check to operate.
The Amazon S3 console was also recently updated -- it now clearly shows any buckets with public permissions.