AWS CloudWatch Cross-Account Logging with EC2 Instance Profile - amazon-web-services

When I originally setup CloudWatch, I created an EC2 Instance Profile to automatically grant access to write to the account's own CloudWatch service. Now, I would like to consolidate the logs from several accounts into a central account.
I'd like to implement a simplified architecture that is based on Centralized Logging on AWS. However, these logs will feed an on-premise ELK stack, so I'm only trying to implement the components outlined in red. I would like to solve this without the use of Kinesis.
Either the CloudWatch Agent (CWAgent) doesn't support assuming a role or I can't wrap my mind around how to craft the EC2 Instance Profile to allow the CWAgent to assume a role in a different account.
Logging Target (AWS Account 111111111111)
IAM LogStreamerRole:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": [
"arn:aws:iam::999999999999:role/EC2CloudWatchLoggerRole"
]
},
"Action": "sts:AssumeRole",
"Condition": {}
}
]
}
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": [
"arn:aws:logs:*:*:*"
]
}
]
}
Logging Source (AWS Account 999999999999)
IAM Instance Profile Role:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "ec2.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Action": "sts:AssumeRole",
"Resource": "arn:aws:iam::111111111111:role/LogStreamerRole"
}
]
}
The CWAgent is producing the following error:
/opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log:
2018-02-12T23:27:43Z E! CreateLogStream / CreateLogGroup with log group name Linux/var/log/messages stream name i-123456789abcdef has errors. Will retry the request: AccessDeniedException: User: arn:aws:sts::999999999999:assumed-role/EC2CloudWatchLoggerRole/i-123456789abcdef is not authorized to perform: logs:CreateLogStream on resource: arn:aws:logs:us-west-2:999999999999:log-group:Linux/var/log/messages:log-stream:i-123456789abcdef
status code: 400, request id: 53271811-1234-11e8-afe1-a3c56071215e
It is still trying to write to its own CloudWatch service, instead of to the central CloudWatch service.

From the logs, I see that the instance profile is used.
arn:aws:sts::999999999999:assumed-role/EC2CloudWatchLoggerRole/i-123456789abcdef
Just add the following to the /etc/awslogs/awscli.conf to assume the LogStreamerRole role.
role_arn = arn:aws:iam::111111111111:role/LogStreamerRole
credential_source=Ec2InstanceMetadata

Related

AWS InvalidParameter when calling the ImportImage operation

I have .ova VM's stored on my S3 bucket, I am trying to create AMI from these OVA.
I was going through this video to Import a VM as an Image Using VM Import/Export to Amazon EC2.
I have created an EC2 Instance which I will use to trigger the necessary CLI commands for Importing.
I have created an IAM Role and attached it to the EC2 Instance.
Please refer to the details of the Role:
Trust Policy
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "vmie.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
Inline Policy for Access to S3 and EC2
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"ec2:CopySnapshot",
"s3:ListAccessPointsForObjectLambda",
"s3:GetAccessPoint",
"s3:PutAccountPublicAccessBlock",
"s3:ListAccessPoints",
"ec2:RegisterImage",
"s3:ListJobs",
"s3:PutStorageLensConfiguration",
"s3:ListMultiRegionAccessPoints",
"s3:ListStorageLensConfigurations",
"ec2:Describe*",
"s3:GetAccountPublicAccessBlock",
"ec2:ModifySnapshotAttribute",
"s3:ListAllMyBuckets",
"s3:PutAccessPointPublicAccessBlock",
"s3:CreateJob",
"ec2:ImportImage"
],
"Resource": "*"
},
{
"Sid": "VisualEditor1",
"Effect": "Allow",
"Action": "s3:*",
"Resource": "arn:aws:s3:::vms"
},
{
"Sid": "AllowStsDecode",
"Effect": "Allow",
"Action": "sts:DecodeAuthorizationMessage",
"Resource": "*"
}
]
}
Inline Policy for KMS Decrypt
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": "kms:Decrypt",
"Resource": "*"
}
]
}
Also, I have attached the AWSImportExportFullAccess managed policy to the Role.
I am using the following command to Import the VM to AMI:
aws ec2 import-image --description "MY_VM_Image" --disk-containers "file://configuration.json"
Here are the contents of configuration.json
[{
"Description": "Image",
"Format": "ova",
"UserBucket": {
"S3Bucket": "vm",
"S3Key": "xzt.ova"
}
}
]
But I am facing the following error:
An error occurred (InvalidParameter) when calling the ImportImage operation: The service role vmimport provided does not exist or does not have sufficient permissions
I tried to have a look at the Troubleshooting document. It states the following
This error can also occur if the user calling ImportImage has Decrypt permission but the vmimport role does not.
So, I have also disabled the default encryption at S3.
Still no luck.
What else permissions are needed to run the command successfully.
I was facing the same issue and it turned out to be an issue with the clock not being in sync with the NTP servers (it was around 6 minutes off). As soon as the time was synced, the aws ec2 import-image worked as expected.
Here is a link for the importance of Time Synchronization in Kerberos:
https://tldp.org/HOWTO/Kerberos-Infrastructure-HOWTO/time-sync.html#:~:text=If%20you%20allow%20your%20clocks,errors%20and%20refuse%20to%20function.

API Gateway does not have permission to assume the provided role

I am trying to invoke a lambda function from an API Gateway. I have followed the next tutorial: https://docs.aws.amazon.com/apigateway/latest/developerguide/integrating-api-with-aws-services-lambda.html
However, I get the following error when I test it from the web of API Gateway:
Execution failed due to configuration error: API Gateway does not have permission to assume the provided role
I have search in google and I have not been able to solve it (this, for instance).
If I go to the IAM Management Console, I can see that the trust relationship allows API Gateway to assume the rol, and the JSON of the trust relationship is the following:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": [
"apigateway.amazonaws.com",
"lambda.amazonaws.com"
]
},
"Action": "sts:AssumeRole"
}
]
}
I have tried also with:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"Service": [
"lambda.amazonaws.com",
"apigateway.amazonaws.com"
]
},
"Action": "sts:AssumeRole"
}
]
}
The policy of the role is the next:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"lambda:InvokeFunction"
],
"Resource": [
"*"
]
}
]
}
What is wrong here? Thank you
To fix this go to the role in your IAM and select the “Trust Relationships” tab. From here edit the policy and for the Principal Service add in “apigateway.amazonaws.com” as seen below. This will grant the API Gateway the ability to assume roles to run your function in addition to the existing lambda permission.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": [
"apigateway.amazonaws.com",
"lambda.amazonaws.com"
]
},
"Action": "sts:AssumeRole"
}
]
}
I guess you have not attached the role to the invoking method i.e the api gateway
Attaching the created role to the api gateway is needed for api to execute the lamda.
Under Execution role, choose Choose an existing role.
Enter the role ARN for the lambda_invoke_function_assume_apigw_role
role you created earlier.
Choose Save.
AWS Link

Datadog AWS integration for multiple aws account

I have two AWS account , I was able to set AWS integration for the first account using Terraform, but when I try to create AWS integration for my second account I am having an error.
I have created a role with in-line policy and we do not have a cross account set up.
! Datadog is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::xxxxxxxxxx:role/DatadogAWSIntegrationRole. See http://docs.datadoghq.com/integrations/aws/
Trust Relationship:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::xxxxxxxxxxxx:root"
},
"Action": "sts:AssumeRole",
"Condition": {
"StringEquals": {
"sts:ExternalId": "xxxxxxxxxxxxxxxxxxxxxxxxxx"
}
}
}
]
}
Can anyone please guide me how to solve this error?
The role arn:aws:iam::xxxxxxxxxx:role/DatadogAWSIntegrationRole also has to have permission to assume the role on the other account.
You'll have to update the DatadogAWSIntegrationRole on the primary account to include:
{
"Version": "2012-10-17",
"Statement": [
...
{
"Effect": "Allow",
"Action": "sts:AssumeRole",
"Resource": "arn:aws:iam::xxxxxxxxxxxx:role/AssumedRoleForDataDogInOtherAccount"
}
]
}

AWS Code Commit Cross Account IAM Roles

I have two AWS accounts:
DEV: 111111111111
PROD: 999999999999
I created an a code commit repo in the prod account called prodRepo.
What I want to do is allow an ec2 instance on the DEV and PROD account to have read-only access to this repo. So git clone, git pull, etc...
I can do this easily on my PROD account using the following IAM instance profile called codecommit-tester
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"codecommit:BatchGetRepositories",
"codecommit:Get*",
"codecommit:GitPull",
"codecommit:List*"
],
"Resource": "arn:aws:codecommit:us-east-1:999999999999:prodRepo"
}
]
}
The Trust Relationship policy is:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "ec2.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
}
Then I use the aws credentials helpers in the git config to perform read-only git operations without having to store credentials on my machine (it gets the credentials for code commit from the instance metadata).
$ cat ~/.gitconfig
[credential]
helper = !aws codecommit credential-helper $#
UseHttpPath = true
The problem I am having is creating an the IAM policy/role on the DEV account to do the same thing as the PROD account. Here is what I tried.
I edited the Trust Relationship on the PROD account to trust the DEV account:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "ec2.amazonaws.com"
},
"Action": "sts:AssumeRole"
},
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::111111111111:root"
},
"Action": "sts:AssumeRole"
}
}
Now I think this means the DEV account can assume this role. On the DEV account I created these IAM policies attached to a role:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"codecommit:BatchGetRepositories",
"codecommit:Get*",
"codecommit:GitPull",
"codecommit:List*"
],
"Resource": "arn:aws:codecommit:us-east-1:999999999999:prodRepo"
}
]
}
{
"Version": "2012-10-17",
"Statement": {
"Effect": "Allow",
"Action": "sts:AssumeRole",
"Resource": "arn:aws:iam::999999999999:role/codecommit-tester"
}
}
I use the credentials helper on the DEV account after launching an ec2 instance using this IAM instance profile and I get this error when performing a git clone:
$ git clone https://git-codecommit.us-east-1.amazonaws.com/v1/repos/prodRepo
Cloning into 'prodRepo'...
fatal: unable to access 'https://git-codecommit.us-east-1.amazonaws.com/v1/repos/prodRepo/': The requested URL returned error: 403
So what did I miss in the IAM roles/policies on the DEV to make this work?
I think you don't need iam role in dev which you mention (On the DEV account I created these IAM policies attached to a role) .... have not tried with instance cross account assume role..
but if you can create new IAM role in prod account with
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "sts:AssumeRole",
"Resource": "arn:aws:iam::999999999999:role/codecommit-tester"
}
]
}
and trust relation would be something like
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::111111111111:root"
},
"Action": "sts:AssumeRole"
}
and just assume new IAM ARN in dev ec2 role.

AWS EC2: IAM policy for ec2:RequestSpotInstances

I need to create policy that would allow user to create spot requests, but with specific subnet and security group only. This is what I did:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "ec2:RequestSpotInstances",
"Resource": [
"arn:aws:ec2:us-east-1:123456789012:image/ami-*",
"arn:aws:ec2:us-east-1:123456789012:subnet/subnet-af016c92",
"arn:aws:ec2:us-east-1:123456789012:subnet/subnet-12a34d3c",
"arn:aws:ec2:us-east-1:123456789012:subnet/subnet-f0e844cd",
"arn:aws:ec2:us-east-1:123456789012:subnet/subnet-026ae728",
"arn:aws:ec2:us-east-1:123456789012:key-pair/*",
"arn:aws:ec2:us-east-1:123456789012:security-group/sg-b5dd94cd",
"arn:aws:ec2:us-east-1:123456789012:security-group/sg-3bda8c42"
]
}
]
}
But my spot request creation still fails:
botocore.exceptions.ClientError: An error occurred (UnauthorizedOperation) when calling the RequestSpotInstances operation: You are not authorized to perform this operation.
What is the minimum subset of permissions for RequestSpotInstances action?
Is there some possibility to debug this?
I know this is an old issue, but I just ran across the same issue in my environment. The solution for me was adding an IAM permission for "PassRole"
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Stmt1479335761363",
"Action": [
"ec2:DescribeInstances",
"ec2:RequestSpotInstances",
"ec2:RunInstances",
"iam:PassRole"
],
"Effect": "Allow",
"Resource": "*"
}]
}
According to the EC2 docs (here), ec2:RequestSpotInstances is an action which falls into the category of "Unsupported Resource-Level Permissions." Unfortunately, you will have to set the resource tag to all resources, like this:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "ec2:RequestSpotInstances",
"Resource": [ "*" ]
}
]
}
As far as debugging goes, don't forget about the IAM policy simulator, which can be accessed from the AWS Console => IAM => User page.