Terraform ELB S3 Permissions Issue - amazon-web-services

Question

I am having an issue using Terraform (v0.9.2) adding services to an ELB (I'm using: https://github.com/segmentio/stack/blob/master/s3-logs/main.tf).
When I run terraform apply I get this error:
* module.solr.module.elb.aws_elb.main: 1 error(s) occurred:
* aws_elb.main: Failure configuring ELB attributes:
InvalidConfigurationRequest: Access Denied for bucket: my-service-
logs. Please check S3bucket permission
status code: 409, request id: xxxxxxxxxx-xxxx-xxxx-xxxxxxxxx
My service looks like this:
module "solr" {
source = "github.com/segmentio/stack/service"
name = "${var.prefix}-${terraform.env}-solr"
environment = "${terraform.env}"
image = "123456789876.dkr.ecr.eu-west-2.amazonaws.com/my-docker-image"
subnet_ids = "${element(split(",", module.vpc_subnets.private_subnets_id), 3)}"
security_groups = "${module.security.apache_solr_group}"
port = "8983"
cluster = "${module.ecs-cluster.name}"
log_bucket = "${module.s3_logs.id}"
iam_role = "${aws_iam_instance_profile.ecs.id}"
dns_name = ""
zone_id = "${var.route53_zone_id}"
}
My s3-logs bucket looks like this:
module "s3_logs" {
source = "github.com/segmentio/stack/s3-logs"
name = "${var.prefix}"
environment = "${terraform.env}"
account_id = "123456789876"
}
I checked in S3 and the bucket policy looks like this:
{
"Version": "2012-10-17",
"Id": "log-bucket-policy",
"Statement": [
{
"Sid": "log-bucket-policy",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::123456789876:root"
},
"Action": "s3:PutObject",
"Resource": "arn:aws:s3:::my-service-logs/*"
}
]
}
As far as I can see ELB should have access to the S3 bucket to store the logs (it's running in the same AWS account).
The bucket and the ELB are all in eu-west-2.
Any ideas on what the problem could be would be much appreciated.

Answer 1

The docs for ELB access logs say that you want to allow a specific Amazon account to be able to write to S3, not your account.
As such you want something like:
{
"Id": "Policy1429136655940",
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Stmt1429136633762",
"Action": [
"s3:PutObject"
],
"Effect": "Allow",
"Resource": "arn:aws:s3:::my-loadbalancer-logs/my-app/AWSLogs/123456789012/*",
"Principal": {
"AWS": [
"652711504416"
]
}
}
]
}
In Terraform you can use the aws_elb_service_account data source to automatically fetch the account ID used for writing logs as can be seen in the example in the docs:
data "aws_elb_service_account" "main" {}
resource "aws_s3_bucket" "elb_logs" {
bucket = "my-elb-tf-test-bucket"
acl = "private"
policy = <<POLICY
{
"Id": "Policy",
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"s3:PutObject"
],
"Effect": "Allow",
"Resource": "arn:aws:s3:::my-elb-tf-test-bucket/AWSLogs/*",
"Principal": {
"AWS": [
"${data.aws_elb_service_account.main.arn}"
]
}
}
]
}
POLICY
}
resource "aws_elb" "bar" {
name = "my-foobar-terraform-elb"
availability_zones = ["us-west-2a"]
access_logs {
bucket = "${aws_s3_bucket.elb_logs.bucket}"
interval = 5
}
listener {
instance_port = 8000
instance_protocol = "http"
lb_port = 80
lb_protocol = "http"
}
}

Answer 2

Even when having everything by the docs, I kept getting the "Access Denied for bucket" error. Removing the encryption from the bucket worked for me.

Answer 3

In the bucket policy, the account number must be NOT yours. Instead it belongs to AWS, and for each region, the account numbers you should use in your bucket policy are listed at: https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/enable-access-logs.html#attach-bucket-policy
For instance, for us-east-1 region the account number is 127311923021.
Although the question is about Terraform, I post CloudFormation snippet created a bucket for ELB's access logs its Bucket policy:
MyAccessLogsBucket:
Type: AWS::S3::Bucket
DeletionPolicy: Retain
MyAllowELBAccessBucketPolicy:
Type: AWS::S3::BucketPolicy
Properties:
Bucket: !Ref MyAccessLogsBucket
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: "Allow"
Principal:
AWS: "arn:aws:iam::127311923021:root"
Action:
- "s3:PutObject"
Resource: !Sub "arn:aws:s3:::${MyAccessLogsBucket}/AWSLogs/*"
In the principle, 127311923021 is used as this is AWS account number which should be used for account number in us-east-1.

Answer 4

Bucket permissions
When you enable access logging, you must specify an S3 bucket for the access logs. The bucket must meet the following requirements.
Requirements
The bucket must be located in the same Region as the load balancer.
Amazon S3-Managed Encryption Keys (SSE-S3) is required. No other encryption options are supported.
The bucket must have a bucket policy that grants Elastic Load Balancing permission to write the access logs to your bucket. Bucket policies are a collection of JSON statements written in the access policy language to define access permissions for your bucket. Each statement includes information about a single permission and contains a series of elements.
Use one of the following options to prepare an S3 bucket for access logging.
Amazon S3-Managed Encryption Keys (SSE-S3) is required. No other encryption options are supported.
So AWS docu says KMS is not supported...

Answer 5

In my case, it was the request_payer option set to Requester. Need to set to BucketOwner to work.