Am trying to write a terraform module for attaching bucket policy to a AWS S3 bucket. Here is the code:
data "aws_iam_policy_document" "make_objects_public" {
# if policy_name == <this-policy-name>, then generate policy
count = "${var.policy_name == "make_objects_public" ? 1 : 0}"
statement {
...
}
}
resource "aws_s3_bucket_policy" "bucket_policy" {
# if policy_name != "", then add the generated policy
count = "${var.policy_name != "" ? 1 : 0}"
bucket = "${var.bucket_name}"
policy = "${data.aws_iam_policy_document.<policy-name-goes-here>.json}"
}
I want to interpolate the policy_name variable while fetching the policy generated by aws_iam_policy_document. I tried few things but sadly they didn't work. Is this possible in terraform ?
I tried out these hacks:
policy = "${data.aws_iam_policy_document."${var.policy_name}".json}"
policy = "${"${format("%s", "data.aws_iam_policy_document.${var.policy_name}.json")}"}"
policy = "${format("%s", "$${data.aws_iam_policy_document.${var.policy_name}.json}")}"
Thanks.
Dynamic resource names are not supported because Terraform must construct the dependency graph before it begins dealing with interpolations, and thus these relationships must be explicit.
A recommended approach for this sort of setup is to break the system down into small modules, which can then be used together by a calling module to produce the desired result without duplicating all of the details.
In this particular situation, you could for example split each policy out into its own re-usable module, and then write one more re-usable module that creates an S3 bucket and associates a given policy with it. Then a calling configuration may selectively instantiate one policy module appropriate to its needs along with the general S3 bucket module, to create the desired result:
module "policy" {
# policy-specific module; only contains the policy data source
source = "../policies/make_objects_public"
# (any other arguments the policy needs...)
}
module "s3_bucket" {
# S3 bucket module creates a bucket and attaches a policy to it
source = "../s3_bucket" # general resource for S3 buckets with attached policies
name = "example"
policy = "${module.policy.policy_json}" # an output from the policy module above
}
As well as avoiding the need for dynamic resource selection, this also increases flexibility by decoupling policy generation from the S3 bucket creation, and in principle allowing a calling module with unusual needs to skip instantiating a policy module altogether and just use aws_iam_policy_document directly.
The above pattern is somewhat analogous to the dependency injection technique, where the system is split into small components (modules, in this case) and then the root config "wires up" these components in a manner appropriate for a specific use-case. This approach has very similar advantages and disadvantages as the general technique.
Related
I'm trying to set the KMS key policy based on whether an IAM role exists or not. The yaml file looks like this.
data "aws_iam_policy_document" "key_policy" {
statement {
sid = "Give Access to the my Role"
actions = [
"kms:GenerateDateKey",
"kms:Decrypt"
]
resources = ["*"]
principals {
type = "AWS"
identifiers = ["arn:aws:iam::${var.account_id}:role/myrole"]
}
}
}
The issue with this terraform script is that it would fail if myrole doesn't exist in my account. We have multiple AWS accounts and only part of them has this myrole. Our CI will run this same script for all the accounts. Is there a way to apply this statement only if myrole exists?
The design principles of terraform (TF) dictate the a current TF configuration file should explicitly define your infrastructure. This means that resources that define your solution are only managed by a given configuration files and current TF state. If some resource is not in TF state, it simply is considered outside of the scope of TF and as non-existent.
As a consequence, TF does not directly support checking for existence of any resources nor conditional creation of resources based on such conditions. This would lead to issues as your TF would not fully represent your architectures.
Therefor, TF docs recommend:
Rather than trying to write a module that itself tries to detect whether something exists and create it if not, we recommend applying the dependency inversion approach: making the module accept the object it needs as an argument, via an input variable.
Based on the above, you should aim to develop your aws_iam_policy_document.key_policy so that it creates the statement in question based on an input variable. This means that your CICD pipeline would have to check for the existence of the role and pass this information to your CF file as an input variable, e.g. my_role_exists with true and false values.
Then, you could use dynamic blocks to make the statement conditional:
data "aws_iam_policy_document" "key_policy" {
dynamic "statement" {
for_each = var.my_role_exists == true ? [1] : []
content {
sid = "Give Access to the my Role"
actions = [
"kms:GenerateDateKey",
"kms:Decrypt"
]
resources = ["*"]
principals {
type = "AWS"
identifiers = ["arn:aws:iam::${var.account_id}:role/myrole"]
}
}
}
}
I'm seeing intermittent Terraform failures which look to me like a race condition internal to Terraform itself:
21:31:37 aws_s3_bucket.jar: Creation complete after 1s
(ID: automatictester.co.uk-my-bucket)
...
21:31:38 * aws_s3_bucket_object.jar: Error putting object in S3 bucket
(automatictester.co.uk-my-bucket): NoSuchBucket: The specified bucket
does not exist
As you can see in the above logs, TF first claims it has created a bucket at 21:31:37, and then says it can't put an object in that bucket because this does not exist at 21:31:38.
The code behind the above error:
resource "aws_s3_bucket" "jar" {
bucket = "${var.s3_bucket_jar}"
acl = "private"
}
...
resource "aws_s3_bucket_object" "jar" {
bucket = "${var.s3_bucket_jar}"
key = "my.jar"
source = "${path.module}/../target/my.jar"
etag = "${md5(file("${path.module}/../target/my.jar"))}"
}
There clearly is an implicit dependency defined between these two, so the only reason for that failure that comes to my mind is the eventually consistent nature of Amazon S3.
How to handle such kind of errors? I believe explcitly defined dependency with depends-on will not provide any value over the implicit dependency which is already there.
Terraform can't see any dependency ordering at all there so is almost certainly trying to do the same 2 actions at the same time and is failing the object creation at pretty much the same time the bucket creates.
Instead you should properly define the dependency between the 2 resources by using either depends_on or better yet referring to the bucket resource's outputs in the object resource like this:
resource "aws_s3_bucket" "jar" {
bucket = "${var.s3_bucket_jar}"
acl = "private"
}
resource "aws_s3_bucket_object" "jar" {
bucket = "${aws_s3_bucket.jar.bucket}"
key = "my.jar"
source = "${path.module}/../target/my.jar"
etag = "${md5(file("${path.module}/../target/my.jar"))}"
}
Terraform now knows that it needs to wait for the S3 bucket to be created and return before it attempts to create the S3 object in the bucket.
I have an infrastructure I'm deploying using Terraform in AWS. This infrastructure can be deployed to different environments, for which I'm using workspaces.
Most of the components in the deployment should be created separately for each workspace, but I have several key components that I wish to be shared between them, primarily:
IAM roles and permissions
They should use the same API Gateway, but each workspace should deploy to different paths and methods
For example:
resource "aws_iam_role" "lambda_iam_role" {
name = "LambdaGeneralRole"
policy = <...>
}
resource "aws_lambda_function" "my_lambda" {
function_name = "lambda-${terraform.workspace}"
role = "${aws_iam_role.lambda_iam_role.arn}"
}
The first resource is a IAM role that should be shared across all instances of that Lambda, and shouldn't be recreated more than once.
The second resource is a Lambda function whose name depends on the current workspace, so each workspace will deploy and keep track of the state of a different Lambda.
How can I share resources, and their state, between different Terraform workspaces?
For the shared resources, I create them in a separate template and then refer to them using terraform_remote_state in the template where I need information about them.
What follows is how I implement this, there are probably other ways to implement it. YMMV
In the shared services template (where you would put your IAM role) I use Terraform backend to store the output data for the shared services template in Consul. You also need to output any information you want to use in other templates.
shared_services template
terraform {
backend "consul" {
address = "consul.aa.example.com:8500"
path = "terraform/shared_services"
}
}
resource "aws_iam_role" "lambda_iam_role" {
name = "LambdaGeneralRole"
policy = <...>
}
output "lambda_iam_role_arn" {
value = "${aws_iam_role.lambda_iam_role.arn}"
}
A "backend" in Terraform determines how state is loaded and how an operation such as apply is executed. This abstraction enables non-local file state storage, remote execution, etc.
In the individual template you invoke the backend as a data source using terraform_remote_state and can use the data in that template.
terraform_remote_state:
Retrieves state meta data from a remote backend
individual template
data "terraform_remote_state" "shared_services" {
backend = "consul"
config {
address = "consul.aa.example.com:8500"
path = "terraform/shared_services"
}
}
# This is where you use the terraform_remote_state data source
resource "aws_lambda_function" "my_lambda" {
function_name = "lambda-${terraform.workspace}"
role = "${data.terraform_remote_state.shared_services.lambda_iam_role_arn}"
}
References:
https://www.terraform.io/docs/state/remote.html
https://www.terraform.io/docs/backends/
https://www.terraform.io/docs/providers/terraform/d/remote_state.html
Resources like aws_iam_role having a name attribute will not create a new instance if the name value matches an already provisioned resource.
So, the following will create a single aws_iam_role named LambdaGeneralRole.
resource "aws_iam_role" "lambda_iam_role" {
name = "LambdaGeneralRole"
policy = <...>
}
...
resource "aws_iam_role" "lambda_iam_role_reuse_existing_if_name_is_LambdaGeneralRole" {
name = "LambdaGeneralRole"
policy = <...>
}
Similarly, the aws provider will effectively creat one S3 bucket name my-store given the following:
resource "aws_s3_bucket" "store-1" {
bucket = "my-store"
acl = "public-read"
force_destroy = true
}
...
resource "aws_s3_bucket" "store-2" {
bucket = "my-store"
acl = "public-read"
force_destroy = true
}
This behaviour holds even if the resources were defined different workspaces with their respective separate Terraform state.
To get the best of this approach, define the shared resources as separate configuration. That way, you don't risk destroying a shared resource after running terraform destroy.
The terraform structure of my project is:
iam/policies/policy1.tf
iam/roles/roles1.tf
policy1.tf includes several policies, etc.:
resource "aws_iam_policy" "policy1A" {
name = "..."
policy = "${data.template_file.policy1ATempl.rendered}"
}
roles1.tf includes several roles, etc.:
resource "aws_iam_role" "role1" {
name = ....
assume_role_policy = ....
}
Now I want to attach policy1A to role1. Since policy1 and role1 are not in the same folder, how do I attach them?
resource "aws_iam_policy_attachment" "attach1" {
name = "attachment1"
roles = ??? (sth. like ["${aws_iam_role.role1.name}"])
policy_arn = ??? (sth. like "${aws_iam_policy.Policy1.arn}")
}
You can't make references like this across directories because Terraform only works in one directory at a time. If you really do need to separate these into different folders (which I would recommend against here) then normally you would use data sources to retrieve information about resources that have already been created.
In this case though the aws_iam_policy data source is currently pretty much useless because it requires the ARN of the policy instead of the path or name. Instead you can construct it if you know the policy name as it fits a well known pattern. You obviously know the name of role(s) you want to attach the policy to and you know the name of the policy so that covers those things.
data "aws_caller_identity" "current" {}
resource "aws_iam_policy_attachment" "attach1" {
name = "attachment1"
roles = "role1"
policy_arn = "arn:aws:iam::${data.aws_caller_identity.current.account_id}:policy/policy1A"
}
This above example does briefly show off how to use data sources as it fetches the AWS account ID, using the aws_caller_identity data source, and dynamically uses that to build the ARN of the policy you are trying to attach.
I would echo the warning in the docs against using the resource aws_iam_policy_attachment unless you are absolutely certain with what you're doing because it will detach the policy from anything else it happens to be attached to. If you want to take a managed policy and attach it to N roles I would instead recommend using aws_iam_role_policy_attachment and it's like instead.
I want to attach one of the pre-existing AWS managed roles to a policy, here's my current code:
resource "aws_iam_role_policy_attachment" "sto-readonly-role-policy-attach" {
role = "${aws_iam_role.sto-test-role.name}"
policy_arn = "arn:aws:iam::aws:policy/ReadOnlyAccess"
}
Is there a better way to model the managed policy and then reference it instead of hardcoding the ARN? It just seems like whenever I hardcode ARNs / paths or other stuff like this, I usually find out later there was a better way.
Is there something already existing in Terraform that models managed policies? Or is hardcoding the ARN the "right" way to do it?
The IAM Policy data source is great for this. A data resource is used to describe data or resources that are not actively managed by Terraform, but are referenced by Terraform.
For your example, you would create a data resource for the managed policy as follows:
data "aws_iam_policy" "ReadOnlyAccess" {
arn = "arn:aws:iam::aws:policy/ReadOnlyAccess"
}
The name of the data source, ReadOnlyAccess in this case, is entirely up to you. For managed policies I use the same name as the policy name for the sake of consistency, but you could just as easily name it readonly if that suits you.
You would then attach the IAM policy to your role as follows:
resource "aws_iam_role_policy_attachment" "sto-readonly-role-policy-attach" {
role = "${aws_iam_role.sto-test-role.name}"
policy_arn = "${data.aws_iam_policy.ReadOnlyAccess.arn}"
}
When using values that Terraform itself doesn't directly manage, you have a few options.
The first, simplest option is to just hard-code the value as you did here. This is a straightforward answer if you expect that the value will never change. Given that these "canned policies" are documented, built-in AWS features they likely fit this criteria.
The second option is to create a Terraform module and hard-code the value into that, and then reference this module from several other modules. This allows you to manage the value centrally and use it many times. A module that contains only outputs is a common pattern for this sort of thing, although you could also choose to make a module that contains an aws_iam_role_policy_attachment resource with the role set from a variable.
The third option is to place the value in some location that Terraform can retrieve values from, such as Consul, and then retrieve it from there using a data source. With only Terraform in play, this ends up being largely equivalent to the second option, though it means Terraform will re-read it on each refresh rather than only when you update the module using terraform init -upgrade, and thus this could be a better option for values that change often.
The fourth option is to use a specialized data source that can read the value directly from the source of truth. Terraform does not currently have a data source for fetching information on AWS managed policies, so this is not an option for your current situation, but can be used to fetch other AWS-defined data such as the AWS IP address ranges, service ARNs, etc.
Which of these is appropriate for a given situation will depend on how commonly the value changes, who manages changes to it, and on the availability of specialized Terraform data sources.
I got a similar situation, and I don't want to use the arn in my terraform script for two reasons,
If we do it on the Web console, we don't really look at the arn, we search the role name, then attach it to the role
The arn is not easy to remember, looks like is not for human
I would rather use the policy name, but not the arn, here is my example
# Get the policy by name
data "aws_iam_policy" "required-policy" {
name = "AmazonS3FullAccess"
}
# Create the role
resource "aws_iam_role" "system-role" {
name = "data-stream-system-role"
assume_role_policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Action = "sts:AssumeRole"
Effect = "Allow"
Sid = ""
Principal = {
Service = "ec2.amazonaws.com"
}
},
]
})
}
# Attach the policy to the role
resource "aws_iam_role_policy_attachment" "attach-s3" {
role = aws_iam_role.system-role.name
policy_arn = data.aws_iam_policy.required-policy.arn
}