Working on an AWS SFTP solution with custom IDP. I have this s3 object block, which is intended to create a folder in s3:
resource "aws_s3_bucket_object" "home_directory" {
for_each = var.idp_users
bucket = aws_s3_bucket.s3.id
key = each.value["HomeDirectory"]
}
And this map variable input for idp_users:
idp_users = {
secret01 = {
Password = "password",
HomeDirectory = "test-directory-1",
Role = "arn:aws:iam::XXXXXXXXXXXX:role/custom_idp_sftp_role",
},
secret02 = {
Password = "password",
HomeDirectory = "test-directory-2",
Role = "arn:aws:iam::XXXXXXXXXXXX:role/custom_idp_sftp_role",
}
}
What I need is to simply add a "/" to the end of the HomeDirectory value in the aws_s3_bucket_object block, which will create a folder with the specific name in the s3 bucket. I know it could just be typed into the variable, but in the spirit of automation I want Terraform to append it manually and save us the hassle. I've monkeyed around with join and concatenate but can't figure out how to simply add a "/" to the end of the HomeDirectory value in the s3 object block. Can anyone provide some insight?
You can do that using string templating:
resource "aws_s3_bucket_object" "home_directory" {
for_each = var.idp_users
bucket = aws_s3_bucket.s3.id
key = "${each.value["HomeDirectory"]}/"
}
Related
I'm creating a cloudtrail using terraform. The problem is my source bucket keeps changing after 3 months. Now I want to give the dynamic S3 bucket value for field_selector.
I'm doing something like this:
resource "aws_cloudtrail" "test" {
name = "test_trail"
s3_bucket_name = bucket.id
enable_logging = true
include_global_service_events = true
is_multi_region_trail = true
enable_log_file_validation = true
advanced_event_selector {
name = "Log download event data"
field_selector {
field = "eventCategory"
equals = ["Data"]
}
field_selector {
field = "resources.type"
equals = ["AWS::S3::Object"]
}
field_selector {
field = "eventName"
equals = ["GetObject"]
}
field_selector {
field = "resources.ARN"
**starts_with = ["aws_s3_bucket.sftp_file_upload_bucket.arn"]**
}
}
Here, I'm giving the arn but logs are not getting created this way but if I hard code the bucket name it's getting created.
When you want to log the object events for a bucket, the ARN is not enough. As the AWS CLI documentation states [1]:
For example, if resources.type equals AWS::S3::Object , the ARN must be in one of the following formats. To log all data events for all objects in a specific S3 bucket, use the StartsWith operator, and include only the bucket ARN as the matching value. The trailing slash is intentional; do not exclude it.
So in your case you would have to fix the last field selector to:
field_selector {
field = "resources.ARN"
starts_with = ["${aws_s3_bucket.sftp_file_upload_bucket.arn}/"]
}
[1] https://awscli.amazonaws.com/v2/documentation/api/latest/reference/cloudtrail/put-event-selectors.html#id11
when using an attribute of a resource you should either specify it like
"${aws_s3_bucket.sftp_file_upload_bucket.arn}"
or without quotes like
aws_s3_bucket.sftp_file_upload_bucket.arn
so, the correct version would be
field_selector {
field = "resources.ARN"
starts_with = [aws_s3_bucket.sftp_file_upload_bucket.arn]
}
I am very new to terraform. My requirement is to upload objects to existing s3 buckets. I want to upload one or more objects from my source to one or more buckets utilizing only one resource. Using count and count.index I can create different numbers of resources. However, doing so will prevent me from using fileset which helps to recursively upload all the contents in the folder.
The basic code look like this. This is for multiple file uploads to single bucket but I would like to modify for multiple uploads to different buckets.;
variable "source_file_path"{
type = list(string)
description = "Path from where objects are to be uploaded"
}
variable "bucket_name"{
type = list(string)
description = "Name or ARN of the bucket to put the file in"
}
variable "data_folder"{
type = list(string)
description = "Object path inside the bucket"
}
resource "aws_s3_bucket_object" "upload_object"{
for_each = fileset(var.source_file_path, "*")
bucket = var.bucket_name
key = "${var.data_folder}${each.value}"
source = "${var.source_file_path}${each.value}"
}
I have created a vars.tfvars file with following values;
source_file_path = ["source1","source2"]
bucket_name = ["bucket1","bucket2"]
data_folder = ["path1","path2"]
So, what I need is, terraform to be able to upload all the files from the source1 to bucket1 s3 bucket by creating path1 inside the bucket. And similarly for source2, bucket2, and path2.
Is this something that can be done in terraform?
From your problem description it sounds like a more intuitive data structure to describe what you want to create would be a map of objects where the keys are bucket names and the values describe the settings for that bucket:
variable "buckets" {
type = map(object({
source_file_path = string
key_prefix = string
}))
}
When defining the buckets in your .tfvars file this will now appear as a single definition with a complex type:
buckets = {
bucket1 = {
source_file_path = "source1"
key_prefix = "path1"
}
bucket2 = {
source_file_path = "source2"
key_prefix = "path2"
}
}
This data structure has one element for each bucket, so it is suitable to use directly as the for_each for a resource describing the buckets:
resource "aws_s3_bucket" "example" {
for_each = each.buckets
bucket = each.key
# ...
}
There is a pre-existing official module hashicorp/dir/template which already encapsulates the work of finding files under a directory prefix, assigning each one a Content-Type based on its filename suffix, and optionally rendering templates. (You can ignore the template feature if you don't need it, by making your directory only contain static files.)
We need one instance of that module per bucket, because each bucket will have its own directory and thus its own set of files, and so we can use for_each chaining to tell Terraform that each instance of this module is related to one bucket:
module "bucket_files" {
for_each = aws_s3_bucket.example
base_dir = var.buckets[each.key].source_file_path
}
The module documentation shows how to map the result of the module to S3 bucket objects, but that example is for only a single instance of the module. In your case we need an extra step to turn this into a single collection of files across all buckets, which we can do using flatten:
locals {
bucket_files_flat = flatten([
for bucket_name, files_module in module.bucket_files : [
for file_key, file in files_module.files : {
bucket_name = bucket_name
local_key = file_key
remote_key = "${var.buckets[each.key].key_prefix}${file_key}"
source_path = file.source_path
content = file.content
content_type = file.content_type
etag = file.digests.md5
}
]
])
}
resource "aws_s3_bucket_object" "example" {
for_each = {
for bf in local.bucket_files_flat :
"s3://${bf.bucket_name}/${bf.remote_key}" => bf
}
# Now the rest of this is basically the same as
# the hashicorp/dir/template S3 example, but using
# the local.bucket_files_flat structure instead
# of the module result directly.
bucket = each.value.bucket_name
key = each.value.remote_key
content_type = each.value.content_type
# The template_files module guarantees that only one of these two attributes
# will be set for each file, depending on whether it is an in-memory template
# rendering result or a static file on disk.
source = each.value.source_path
content = each.value.content
# Unless the bucket has encryption enabled, the ETag of each object is an
# MD5 hash of that object.
etag = each.value.etag
}
Terraform needs a unique tracking key for each instance of aws_s3_bucket_object.example, and so I just arbitrarily decided to use the s3:// URI convention here, since I expect that's familiar to folks accustomed to working with S3. This means that the resource block will declare instances with addresses like this:
aws_s3_bucket_object.example["s3://bucket1/path1example.txt"]
aws_s3_bucket_object.example["s3://bucket2/path2other_example.txt"]
Because these objects are uniquely identified by their final location in S3, Terraform will understand changes to the files as updates in-place, but any changes to the location as removing an existing object and adding a new one at the same time.
(I replicated the fact that your example just concatenated the path prefix with the filename without any intermediate separator, and so that's why it appears as path1example.txt above and not path1/example.txt. If you want the slash in there, you can add it to the expression which defined remote_key inside local.bucket_files_flat.)
I have a few secerts in aws that were created manually. Is there a way to list them with data "aws_secretsmanager_secret"?
My goal is to get an list/index of the ARNs and then use it in a daymnic block. I want to try and avoid writing multiple data source blocks.
All the sercerts have a similar naming prefix:
db-credentials/${var.env-name}/<db-user>
The <db-user> changes of course from user to user.
So I guess I'm looking to iterate with data source over all secrets which falls into this naming pattern and get a list of their ARN. After that use each ARN indie a daymnic block
The daynic block will be used inside resource "aws_db_proxy" in the auth block
if anyone will find this useful I manged to do it like this:
locals {
secrets_list = [
"db-credentials/${var.env-name}/user1",
"db-credentials/${var.env-name}/user2",
"db-credentials/${var.env-name}/user3"
]
}
data "aws_secretsmanager_secret" "rds_secrets" {
for_each = toset(local.secrets_list)
name = each.key
}
resource "aws_db_proxy" "rds_db_proxy" {
name = "${var.env-name}-rds-proxy"
engine_family = "MYSQL"
idle_client_timeout = 900
require_tls = true
.
.
.
.
dynamic "auth" {
for_each = local.secrets_list
content {
secret_arn = data.aws_secretsmanager_secret.rds_secrets[auth.value].arn
auth_scheme = "SECRETS"
iam_auth = "REQUIRED"
}
}
}
Amazon Athena reads data from input Amazon S3 buckets using the IAM credentials of the user who submitted the query; query results are stored in a separate S3 bucket.
Here is the script in Hashicorp site https://www.terraform.io/docs/providers/aws/r/athena_database.html
resource "aws_s3_bucket" "hoge" {
bucket = "hoge"
}
resource "aws_athena_database" "hoge" {
name = "database_name"
bucket = "${aws_s3_bucket.hoge.bucket}"
}
Where it says
bucket - (Required) Name of s3 bucket to save the results of the query execution.
How can I specify the input S3 bucket in the terraform script?
You would use the storage_descriptor argument in the aws_glue_catalog_table resource:
https://www.terraform.io/docs/providers/aws/r/glue_catalog_table.html#parquet-table-for-athena
Here is an example of creating a table using CSV file(s):
resource "aws_glue_catalog_table" "aws_glue_catalog_table" {
name = "your_table_name"
database_name = "${aws_athena_database.your_athena_database.name}"
table_type = "EXTERNAL_TABLE"
parameters = {
EXTERNAL = "TRUE"
}
storage_descriptor {
location = "s3://<your-s3-bucket>/your/file/location/"
input_format = "org.apache.hadoop.mapred.TextInputFormat"
output_format = "org.apache.hadoop.mapred.TextInputFormat"
ser_de_info {
name = "my-serde"
serialization_library = "org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe"
parameters = {
"field.delim" = ","
"skip.header.line.count" = "1"
}
}
columns {
name = "column1"
type = "string"
}
columns {
name = "column2"
type = "string"
}
}
}
The input S3 bucket is specified in each table you create in the database, as such, there's no global definition for it.
As of today, the AWS API doesn't have much provision for Athena management, as such, neither does the aws CLI command, and nor does Terraform. There's no 'proper' way to create a table via these means.
In theory, you could create a named query to create your table, and then execute that query (for which there is API functionality, but not yet Terraform). It seems a bit messy to me, but it would probably work if/when TF gets the StartQuery functionality. The asynchronous nature of Athena makes it tricky to know when that table has actually been created though, and so I can imagine TF won't fully support table creation directly.
TF code that covers the currently available functionality is here: https://github.com/terraform-providers/terraform-provider-aws/tree/master/aws
API doco for Athena functions is here: https://docs.aws.amazon.com/athena/latest/APIReference/API_Operations.html
How to create a multiple folders inside an existing bucket using terraform.
example: bucket/folder1/folder2
resource "aws_s3_bucket_object" "folder1" {
bucket = "${aws_s3_bucket.b.id}"
acl = "private"
key = "Folder1/"
source = "/dev/null"
}
While the answer of Nate is correct, this would lead to a lot of code duplication. A better solution in my opinion would be to work with a list and loop over it.
Create a variable (variable.tf file) that contains a list of possible folders:
variable "s3_folders" {
type = "list"
description = "The list of S3 folders to create"
default = ["folder1", "folder2", "folder3"]
}
Then alter the piece of code you already have:
resource "aws_s3_bucket_object" "folders" {
count = "${length(var.s3_folders)}"
bucket = "${aws_s3_bucket.b.id}"
acl = "private"
key = "${var.s3_folders[count.index]}/"
source = "/dev/null"
}
Apply the same logic as you did to create the first directory.
resource "aws_s3_bucket_object" "folder1" {
bucket = "${aws_s3_bucket.b.id}"
acl = "private"
key = "Folder1/Folder2/"
source = "/dev/null"
}
There a no tips for windows users but this should work for you.
Slightly easier than using an empty file as "source"
resource "aws_s3_bucket_object" "output_subdir" {
bucket = "${aws_s3_bucket.file_bucket.id}"
key = "output/"
content_type = "application/x-directory"
}
resource "aws_s3_bucket_object" "input_subdir" {
bucket = "${aws_s3_bucket.file_bucket.id}"
key = "input/"
content_type = "application/x-directory"
}