Terragrunt path resolution - locally referenced modules - splitting into multiple environments - amazon-web-services

Question

For business-related reasons, I am not able to split my infrastructure using versioned modules. As it is still beneficial to split the environments, and I would like to avoid the copy/paste root modules, which are basically just instantiation of child modules, over multiple directories, each representing its own environment, I would like to specify the source of root module in terragrunt.hcl
To make life easier, I have built a small part of the current infrastructure, just with a single module to speed up the development.
The current structure of the project looks like this:
├── config
│ ├── common.tfvars
│ └── terragrunt.hcl
├── environments
│ └── dev
├── infrastructure-modules
│ └── ecs
├── terraform-modules
│ ├── terraform-aws-ecr
All my infrastructure is described in infrastructure-modules/ecs/*.tf files, which are basically just instantiating child modules declared in terraform-modules/terraform-aws-*/.
With that, I can simply execute the terragrunt (terraform commands) from the infrastructure-modules/ecs directory.
To have a possibility to create the same environment in another account, I have introduced a new directory environments/dev/eu-central-1/ecs as shown on tree output from the root directory.
The environments/dev/eu-central-1/ecs, consists just of two files, terragrunt.hcl and common.tfvars.
I guess, that the usage of the common.tfvars is quite self-explanatory, where my terragrunt.hcl consists of remote_state {} and terraform {} blocks.
The important part of the terragrunt configuration file:
remote_state {}
terraform {
source = "../../../../infrastructure-modules/ecs"
{...}
}
Above I am basically referencing my root modules, declared in infrastructure-modules/ecs/*.tf. Where my root modules are instantiating child-modules declared in terraform-modules/terraform-aws-*/.
Child modules from infrastructure-modules/ecs/*.tf are instantianed like this:
module my_module {
source = "../../terraform-modules/terraform-aws-*"
{...}
}
In an ideal world, I would be able to execute terragrunt (terraform) commands from environments/dev/eu-central-1/ecs directory, but as I am using local (relative) paths, this is failing during the initialization of the modules, as the root module my_module loads the child module with following relative path:
module my_module {
source = "../../terraform-modules/terraform-aws-*"
{...}
}
This is causing a module instantiation in environments/dev/eu-central-1/ecs to fail as the relative path is different, based on parent module instantiation.
Initializing modules...
- my_module in
Error: Unreadable module directory
Unable to evaluate directory symlink: lstat ../../terraform-modules: no such
file or directory
So far, according to the documentation, path_relative_*, should be able to return the relative path between the path specified in its include block and the current terragrunt.hcl, but the problem here is that I am not having any include {} block(s) within my terragrunt.hcl files and thus this approach doesn't works. Symlinks are the last option.
EDIT
If I inspect the .terragrunt-cache/* on path environments/dev/eu-central-1/ecs I can confirm that all the "root" modules have been downloaded(copied over) into cache directory.
However, the module is being instantiated like this, and it tries to fetch the actual modules (Terraform modules) from directory two levels above.
module my_modules {
source = "../..//terraform-modules/terraform-aws-ecr"
So basically, I need to tell Terragrunt to download/fetch the modules from other path.
EDIT 2:
Inspecting .terragrunt-cache in the directory where I am running init shows, the the terraform-modules are never downloaded in the
terraform-infrastructure/environments/dev/eu-central-1/ecs/.terragrunt-cache/.
If I change my terraform-infrastructure/infrastructure-modules/ecs/ecr-repos.tf, from
module ecr_lz_ingestion {
source = "../../terraform-modules//terraform-aws-ecr"
{<MODULE_ARGUMENTS>}
}
}
to:
module ecr_lz_ingestion {
source = "../../../../../../../../terraform-modules//terraform-aws-ecr"
{<MODULE_ARGUMENTS>}
}
}
Terraform is able to initialize the child-modules as I have given a relative path to terraform-modules/ in the directory root, which is obviously a workaround.
Somehow I am expecting the Terragrunt to download both directories, terraform-modules and infrastructure-modules for the relative paths in module instantiation to work.

Answer 1

Based off the additional information you provided, I understand that this is your current directory structure:
terraform-infrastructure
├── config
│ ├── common.tfvars
│ └── terragrunt.hcl
├── environments
│ └── dev
│ └── eu-central-1
│ └── ecs
│ └── terragrunt.hcl
├── infrastructure-modules
│ └── ecs
├── terraform-modules
│ ├── terraform-aws-ecr
│
├── terragrunt.hcl
Notice the terragrunt.hcl in the parent directory that I added.
That terragrunt.hcl is considered the parent file and can include code that can be shared among other terragrunt files.
It can include something like this:
remote_state {}
In your eu-central-1/ecs folder, add the following to your terragrunt file:
include {
// searches up the directory tree from the current terragrunt.hcl file
// and returns the absolute path to the first terragrunt.hcl
path = find_in_parent_folders()
}
// using the path relative from the path stated in the include block
terraform {
source = "${path_relative_from_include()}//infrastructure-modules”
}
This should keep relative pathing intact when child modules are instantiating.
Edit:
From your GitHub issue, it would be best to either move the double slash to somewhere the modules share a common path with. Either that or to just consolidate your modules into a single folder.