Terraform and Gitlab CI pipeline for multiple projects - amazon-web-services

Question

We have an existing Gitlab CI EE pipeline for Terraform that works, one env at a time, each of them in a different AWS account.
However, we want to be able to scale pipeline for different teams which uses TF for their IaC requirements.
Is there a way to do it? different IaC repo/branch per team with same pipeline to handle TF deployments, with S3 as backend?
Any other way to do it? We have Gitlab CI EE and Terraform opensource edition.

Answer 1

I did something similar.
I had multiple repositories for each terraform module and a main repo that calls each module You can check how to do it here, in this main repository I had something like this:
- dev/
- main.tf
- variables.tf
- dev.tfvars
- backend.tf
- test/
- prd/
- .gitlab-ci.yml
Each environment had its own folder/files with the s3 and variables necessary to run. The main file is where all the other repos are called. The pipeline is triggered when it detects changes in the paths and it is initialized on the environment depending on the commit branch:
include:
- template: Terraform/Base.latest.gitlab-ci.yml
before_script:
- echo "${CI_COMMIT_BRANCH}"
- mkdir -p ~/.ssh
- touch ~/.ssh/known_hosts
- echo "$SSH_PUBLIC_KEY" > ~/.ssh/id_rsa.pub #this is to allow the main repository to get the information from the other ones, all of them are private
- echo "$SSH_PRIVATE_KEY" > ~/.ssh/id_rsa
- echo "$KNOWN_HOSTS" > ~/.ssh/known_hosts
- terraform --version
stages:
- init
init:
stage: init
environment: $CI_COMMIT_BRANCH
script:
- terraform init # just an example
rules:
- if: $CI_COMMIT_BRANCH == "dev" || $CI_COMMIT_BRANCH == "tst" || $CI_COMMIT_BRANCH == "prd"
changes:
- dev/*
- tst/*
- prd/*
I must say this is not the best way to do it, there are some "security" points to mention but they can be solved with a little ingenuity such as: I have understood that the backend.tf shouldn't be explicit in each folder neither the .tfvars file. Someone told me that using terraform enterprise these issues could be solved. Another "dirty thing" about this is that there's kind of duplicated code because each environment folder contains the same main file, outputs and variables.
So far, the way I did it works. I hope this can give you an idea :)
Good luck.