I'm having a set of Terraform files and in particular one variables.tf file which sort of holds my variables like aws access key, aws access token etc. I want to now automate the resource creation on AWS using GitLab CI / CD.
My plan is the following:
Write a .gitlab-ci-yml file
Have the terraform calls in the .gitlab-ci.yml file
I know that I can have secret environment variables in GitLab, but I'm not sure how I can push those variables into my Terraform variables.tf file which looks like this now!
# AWS Config
variable "aws_access_key" {
default = "YOUR_ADMIN_ACCESS_KEY"
}
variable "aws_secret_key" {
default = "YOUR_ADMIN_SECRET_KEY"
}
variable "aws_region" {
default = "us-west-2"
}
In my .gitlab-ci.yml, I have access to the secrets like this:
- 'AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}'
- 'AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}'
- 'AWS_DEFAULT_REGION=${AWS_DEFAULT_REGION}'
How can I pipe it to my Terraform scripts? Any ideas? I would need to read the secrets from GitLab's environment and pass it on to the Terraform scripts!
Which executor are you using for your GitLab runners?
You don't necessarily need to use the Docker executor but can use a runner installed on a bare-metal machine or in a VM.
If you install the gettext package on the respective machine/VM as well you can use the same method as I described in Referencing gitlab secrets in Terraform for the Docker executor.
Another possibility could be that you set
job:
stage: ...
variables:
TF_VAR_SECRET1: ${GITLAB_SECRET}
or
job:
stage: ...
script:
- export TF_VAR_SECRET1=${GITLAB_SECRET}
in your CI job configuration and interpolate these. Please see Getting an Environment Variable in Terraform configuration? as well
Bear in mind that terraform requires a TF_VAR_ prefix to environment variables. So actually you need something like this in .gitlab-ci.yml
- 'TF_VAR_AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}'
- 'TF_VAR_AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}'
- 'TF_VAR_AWS_DEFAULT_REGION=${AWS_DEFAULT_REGION}'
Which also means you could just set the variable in the pipeline with that prefix as well and not need this extra mapping step.
I see you actually did discover this per your comment---I'm still posting this answer since I missed your comment the first time and it would have saved me an hour of work.
Related
I had old Terraform configuration, worked perfect.
In short, I had static website application I needed to deploy using Cloudfront & S3. Then, I need another application to deploy in the same way, but in other sub-domain.
For ease of helping, you can check the full source code here:
Old Terraform configuration: https://github.com/tal-rofe/tf-old
New Terraform configuration: https://github.com/tal-rofe/tf-new
So, my domain is example.io, and in the old configuration I had only static application deployed on app.example.com.
But, as I need an another application, it's going to be deployed on docs.example.com.
To avoid a lot of code duplication, I decided on creating a local module for deploying a generic application onto Cloudfront & S3.
After doing so, seems like terraform apply and terraform plan succeeds (not really, as no resources were changed at all!): Apply complete! Resources: 0 added, 0 changed, 0 destroyed.
Not only no changes, but I get an old output:
cloudfront_distribution_id = "blabla"
eks_kubeconfig = <sensitive>
This cloudfront_distribution_id output, was the correct output using the old configuration. I expect to get these new outputs, as configured:
output "frontend_cloudfront_distribution_id" {
description = "The distribution ID of deployed Cloudfront frontend"
value = module.frontend-static.cloudfront_distribution_id
}
output "docs_cloudfront_distribution_id" {
description = "The distribution ID of deployed Cloudfront docs"
value = module.docs-static.cloudfront_distribution_id
}
output "eks_kubeconfig" {
description = "EKS Kubeconfig content"
value = module.eks-kubeconfig.kubeconfig
sensitive = true
}
I'm using GitHub actions to apply my Terraform configuration with these steps:
- name: Terraform setup
uses: hashicorp/setup-terraform#v2
with:
terraform_wrapper: false
- name: Terraform core init
env:
TERRAFORM_BACKEND_S3_BUCKET: ${{ secrets.TERRAFORM_BACKEND_S3_BUCKET }}
TERRAFORM_BACKEND_DYNAMODB_TABLE: ${{ secrets.TERRAFORM_BACKEND_DYNAMODB_TABLE }}
run: |
terraform -chdir="./terraform/core" init \
-backend-config="bucket=$TERRAFORM_BACKEND_S3_BUCKET" \
-backend-config="dynamodb_table=$TERRAFORM_BACKEND_DYNAMODB_TABLE" \
-backend-config="region=$AWS_REGION"
- name: Terraform core plan
run: terraform -chdir="./terraform/core" plan -no-color -out state.tfplan
- name: Terraform core apply
run: terraform -chdir="./terraform/core" apply state.tfplan
I used the same steps in my old & new Terraform configurations.
I want to re-use the logic written in my static-app module twice. So basically I want to be able to create static application just by using the module I've configured.
You cannot define the outputs in the root module and expect it to work because you are already using a different module in your static-app module (i.e., you are nesting modules). Since you are using the terraform module there (denoted with source = "terraform-aws-modules/cloudfront/aws") you are limited to what that module provides as outputs and hence can only define those outputs on the module level, not root level. I see you are referencing the EKS output works, but the difference here is that that particular module is not nested and is called directly (from your repo):
module "eks" {
source = "terraform-aws-modules/eks/aws"
version = "19.5.1"
.
.
.
}
The way I would suggest fixing this is to call the Cloudfront module from the root module (i.e., core in your example):
module "frontend-static" {
source = "terraform-aws-modules/cloudfront/aws"
version = "3.1.0"
... rest of the configuration ...
}
module "docs-static" {
source = "terraform-aws-modules/cloudfront/aws"
version = "3.1.0"
... rest of the configuration ...
}
The outputs you currently have defined in your repo with new configuration (tf-new) should work out-of-the-box with this change. Alternatively, you could write your own module and then you can control which outputs you will have.
thanks in advance for any help/guidance you could provide.
Context
I am currently using CDK in a project to create AWS resources (a few Lambda functions) and SAM to test locally, this works wonderfully but I'm struggling with environment variables to be used locally with my setup of CDK + SAM.
I run and test the project locally via the command
$ cdk synth --no-staging > template.yaml && sam local start-api
Deployments are done via
$ cdk deploy testStack123 --context secretToken=123
The issue came when I had to include (locally) a sensitive token required for my project and I couldn't figure out how to differentiate like how you would in a project, for example, that only uses AWS SAM where you can define:
your local environment variables via a env.json file
and your environment variables you want to use for deployment that you'd pass in via
$ sam deploy --stack-name=testStack123 --secretToken=123
What I tried?
Sam's --env-vars command such as:
$ sam local start-api --env-vars env.json
but since I'm not managing the template.yaml myself instead I'm relaying on CDK's synth command to output the CloudFormation, there is no way I can reliably reference the Lambda function names in the env.json to pass local environment variables via --env-vars env.json.
// env.json example
{
"TestLambdaFunction": { // Will fail as its referenced in template.yaml as TestLambdaFunction67CA3BED for example
"SECRET_TOKEN": "123"
"ENVIRONMENT": "test"
},
}
I tried the runtime context via cdk.json that the AWS team suggests for CDK envrionments but I noticed that it is also required to push the cdk.json file to the repo, so you always ran the risk of the dev not noticing that they’re accidentally staging and pushing sensitive tokens to the repo. However, this solution would work for both CI and local dev, but comes with the risk mentioned before.
Any advice on how to best solve this so I can make it so local environments can safely be passed via an (git)ignored filed such as env.json but actually work by referencing the Lambdas correctly that are emitted by the synthesised CloudFormation template cia cdk synth.
The generated Logical IDs like TestLambdaFunction67CA3BED are stable (unless you change the Construct ID or construct tree), so are generally fine to use in env.json Alternatively, you can place all the env vars under a Parameters key:
// env.json
{
"Parameters": {
"TABLE_NAME": "localtable",
"BUCKET_NAME": "testBucket",
"STAGE": "dev"
}
}
I've tried to run the following commands as part of a bash script runs in BashOperator:
aws cli ls s3://bucket
aws cli cp ... ...
The script runs successfully, however the aws cli commands return error, showing that aws cli doesn't run with the needed permissions (as was defined in airflow-worker-node role)
Investigating the error:
I've upgraded awscli in the docker running the pod - to version 2.4.9 (I've understood that old version of awscli doesn't support access to s3 based on permission grant by aws role
I've Investigated the pod running my bash_script using the BashOperator:
Using k9s, and D (describe) command:
I saw that ARN_ROLE is defined correctly
Using k9s, and s (shell) command:
I saw that pod environment variables are correct.
aws cli worked with the needed permissions and can access s3 as needed.
aws sts get-caller-identity - reported the right role (airflow-worker-node)
Running the above commands as part of the bash-script which was executed in the BashOperator gave me different results:
Running env showed limited amount of env variables
aws cli returned permission related error.
aws sts get-caller-identity - reported the eks role (eks-worker-node)
How can I grant aws cli in my BashOperator bash-script the needed permissions?
Reviewing the BashOperator source code, I've noticed the following code:
https://github.com/apache/airflow/blob/main/airflow/operators/bash.py
def get_env(self, context):
"""Builds the set of environment variables to be exposed for the bash command"""
system_env = os.environ.copy()
env = self.env
if env is None:
env = system_env
else:
if self.append_env:
system_env.update(env)
env = system_env
And the following documentation:
:param env: If env is not None, it must be a dict that defines the
environment variables for the new process; these are used instead
of inheriting the current process environment, which is the default
behavior. (templated)
:type env: dict
:param append_env: If False(default) uses the environment variables passed in env params
and does not inherit the current process environment. If True, inherits the environment variables
from current passes and then environment variable passed by the user will either update the existing
inherited environment variables or the new variables gets appended to it
:type append_env: bool
If bash operator input env variables is None, it copies the env variables of the father process.
In my case, I provided some env variables therefore it didn’t copy the env variables of the father process into the chid process - which caused the child process (the BashOperator process) to use the default arn_role of eks-worker-node.
The simple solution is to set the following flag in BashOperator(): append_env=True which will append all existing env variables to the env variables I added manually.
I've figured out that in the version I'm running (2.0.1) it isn't supported (it is supported in later versions).
As a temp solution I've add **os.environ - to the BashOperator env parameter:
return BashOperator(
task_id="copy_data_from_mcd_s3",
env={
"dag_input": "{{ dag_run.conf }}",
......
**os.environ,
},
# append_env=True,- should be supported in 2.2.0
bash_command="utils/my_script.sh",
dag=dag,
retries=1,
)
Which solve the problem.
I am using AWS SAM.
I have created a samconfig.toml file with the following entry:
[default.build.parameters]
container_env_var_file = "envDefault.json"
When I do sam build I see in .aws-sam/build.toml
The env values from envDefault.json
But when I check the template .aws-sam/build/template.yaml
I see the original values, not the overwrites I have in envDefault.json
What is the best way to sam deploy with overwrites of the env variables for each environment I am deploying to?
I am trying to avoid entering parameters manually during the deploy process.
It seems you are trying to set build parameters not deploy parameters.
All the build parameters section in AWS SAM toml file starts with, [default.build.parameters]
And for the overriding deployment parameters, you have to use [default.deploy.parameters] section (default denotes the default profile).
So in your case, it will be like this.
[default.deploy.parameters]
parameter_overrides = "ParamKey=\"ParamValue\""
or you can use command line,
sam deploy --parameter-overrides
But if you are trying to get some values for your lambda functions based on environment, I strongly suggest the usage of parameter store.
Currently our singleton application including 5 containers goes through AWS pipeline into code build and then code deploy into ECS services. During codebuild base on an ENV set in codebuild $Stage it can be dev, prod or staging and loads a specific config file for which contains all the ENV variables each container needs. See below:
build:
commands:
#Get commit id
- "echo STAGE $STAGE"
- "export STAGE=$STAGE"
#Assigning AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY needs to be done in two steps, otherwise it ends up in "Partial credentials found in env" error
- "export ANSIBLE_VARS=\"\
USE_EXISTING_VPC=true \
DISABLE_BASIC_AUTH=true\""
- "export DOCKER_ARGS=\"-e COMMIT_ID=$GIT_COMMIT -e APP_ENV=$STAGE
Problem 1: is these config files are within the repo and anybody can modify them. So there are lots of human errors like the production redirect Url is pointing to the wrong place, or new ENV is not set.
So I want to move away from loading different config files and move ENV variables to AWS to handle. Something like during code build it will load from parameter store. Is this correct way?
Problem 2 is there are lots of ENV variables, is the only option to list them one by one in the CloudFormation template ? Are there any other better way to load all of ENV variable into DOCKER_ARG from above build command ?