Terraform unable to assume roles with MFA enabled - amazon-web-services

Question

I'm having a terrible time getting Terraform to assume an IAM role with another account with MFA required. Here's my setup
AWS Config
[default]
region = us-west-2
output = json
[profile GEHC-000]
region = us-west-2
output = json
....
[profile GEHC-056]
source_profile = GEHC-000
role_arn = arn:aws:iam::~069:role/hc/hc-master
mfa_serial = arn:aws:iam::~183:mfa/username
external_id = ~069
AWS Credentials
[default]
aws_access_key_id = xxx
aws_secret_access_key = xxx
[GEHC-000]
aws_access_key_id = same as above
aws_secret_access_key = same as above
Policies assigned to IAM user
STS Policy
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AssumeRole",
"Effect": "Allow",
"Action": [
"sts:AssumeRole"
],
"Resource": [
"arn:aws:iam::*:role/hc/hc-master"
]
}
]
}
User Policy
{
"Statement": [
{
"Action": [
"iam:*AccessKey*",
"iam:*MFA*",
"iam:*SigningCertificate*",
"iam:UpdateLoginProfile*",
"iam:RemoveUserFromGroup*"
],
"Effect": "Allow",
"Resource": [
"arn:aws:iam::~183:mfa/${aws:username}",
"arn:aws:iam::~183:mfa/*/${aws:username}",
"arn:aws:iam::~183:mfa/*/*/${aws:username}",
"arn:aws:iam::~183:mfa/*/*/*${aws:username}",
"arn:aws:iam::~183:user/${aws:username}",
"arn:aws:iam::~183:user/*/${aws:username}",
"arn:aws:iam::~183:user/*/*/${aws:username}",
"arn:aws:iam::~183:user/*/*/*${aws:username}"
],
"Sid": "Write"
},
{
"Action": [
"iam:*Get*",
"iam:*List*"
],
"Effect": "Allow",
"Resource": [
"*"
],
"Sid": "Read"
},
{
"Action": [
"iam:CreateUser*",
"iam:UpdateUser*",
"iam:AddUserToGroup"
],
"Effect": "Allow",
"Resource": [
"*"
],
"Sid": "CreateUser"
}
],
"Version": "2012-10-17"
}
Force MFA Policy
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "BlockAnyAccessOtherThanAboveUnlessSignedInWithMFA",
"Effect": "Deny",
"NotAction": "iam:*",
"Resource": "*",
"Condition": {
"BoolIfExists": {
"aws:MultiFactorAuthPresent": "false"
}
}
}
]
}
main.tf
provider "aws" {
profile = "GEHC-056"
shared_credentials_file = "${pathexpand("~/.aws/config")}"
region = "${var.region}"
}
data "aws_iam_policy_document" "test" {
statement {
sid = "TestAssumeRole"
effect = "Allow"
actions = [
"sts:AssumeRole",
]
principals = {
type = "AWS"
identifiers = [
"arn:aws:iam::~183:role/hc-devops",
]
}
sid = "BuUserTrustDocument"
effect = "Allow"
principals = {
type = "Federated"
identifiers = [
"arn:aws:iam::~875:saml-provider/ge-saml-for-aws",
]
}
condition {
test = "StringEquals"
variable = "SAML:aud"
values = ["https://signin.aws.amazon.com/saml"]
}
}
}
resource "aws_iam_role" "test_role" {
name = "test_role"
path = "/"
assume_role_policy = "${data.aws_iam_policy_document.test.json}"
}
Get Caller Identity
bash-4.4$ aws --profile GEHC-056 sts get-caller-identity
Enter MFA code for arn:aws:iam::772660252183:mfa/503072343:
{
"UserId": "AROAIWCCLC2BGRPQMJC7U:botocore-session-1537474244",
"Account": "730993910069",
"Arn": "arn:aws:sts::730993910069:assumed-role/hc-master/botocore-session-1537474244"
}
And the error:
bash-4.4$ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.
Error: Error refreshing state: 1 error(s) occurred:
* provider.aws: Error creating AWS session: AssumeRoleTokenProviderNotSetError: assume role with MFA enabled, but AssumeRoleTokenProvider session option not set.

Answer 1

Terraform doesn't currently support prompting for the MFA token when being ran as it is intended to be ran in a less interactive fashion as much as possible and it would apparently require significant rework of the provider structure to support this interactive provider configuration. There's more discussion about this in this issue.
As also mentioned in that issue the best bet is to use some form of script/tool that already assumes the role prior to running Terraform.
I personally use AWS-Vault and have written a small shim shell script that I symlink to from terraform (and other things such as aws that I want to use AWS-Vault to grab credentials for) that detects what it's being called as, finds the "real" binary using which -a, and then uses AWS-Vault's exec to run the target command with the specified credentials.
My script looks like this:
#!/bin/bash
set -eo pipefail
# Provides a shim to override target executables so that it is executed through aws-vault
# See https://github.com/99designs/aws-vault/blob/ae56f73f630601fc36f0d68c9df19ac53e987369/USAGE.md#overriding-the-aws-cli-to-use-aws-vault for more information about using it for the AWS CLI.
# Work out what we're shimming and then find the non shim version so we can execute that.
# which -a returns a sorted list of the order of binaries that are on the PATH so we want the second one.
INVOKED=$(basename $0)
TARGET=$(which -a ${INVOKED} | tail -n +2 | head -n 1)
if [ -z ${AWS_VAULT} ]; then
AWS_PROFILE="${AWS_DEFAULT_PROFILE:-read-only}"
(>&2 echo "Using temporary credentials from ${AWS_PROFILE} profile...")
exec aws-vault exec "${AWS_PROFILE}" --assume-role-ttl=60m -- "${TARGET}" "$#"
else
# If AWS_VAULT is already set then we want to just use the existing session instead of nesting them
exec "${TARGET}" "$#"
fi
It will use a profile in your ~/.aws/config file that matches the AWS_DEFAULT_PROFILE environment variable you have set, defaulting to a read-only profile which may or may not be a useful default for you. This makes sure that AWS-Vault assumes the IAM role, grabs the credentials and sets them as environment variables for the target process.
This means that as far as Terraform is concerned it is being given credentials via environment variables and this just works.

Answer 2

One other way is to use credential_process in order to generate the credentials with a local script and cache the tokens in a new profile (let's call it tf_temp)
This script would :
check if the token is still valid for the profile tf_temp
if token is valid, extract the token from existing config using aws configure get xxx --profile tf_temp
if token is not valid, prompt use to enter mfa token
generate the session token with aws assume-role --token-code xxxx ... --profile your_profile
set the temporary profile token tf_temp using aws configure set xxx --profile tf_temp
You would have:
~/.aws/credentials
[prod]
aws_secret_access_key = redacted
aws_access_key_id = redacted
[tf_temp]
[tf]
credential_process = sh -c 'mfa.sh arn:aws:iam::{account_id}:role/{role} arn:aws:iam::{account_id}:mfa/{mfa_entry} prod 2> $(tty)'
mfa.sh
gist
move this script in /bin/mfa.sh or /usr/local/bin/mfa.sh :
#!/bin/sh
set -e
role=$1
mfa_arn=$2
profile=$3
temp_profile=tf_temp
if [ -z $role ]; then echo "no role specified"; exit 1; fi
if [ -z $mfa_arn ]; then echo "no mfa arn specified"; exit 1; fi
if [ -z $profile ]; then echo "no profile specified"; exit 1; fi
resp=$(aws sts get-caller-identity --profile $temp_profile | jq '.UserId')
if [ ! -z $resp ]; then
echo '{
"Version": 1,
"AccessKeyId": "'"$(aws configure get aws_access_key_id --profile $temp_profile)"'",
"SecretAccessKey": "'"$(aws configure get aws_secret_access_key --profile $temp_profile)"'",
"SessionToken": "'"$(aws configure get aws_session_token --profile $temp_profile)"'",
"Expiration": "'"$(aws configure get expiration --profile $temp_profile)"'"
}'
exit 0
fi
read -p "Enter MFA token: " mfa_token
if [ -z $mfa_token ]; then echo "MFA token can't be empty"; exit 1; fi
data=$(aws sts assume-role --role-arn $role \
--profile $profile \
--role-session-name "$(tr -dc A-Za-z0-9 </dev/urandom | head -c 20)" \
--serial-number $mfa_arn \
--token-code $mfa_token | jq '.Credentials')
aws_access_key_id=$(echo $data | jq -r '.AccessKeyId')
aws_secret_access_key=$(echo $data | jq -r '.SecretAccessKey')
aws_session_token=$(echo $data | jq -r '.SessionToken')
expiration=$(echo $data | jq -r '.Expiration')
aws configure set aws_access_key_id $aws_access_key_id --profile $temp_profile
aws configure set aws_secret_access_key $aws_secret_access_key --profile $temp_profile
aws configure set aws_session_token $aws_session_token --profile $temp_profile
aws configure set expiration $expiration --profile $temp_profile
echo '{
"Version": 1,
"AccessKeyId": "'"$aws_access_key_id"'",
"SecretAccessKey": "'"$aws_secret_access_key"'",
"SessionToken": "'"$aws_session_token"'",
"Expiration": "'"$expiration"'"
}'
Use the tf profile in provider settings. The first time, you will be prompted mfa token :
# terraform apply
Enter MFA token: 428313
This solution works fine with terraform and/or terragrunt

Answer 3

I've used a very simple, albeit perhaps dirty, solution to work around this:
First, let TF pick credentials from environment variables. Then:
AWS credentials file:
[access]
aws_access_key_id = ...
aws_secret_access_key = ...
region = ap-southeast-2
output = json
[target]
role_arn = arn:aws:iam::<target nnn>:role/admin
source_profile = access
mfa_serial = arn:aws:iam::<access nnn>:mfa/my-user
In console
CREDENTIAL=$(aws --profile target sts assume-role \
--role-arn arn:aws:iam::<target nnn>:role/admin --role-session-name TFsession \
--output text \
--query "Credentials.[AccessKeyId,SecretAccessKey,SessionToken,Expiration]")
<enter MFA>
#echo "CREDENTIAL: ${CREDENTIAL}"
export AWS_ACCESS_KEY_ID=$(echo ${CREDENTIAL} | cut -d ' ' -f 1)
export AWS_SECRET_ACCESS_KEY=$(echo ${CREDENTIAL} | cut -d ' ' -f 2)
export AWS_SESSION_TOKEN=$(echo ${CREDENTIAL} | cut -d ' ' -f 3)
terraform plan
UPDATE: a better solution is to use https://github.com/remind101/assume-role to achieve the same outcome.

Answer 4

I personally use aws-vault and it works very well with terraform while IAM MFA is enabled.
Install aws-vault: https://github.com/99designs/aws-vault.git
add(store) your AWS credential to the aws-vault: aws-vault add profilename
update your ~/.aws/config file, add "mfa_serial" and "role_arn" information.
[profile <profilename>]
region = <region>
mfa_serial = arn:aws:iam::<AWSAccountA>:mfa/<username>
role_arn = arn:aws:iam::<AWSAccountB>:role/<rolename>
use the following command to run terraform command:
$aws-vault exec <profilename> -- terraform apply
$<input your mfa code>
done.