CloudFormation cfn-init breaking in AWS::EC2::LaunchTemplate on Ubuntu base - amazon-web-services

Question

I'm trying to use the CloudFormation cfn-init to bootstrap creation of on-demand compute nodes in a cluster built on Ubuntu 18.04. For some reason, cnf-init enter a dead loop. This is the CloudFormation that I am trying to use:
Resources:
InstanceRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service: ec2.amazonaws.com
Action: sts:AssumeRole
Policies:
- PolicyName: root
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- ec2:DescribeTags
- cloudformation:DescribeStackResource
Resource: '*'
InstanceProfile:
Type: AWS::IAM::InstanceProfile
Properties:
Roles:
- !Ref InstanceRole
LaunchTemplate:
Type: AWS::EC2::LaunchTemplate
Metadata:
AWS::CloudFormation::Init:
configSets:
default:
- "basic"
basic:
files:
/home/ubuntu/.emacs:
content: !Sub |
;; ========== Configuring basic Emacs behavior ==========
;; Try to auto split vertically all the time
(setq split-height-threshold nil)
;; ========== Enable Line and Column Numbering ==========
;; Show line-number in the mode line
(line-number-mode 1)
;; Show column-number in the mode line
(column-number-mode 1)
;; Display size in human format in Dired mode
(setq dired-listing-switches "-alh")
mode: "000644"
owner: "ubuntu"
group: "ubuntu"
packages:
apt:
build-essential: []
emacs-nox: []
Properties:
LaunchTemplateData:
ImageId: ami-07a29e5e945228fa1
IamInstanceProfile:
Arn: !GetAtt [ InstanceProfile, Arn ]
UserData:
Fn::Base64:
!Sub |
#!/bin/bash -x
# Install the aws CloudFormation Helper Scripts
apt-get update -y && apt-get upgrade -y
apt-get install -y python2.7
update-alternatives --install /usr/bin/python python /usr/bin/python2.7 1
curl https://bootstrap.pypa.io/get-pip.py --output get-pip.py
python get-pip.py
rm get-pip.py
pip install https://s3.amazonaws.com/cloudformation-examples/aws-cfn-bootstrap-latest.tar.gz
## Running init stack
cfn-init -v --stack ${AWS::StackName} --resource LaunchTemplate --region ${AWS::Region}
LaunchTemplateName: MyLaunchTemplate
Looking at /var/log/cfn-init.log is not really helpfull
2020-11-11 17:17:59,172 [DEBUG] CloudFormation client initialized with endpoint https://cloudformation.us-west-2.amazonaws.com
2020-11-11 17:17:59,172 [DEBUG] Describing resource LaunchTemplate in stack LaunchTemplate
2020-11-11 17:17:59,237 [ERROR] Throttling: Rate exceeded
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/cfnbootstrap/util.py", line 162, in _retry
return f(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/cfnbootstrap/util.py", line 234, in _timeout
raise exc[0]
HTTPError: 400 Client Error: Bad Request
2020-11-11 17:17:59,237 [DEBUG] Sleeping for 0.143176 seconds before retrying
2020-11-11 17:17:59,381 [DEBUG] Describing resource LaunchTemplate in stack LaunchTemplate
2020-11-11 17:17:59,445 [ERROR] Throttling: Rate exceeded
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/cfnbootstrap/util.py", line 162, in _retry
return f(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/cfnbootstrap/util.py", line 234, in _timeout
raise exc[0]
HTTPError: 400 Client Error: Bad Request
2020-11-11 17:17:59,445 [DEBUG] Sleeping for 1.874780 seconds before retrying
2020-11-11 17:18:01,323 [DEBUG] Describing resource LaunchTemplate in stack LaunchTemplate
Investigating /var/log/cloud-init.log, I can see where it breaks first:
(...)
2020-11-11 17:16:57,175 - util.py[DEBUG]: Running command ['/var/lib/cloud/instance/scripts/part-001'] with allowed return codes [0] (shell=False, capture=False)
2020-11-11 17:21:17,126 - util.py[WARNING]: Failed running /var/lib/cloud/instance/scripts/part-001 [1]
2020-11-11 17:21:17,129 - util.py[DEBUG]: Failed running /var/lib/cloud/instance/scripts/part-001 [1]
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 878, in runparts
File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 2164, in subp
break
cloudinit.util.ProcessExecutionError: Unexpected error while running command.
Command: ['/var/lib/cloud/instance/scripts/part-001']
Exit code: 1
Reason: -
Stdout: -
Stderr: -
(...)
which is the content of the UserData of the template:
$ cat /var/lib/cloud/instance/scripts/part-001
#!/bin/bash -x
# Install the aws CloudFormation Helper Scripts
apt-get update -y && apt-get upgrade -y
apt-get install -y python2.7
update-alternatives --install /usr/bin/python python /usr/bin/python2.7 1
curl https://bootstrap.pypa.io/get-pip.py --output get-pip.py
python get-pip.py
rm get-pip.py
pip install https://s3.amazonaws.com/cloudformation-examples/aws-cfn-bootstrap-latest.tar.gz
## Running init stack
cfn-init -v --stack LaunchTemplate --resource LaunchTemplate --region us-west-2
Even thought I set cloudformation:DescribeStackResource in the InstanceRole, running the script as root returns the following error:
(...)
Successfully built aws-cfn-bootstrap
+ cfn-init -v --stack LaunchTemplate --resource LaunchTemplate --region us-west-2
AccessDenied: Instance i-0bcf477579987a0e8 is not allowed to call DescribeStackResource for LaunchTemplate
This is really strange as when I do the same within a AWS::EC2::Instance using the same AMI work just fine. Any idea what's going on here? What am'I missing?
Thanks

Answer 1

This could be because --resource LaunchTemplate is incorrect. It should the ASG or instance resource that uses the launchtemplate, not the LaunchTemplate itself.