AWS CloudFormation 'UserData' Doesn't seem to work - amazon-web-services

Question

I'm writing an AWS CloudFormation script to build an EC2 instance. I'd like to provision the instance by installing some packages, downloading some repos and running some scripts. Amazon tells me I can do this in CloudFormation with the UserData field. However, it just doesn't seem to work at all.
Here is what i'm working with currently:
DWHServer:
Type: "AWS::EC2::Instance"
Properties:
DisableApiTermination: false # no termination protection
EbsOptimized: false # optimize for elastic block store
IamInstanceProfile: !Ref DWHServerIAMIP
ImageId: "ami-5189a661" # ubunty-trusty-14.04-amd64-server-20150325
InstanceInitiatedShutdownBehavior: "terminate"
InstanceType: "t2.medium"
KeyName: !FindInMap [EnvMap, KeyPair, !Ref EnvType]
Monitoring: true
SecurityGroupIds:
- !Ref DWHServerSG
SourceDestCheck: true # ??
SubnetId: "subnet-aed2ecf6" # Stage-etl-2c
UserData: !Base64
"Fn::Join": ["", ["#!/bin/bash -xe\n", "touch ~/confirm_work.txt\n"]]
This is the most simple example. I just want it to make a file to prove that it's running. But it doesn't even do that. The docs say to look at something called /var/log/cloud-init-output.log. I looked there, but don't see anything about UserData. There does seem to be some sort of network error, but I'm not sure how to interpret it or what to do about it.
Here are the contents of the cloud-init-output.log file on the instance:
Cloud-init v. 0.7.5 running 'init-local' at Sat, 04 Mar 2017 02:40:07 +0000. Up 3.85 seconds.
Cloud-init v. 0.7.5 running 'init' at Sat, 04 Mar 2017 02:40:09 +0000. Up 6.01 seconds.
ci-info: +++++++++++++++++++++++++Net device info+++++++++++++++++++++++++
ci-info: +--------+------+-----------+---------------+-------------------+
ci-info: | Device | Up | Address | Mask | Hw-Address |
ci-info: +--------+------+-----------+---------------+-------------------+
ci-info: | lo | True | 127.0.0.1 | 255.0.0.0 | . |
ci-info: | eth0 | True | 10.0.7.84 | 255.255.255.0 | 0a:3a:b0:a4:96:5d |
ci-info: +--------+------+-----------+---------------+-------------------+
ci-info: ++++++++++++++++++++++++++++++Route info++++++++++++++++++++++++++++++
ci-info: +-------+-------------+----------+---------------+-----------+-------+
ci-info: | Route | Destination | Gateway | Genmask | Interface | Flags |
ci-info: +-------+-------------+----------+---------------+-----------+-------+
ci-info: | 0 | 0.0.0.0 | 10.0.7.1 | 0.0.0.0 | eth0 | UG |
ci-info: | 1 | 10.0.7.0 | 0.0.0.0 | 255.255.255.0 | eth0 | U |
ci-info: +-------+-------------+----------+---------------+-----------+-------+
Mar 4 02:40:11 ubuntu pollinate[723]: ERROR: Network communication failed [60]\n02:40:10.394529 * Hostname was NOT found in DNS cache
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
^M 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 002:40:10.407240 * Trying 91.189.94.24...
02:40:10.550022 * Connected to entropy.ubuntu.com (91.189.94.24) port 443 (#0)
02:40:10.551661 * successfully set certificate verify locations:
02:40:10.551698 * CAfile: /etc/pollinate/entropy.ubuntu.com.pem
CApath: /dev/null
02:40:10.551804 * SSLv3, TLS handshake, Client hello (1):
02:40:10.551832 } [data not shown]
02:40:10.711080 * SSLv3, TLS handshake, Server hello (2):
02:40:10.711129 { [data not shown]
02:40:10.711191 * SSLv3, TLS handshake, CERT (11):
02:40:10.711216 { [data not shown]
02:40:10.711490 * SSLv3, TLS alert, Server hello (2):
02:40:10.711520 } [data not shown]
02:40:10.711602 * SSL certificate problem: unable to get local issuer certificate
^M 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
02:40:10.711732 * Closing connection 0
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: http://curl.haxx.se/docs/sslcerts.html
curl performs SSL certificate verification by default, using a "bundle"
of Certificate Authority (CA) public keys (CA certs). If the default
bundle file isn't adequate, you can specify an alternate file
using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
the bundle, the certificate verification probably failed due to a
problem with the certificate (it might be expired, or the name might
not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
the -k (or --insecure) option.
2017-03-04 02:40:11,144 - util.py[WARNING]: Running seed_random (<module 'cloudinit.config.cc_seed_random' from '/usr/lib/python2.7/dist-packages/cloudinit/config/cc_seed_random.pyc'>) failed
Generating public/private rsa key pair.
Your identification has been saved in /etc/ssh/ssh_host_rsa_key.
Your public key has been saved in /etc/ssh/ssh_host_rsa_key.pub.
The key fingerprint is:
0c:54:09:ab:bc:b8:63:b5:6c:d2:d5:47:21:4a:38:6f root#ip-10-0-7-84
The key's randomart image is:
+--[ RSA 2048]----+
| .oo.. |
| o...o . |
| +o. . . |
| . .Eo . |
| o. .S. |
| .... . . |
| .+.o . |
| +.= |
| ..+ |
+-----------------+
Generating public/private dsa key pair.
Your identification has been saved in /etc/ssh/ssh_host_dsa_key.
Your public key has been saved in /etc/ssh/ssh_host_dsa_key.pub.
The key fingerprint is:
89:26:94:17:79:6d:45:15:fc:5f:37:95:31:2e:e9:f7 root#ip-10-0-7-84
The key's randomart image is:
+--[ DSA 1024]----+
| .. . oooo+o|
| .... o +.o|
| o .. . o o.|
| . . . . . ..+|
| . o S . .=|
| o . o|
| E|
| |
| |
+-----------------+
Generating public/private ecdsa key pair.
Your identification has been saved in /etc/ssh/ssh_host_ecdsa_key.
Your public key has been saved in /etc/ssh/ssh_host_ecdsa_key.pub.
The key fingerprint is:
af:a2:c7:b3:95:5c:17:2e:ce:69:b3:f6:39:c7:67:91 root#ip-10-0-7-84
The key's randomart image is:
+--[ECDSA 256]---+
| |
| |
| . |
| . . |
| S o o .|
| . * + E |
| . + B . .|
| =. o.o..o o|
| .o.+....oo o |
+-----------------+
Cloud-init v. 0.7.5 running 'modules:config' at Sat, 04 Mar 2017 02:40:14 +0000. Up 11.53 seconds.
Generating locales... en_US.UTF-8... up-to-date
Generation complete.
Cloud-init v. 0.7.5 running 'modules:final' at Sat, 04 Mar 2017 02:40:17 +0000. Up 13.61 seconds.
+ touch /root/confirm_work.txt
Cloud-init v. 0.7.5 finished at Sat, 04 Mar 2017 02:40:17 +0000. Datasource DataSourceEc2. Up 13.83 seconds
Any tips would be greatly appreciated. Thanks!

Answer 1

Look at the second to last entry in the log:
+ touch /root/confirm_work.txt
The command is indeed invoked. Note that all commands in your EC2 user data will show up in that log file (/var/log/cloud-init-output.log) with a PLUS sign prepended to it (like above). Is it possible that the touch command is not there? That would be surprising. But add an "echo" command before the touch, you should see the output and that would confirm that it's all working fine. Maybe you're trying to touch in a directory you don't have access. Maybe try to touch a file in /tmp to narrow things down...

Answer 2

Protip: Always use fully qualified paths in scripts. Try this for your userdata. Does it help?
UserData: !Base64
"Fn::Join": ["\n", ["#!/bin/bash -xe", "/bin/touch /tmp/confirm_work.txt"]]