I have an issue with Terraform provisioning. When I run terraform first time I am using SSH key generated in AWS console. This key is being added to ubuntu user (it's Ubuntu 16.04 AMI). Then I run remote-exec provisioning:
provisioner "remote-exec" {
inline = [
"sudo apt -y update && sudo apt install -y python"
]
connection {
user = "ubuntu"
private_key = "${file("${var.aws_default_key_name}.pem")}"
}
}
I need python being installed so I can use Ansible later. That's the only place where I need this key, never more, because I create my own user with my private key. However, when I try to run terraform later it searches for a file file("${var.aws_default_key_name}.pem".
Now I have a question how to skip this provisioning on subsequent runs?
I don't want to store SSH key in the repository.
I could create an empty file to "trick" terraform, but I don't like this solution.
Any better ideas?
Instead of doing provisioning in the aws_instance block, move it out to a null_resource block, with appropriate triggers.
resource "aws_instance" "cluster" {
count = 3
# ...
}
resource "null_resource" "cluster" {
# Changes to any instance of the cluster requires re-provisioning
triggers {
cluster_instance_ids = "${join(",", aws_instance.cluster.*.id)}"
}
connection {
host = "${element(aws_instance.cluster.*.public_ip, 0)}"
}
provisioner "remote-exec" {
inline = [something]
}
}
If your triggers do not change the null_resource provisioning will not be triggered on subsequent runs.
Sparrowform is a lightweight provisioner for Terraform based infrastructure. The benefits against other provision tools, is that stage of terraform apply which does infrastructure bootstrap is decoupled from provision stage, so you may do this:
$ terraform apply # does infra bootstrap
$ nano sparrowfile # Sparrowdo equivalent for remote-exec chunk
#!/usr/bin/env perl6
bash 'apt -y update';
package-install 'python';
$ sparrowform --ssh_user=my-user --ssh_private_key=/path/to/key # do provision stage
Obviously you are free not to run sparrowform in subsequent runs. It does it's job (install ansible related dependencies, that is it). Then you drop your initial ssh_private_key and go with new private key ( ansible related I guess ?)
PS. disclosure - I am the tool author
Related
I have an already created EC2 instance using terraform, I want to turn it off/on just modifying a variable and using terraform apply -auto-approve, this is done already (check the code below), the issue I have is that I need that terraform refresh itself after the local-exec execution and after the instance has changed its state from on->off or vice versa because I want terraform to save the state of the last action (ec2 off or on).
This is what I have so far:
resource "null_resource" "change_instance_state" {
count = var.instance-state == "keep" ? 0 : var.instance_count
provisioner "local-exec" {
on_failure = fail
interpreter = ["/bin/bash", "-c"]
command = <<EOT
echo "Warning! Performing the --> ${lookup(var.instance-state-map, var.instance-state)} <-- operation on instance/s having:"
echo "ec2-id: ${aws_instance.dev_node[count.index].id}"
echo "ec2-ip: ${aws_instance.dev_node[count.index].public_ip}"
echo "ec2-public dns: ${aws_instance.dev_node[count.index].public_dns}"
# Performing the action
aws ec2 ${lookup(var.instance-state-map, var.instance-state)} --instance-ids ${aws_instance.dev_node[count.index].id}
echo "******* Action Performed *******"
EOT
}
# this setting will trigger the script every time var.instance-state change
triggers = {
last-modified-state = "${var.instance-state}"
}
}
Notes:
I tried to update the state adding terraform apply -refresh-only -auto-approve at the end of the script but, you all know that terraform locked itself and this action can't be done while terraform is running.
My end goal is to create a 3 node kubeadm kubernetes cluster with terraform and/or ansible.
As of now I am provisioning three identical instances with terraform.
Then with remote-exec and inline installing packages that all instances share between themselves.
Now I want to install specific packages only on one of those three instances. Trying to achieve this using local-exec.
I am struggling with connecting only to 1 instance with local-exec. I know how to connect to all of them and execute playbook against three instances. But the end goal is to connect to one instance only.
the code snipped:
resource "aws_instance" "r100c96" {
count = 3
ami = "ami-0b9064170e32bde34"
instance_type = "t2.micro"
key_name = local.key_name
tags = {
Name = "terra-ans${count.index}"
}
provisioner "remote-exec" {
connection {
host = "${self.public_ip}"
type = "ssh"
user = local.ssh_user
private_key = file(local.private_key_path)
}
inline = ["sudo hostnamectl set-hostname test"]
}
provisioner "local-exec" {
command = "ansible-playbook -i ${element((aws_instance.r100c96.*.public_ip),0)}, --private-key ${local.private_key_path} helm.yaml"
}
...
}
Thanks,
You can use null_resource, and and run your remote-exec for selected instance only, once all three instances in aws_instance.r100c96 are provisioned.
I think instead of * use the count.index, on every loop run it will pass the specific VM IP.
Also, there are multiple ways to provision a VM using Ansible.
Consider if you can dynamically build your Hosts file and provision them in parallel instead one at a time.
As we are able to display predefined variables aws_instance.my-instance.public_ip values through output variables at the end of the execution of terraform apply.
Similar way, is there a way to output custom information from the new instance at the end such as output a system file any command output such echo hello! or cat /var/log/hello.log?
You can use Terraform Provisioners. They are basically interface to run commands and script to remote machine (or local depending over the provisioner) to achieve some tasks, which in most cases will be bootstrapping matters.
resource "aws_instance" "example" {
ami = "ami-b374d5a5"
instance_type = "t2.micro"
provisioner "local-exec" {
command = "echo ${aws_instance.example.public_ip} > ip_address.txt"
}
}
You can read more about them here: https://learn.hashicorp.com/terraform/getting-started/provision
However, keep in mind that provisioners are terraform objects and not bound to instances, so they only execute when you use Terraform to spin up or edit instances. These bootstrapping scripts wont come into effect if your instance is created by an ASG during an scale-out operation or by an orchestration tool. For that purpose, using instance's user_data is the best option.
As #SajjadHashmi, you can use local-exec to run commands on your local host with some limitations.
Thus in desperation you can use ssh and scp commands on your local host to get the files from from the instance and execute commands there. This is not a very nice way, but as a measure of the last resort could be considered in some scenarios.
resource "aws_instance" "web" {
# other attributes
provisioner "local-exec" {
command = <<-EOL
# give time to instance to properly boot
sleep 30
# download a /var/log/cloud-init-output.log
# file from the instance to the host's /tmp folder
scp -i private_ssh_key -oStrictHostKeyChecking=no ubuntu#${self.public_ip}:/var/log/cloud-init-output.log /tmp
# execute command ls -a on the instance and save output to
# local file /tmp/output.txt
ssh -i private_ssh_key -oStrictHostKeyChecking=no ubuntu#${self.public_ip} ls -a >> /tmp/output.txt
EOL
}
}
I've been struggling with this issue for the last few days. I have an instance that is created using a terraform template with userdata that is specified from a template file. The instance uses the Debian Jessie community AMI and I am able to view the user data on the instance through wget. I've tried a copy of the AMI, I've tried using #cloud-boothook, and I've tried putting the userdata script inside the main TF template.
I checked the cloud-init-output.log logs and there is an error that my research indicates to be a problem with environment variables and sudo but I would still expect that the temp file would get created because there is no sudo call preceeding that echo line (from here:user data scripts fails without giving reason).
util.py[WARNING]: Running scripts-user (<module 'cloudinit.config.cc_scripts_user' from '/usr/lib/python2.7/dist-packages/cloudinit/config/cc_scripts_user.pyc'>) failed
Section of my TF template that creates the instance:
resource "aws_instance" "example" {
ami = "ami-116d857a"
instance_type = "t2.micro"
source_dest_check = "False"
subnet_id = "${aws_subnet.public.id}"
key_name = "my-generic-keyname"
vpc_security_group_ids = ["${aws_security_group.vpn-sg.id}"]
user_data = "${data.template_file.bootscript.rendered}"
depends_on = ["aws_subnet.public"]
}
User data contained in the template file:
#!/bin/bash
echo 'Running user data' > /tmp/user-script-output.txt
sudo apt install strongswan-starter -y
sudo apt install curl -y
# Write secrets file
[cat <<-EOF > /etc/ipsec.secrets
# This file holds shared secrets or RSA private keys for authentication.
# RSA private key for this host, authenticating it to any other host
# which knows the public part.
privateip : PSK "$clientPSK"
EOF
Expected Behavior:
When I SSH into the instance, I can view the metadata no problem but the file in /tmp isn't created, curl is not installed, and neither is the strongswan package.
I appreciate the help!
I am currently migrating my config management on AWS to Terraform to make it more pluggable. What I like is the possibility to manage rolling updates to an Autoscaling Group where Terraform waits until the new instances are in service before it destroys the old infrastructure.
This works fine with the "bare" infrastructure. But I ran into a problem when update the actual app instances.
The code is deployed via AWS CodeDeploy and I can tell Terraform to use the generated name of the new Autoscaling Group as deployment target but it doesn't deploy the code to the new instances on startup. When I manually select "deploy changes to the deployment group" the deployment starts successfully.
Any ideas how to automate this step?
https://www.terraform.io/docs/provisioners/local-exec.html might be able to do this. Couple assumptions
You've got something like aws-cli installed where you're running terraform.
You've got your dependencies setup so that your CodeDeploy step would be one of the last things executed. If that's not the case you can play with depends_on https://www.terraform.io/intro/getting-started/dependencies.html#implicit-and-explicit-dependencies
Once your code has been posted, you would just add a
resource "something" "some_name" {
# Whatever config you've setup for the resource
provisioner "local-exec" {
command = "aws deploy create-deployment"
}
}
FYI the aws deploy create-deployment command is not complete, so you'll have to play with that in your environment till you've got the values needed to trigger the rollout but hopefully this is enough to get you started.
You can trigger the deployment directly in your user-data in the
resource "aws_launch_configuration" "my-application" {
name = "my-application"
...
user_data = "${data.template_file.node-init.rendered}"
}
data "template_file" "node-init" {
template = "${file("${path.module}/node-init.yaml")}"
}
Content of my node-init.yaml, following recommendations of this documentation: https://aws.amazon.com/premiumsupport/knowledge-center/codedeploy-agent-launch-configuration/
write_files:
- path: /root/configure.sh
content: |
#!/usr/bin/env bash
REGION=$(curl 169.254.169.254/latest/meta-data/placement/availability-zone/ | sed 's/[a-z]$//')
yum update -y
yum install ruby wget -y
cd /home/ec2-user
wget https://aws-codedeploy-$REGION.s3.amazonaws.com/latest/install
chmod +x ./install
./install auto
# Add the following line for your node to update itself
aws deploy create-deployment --application-name=<my-application> --region=ap-southeast-2 --deployment-group-name=<my-deployment-group> --update-outdated-instances-only
runcmd:
- bash /root/configure.sh
In this implementation the node is responsible for triggering the deployment itself. This is working perfectly so far for me but can result in deployment fails if the ASG is creating several instances at the same time (in that case the failed instances will be terminated quickly because not healthy).
Of course, you need to add the sufficient permissions to the role associated to your nodes to trigger the deployment.
This is still a workaround and if someone knows solution behaving the same way as cfn-init, I am interested.