How do I send cloud-init script to a gcp instance using terraform?
Documentation is very sparse around this topic.
You need the following:
A cloud-init file (say 'conf.yaml')
#cloud-config
# Create an empty file on the system
write_files:
- path: /root/CLOUD_INIT_WAS_HERE
cloudinit_config data source
gzip and base64_encode must be set to false (they are true by default).
data "cloudinit_config" "conf" {
gzip = false
base64_encode = false
part {
content_type = "text/cloud-config"
content = file("conf.yaml")
filename = "conf.yaml"
}
}
A metadata section under the google_compute_instance resource
metadata = {
user-data = "${data.cloudinit_config.conf.rendered}"
}
Related
I trying to figure out using AWS CLI how to upload only certain files to S3 bucket based on the mime type.
Currently I am using terraform script to do that.
locals {
mime_types = jsondecode(file("${path.module}/mime.json"))
}
resource "aws_s3_object" "frontend_bucket_objects" {
for_each = fileset("${var.build_folder}", "**")
bucket = "frontend-bucket"
key = each.value
source = "${var.build_folder}\\${each.value}"
content_type = lookup(local.mime_types, regex("\\.[^.]+$", "${each.value}"), null)
etag = filemd5("${var.build_folder}\\${each.value}")
}
mime.json
{
".aac": "audio/aac",
".abw": "application/x-abiword",
".arc": "application/x-freearc",
".avif": "image/avif",
".avi": "video/x-msvideo",
".azw": "application/vnd.amazon.ebook",
".bin": "application/octet-stream",
".bmp": "image/bmp",
".css": "text/css",
".csv": "text/csv",
".doc": "application/msword",
..
.. etc etc
}
I want to upload using aws-cli but not able to figure out how to include files based on mime.
This is what I have right now which uploads entire source folder.
`aws s3 cp build/ s3://frontend-bucket --recursive`
I wanna create separate log streams for each fille with pattern sync-.log (for example filename sync-site1.log, stream name - server-sync-site1). There are a lot of log files with sync-.log pattern, so do it manually for each is a bad option for me. How can I do it? I'm using an older agent.
Here is my current config:
[server-sync.log]
datetime_format = %Y-%m-%dT%H:%M:%S,%f
file = /home/ec2-user/log/sync/sync-*.log
buffer_duration = 5000
log_stream_name = server-sync
initial_position = end_of_file
multi_line_start_pattern = {datetime_format}
log_group_name = server
There is a well defined API Call for this purpose CreateLogStream, corresponding api call in boto3 is create_stream
response = client.create_log_stream(
logGroupName='string',
logStreamName='string'
)
There is a terraform resource as well named aws_cloudwatch_log_stream
resource "aws_cloudwatch_log_group" "yada" {
name = "Yada"
}
resource "aws_cloudwatch_log_stream" "foo" {
name = "SampleLogStream1234"
log_group_name = aws_cloudwatch_log_group.yada.name
}
I am trying to create a Windows Ec2 instance from AMI and executing a powershell command on that as :
data "aws_ami" "ec2-worker-initial-encrypted-ami" {
filter {
name = "tag:Name"
values = ["ec2-worker-initial-encrypted-ami"]
}
}
resource "aws_instance" "my-test-instance" {
ami = "${data.aws_ami.ec2-worker-initial-encrypted-ami.id}"
instance_type = "t2.micro"
tags {
Name = "my-test-instance"
}
provisioner "local-exec" {
command = "C:\\ProgramData\\Amazon\\EC2-Windows\\Launch\\Scripts\\InitializeInstance.ps1 -Schedule",
interpreter = ["PowerShell"]
}
}
and I am facing following error :
aws_instance.my-test-instance: Error running command 'C:\ProgramData\Amazon\EC2-Windows\Launch\Scripts\InitializeInstance.ps1
-Schedule': exit status 1. Output: The term 'C:\ProgramData\Amazon\EC2-Windows\Launch\Scripts\InitializeInstance.ps1'
is not recognized as the name of a cmdlet, function, script file, or
operable program. Check the spelling of the name, or if a path was
included, verify that the path is correct and try again. At line:1
char:72
C:\ProgramData\Amazon\EC2-Windows\Launch\Scripts\InitializeInstance.ps1
<<<< -Schedule
CategoryInfo : ObjectNotFound: (C:\ProgramData...izeInstance.ps1:String) [],
CommandNotFoundException
FullyQualifiedErrorId : CommandNotFoundException
You are using a local-exec provisioner which runs the request powershell code on the workstation running Terraform:
The local-exec provisioner invokes a local executable after a resource
is created. This invokes a process on the machine running Terraform,
not on the resource.
It sounds like you want to execute the powershell script on the resulting instance in which case you'll need to use a remote-exec provisioner which will run your powershell on the target resource:
The remote-exec provisioner invokes a script on a remote resource
after it is created. This can be used to run a configuration
management tool, bootstrap into a cluster, etc.
You will also need to include connection details, for example:
provisioner "remote-exec" {
command = "C:\\ProgramData\\Amazon\\EC2-Windows\\Launch\\Scripts\\InitializeInstance.ps1 -Schedule",
interpreter = ["PowerShell"]
connection {
type = "winrm"
user = "Administrator"
password = "${var.admin_password}"
}
}
Which means this instance must also be ready to accept WinRM connections.
There are other options for completing this task though. Such as using userdata, which Terraform also supports. This might look like the following example:
Example of using a userdata file in Terraform
File named userdata.txt:
<powershell>
C:\\ProgramData\\Amazon\\EC2-Windows\\Launch\\Scripts\\InitializeInstance.ps1 -Schedule
</powershell>
Launch instance using the userdata file:
resource "aws_instance" "my-test-instance" {
ami = "${data.aws_ami.ec2-worker-initial-encrypted-ami.id}"
instance_type = "t2.micro"
tags {
Name = "my-test-instance"
}
user_data = "${file(userdata.txt)}"
}
The file interpolation will read the contents of the userdata file as string to pass to userdata for the instance launch. Once the instance launches it should run the script as you expect.
What Brian is claiming is correct, you will get "invalid or unknown key: interpreter" error.
To correctly run powershell you will need to run it as following, based on Brandon's answer:
provisioner "remote-exec" {
connection {
type = "winrm"
user = "Administrator"
password = "${var.admin_password}"
}
inline = [
"powershell -ExecutionPolicy Unrestricted -File C:\\ProgramData\\Amazon\\EC2-Windows\\Launch\\Scripts\\InitializeInstance.ps1 -Schedule"
]
}
Edit
To copy the files over to the machine use the below:
provisioner "file" {
source = "${path.module}/some_path"
destination = "C:/some_path"
connection {
host = "${azurerm_network_interface.vm_nic.private_ip_address}"
timeout = "3m"
type = "winrm"
https = true
port = 5986
use_ntlm = true
insecure = true
#cacert = "${azurerm_key_vault_certificate.vm_cert.certificate_data}"
user = var.admin_username
password = var.admin_password
}
}
Update:
Currently provisioners are not recommended by hashicorp, full instructions and explanation (it is long) can be found at: terraform.io/docs/provisioners/index.html
FTR: Brandon's answer is correct, except the example code provided for the remote-exec includes keys that are unsupported by the provisioner.
Neither command nor interpreter are supported keys.
https://www.terraform.io/docs/provisioners/remote-exec.html
According to documentation, using terraform, I'm able to create a droplet on digital ocean:
resource "digitalocean_volume" "foobar" {
region = "nyc1"
name = "baz"
size = 100
description = "an example volume"
}
So, I'm also able to add a volume to it:
resource "digitalocean_droplet" "foobar" {
name = "baz"
size = "1gb"
image = "coreos-stable"
region = "nyc1"
volume_ids = ["${digitalocean_volume.foobar.id}"]
}
I'd like to know how to mount this on a desired location.
I need to mount it automatically. I mean, when droplet is up I need to the volume is mounted. I was thinking about using chef...
Any ideas?
To mount the volume automatically, you can use user_data via cloud init to run a script as follow:
This is how your digitalocean_droplet resources should reflect:
resource "digitalocean_droplet" "foobar" {
name = "baz"
size = "1gb"
image = "coreos-stable"
region = "nyc1"
volume_ids = ["${digitalocean_volume.foobar.id}"]
# user data
user_data = "${data.template_cloudinit_config.cloudinit-example.rendered}"
}
Then your cloud.init file that contains the cloudinit_config should be as bellow. It will reference the shell script in ${TERRAFORM_HOME}/scripts/disk.sh that would mount your volume automatically:
provider "cloudinit" {}
data "template_file" "shell-script" {
template = "${file("scripts/disk.sh")}"
}
data "template_cloudinit_config" "cloudinit-example" {
gzip = false
base64_encode = false
part {
content_type = "text/x-shellscript"
content = "${data.template_file.shell-script.rendered}"
}
}
The shell script to mount the volume automatically on startup is in ${TERRAFORM_HOME}/scripts/disk.sh
It will first check if a file system exist. If true it wouldn't format the disk if not it will
#!/bin/bash
DEVICE_FS=`blkid -o value -s TYPE ${DEVICE}`
if [ "`echo -n $DEVICE_FS`" == "" ] ; then
mkfs.ext4 ${DEVICE}
fi
mkdir -p /data
echo '${DEVICE} /data ext4 defaults 0 0' >> /etc/fstab
mount /data
I hope this helps
Mounting the volume needs to be done from the guest OS itself using mount, fstab, etc.
The digital ocean docs cover this here.
Using Chef you could use resource_mount to mount it in an automated fashion.
The device name will be /dev/disk/by-id/scsi-0DO_Volume_YOUR_VOLUME_NAME. So, using the example from the Terraform docs, it would be /dev/disk/by-id/scsi-0DO_Volume_baz.
my .s3cfg with GPG encryption passphrase and other security settings. Would you recommend other security hardening?
[default]
access_key = $USERNAME
access_token =
add_encoding_exts =
add_headers =
bucket_location = eu-central-1
ca_certs_file =
cache_file =
check_ssl_certificate = True
check_ssl_hostname = True
cloudfront_host = cloudfront.amazonaws.com
default_mime_type = binary/octet-stream
delay_updates = False
delete_after = False
delete_after_fetch = False
delete_removed = False
dry_run = False
enable_multipart = True
encoding = UTF-8
encrypt = False
expiry_date =
expiry_days =
expiry_prefix =
follow_symlinks = False
force = False
get_continue = False
gpg_command = /usr/local/bin/gpg
gpg_decrypt = %(gpg_command)s -d --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
gpg_encrypt = %(gpg_command)s -c --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
gpg_passphrase = $PASSPHRASE
guess_mime_type = True
host_base = s3.amazonaws.com
host_bucket = %(bucket)s.s3.amazonaws.com
human_readable_sizes = False
invalidate_default_index_on_cf = False
invalidate_default_index_root_on_cf = True
invalidate_on_cf = False
kms_key =
limitrate = 0
list_md5 = False
log_target_prefix =
long_listing = False
max_delete = -1
mime_type =
multipart_chunk_size_mb = 15
multipart_max_chunks = 10000
preserve_attrs = True
progress_meter = True
proxy_host =
proxy_port = 0
put_continue = False
recursive = False
recv_chunk = 65536
reduced_redundancy = False
requester_pays = False
restore_days = 1
secret_key = $PASSWORD
send_chunk = 65536
server_side_encryption = False
signature_v2 = False
simpledb_host = sdb.amazonaws.com
skip_existing = False
socket_timeout = 300
stats = False
stop_on_error = False
storage_class =
urlencoding_mode = normal
use_https = True
use_mime_magic = True
verbosity = WARNING
website_endpoint = http://%(bucket)s.s3-website-%(location)s.amazonaws.com/
website_error =
website_index = index.html
I use this command to upload/sync my local folder to Amazon S3.
s3cmd -e -v put --recursive --dry-run /Users/$USERNAME/Downloads/ s3://dgtrtrtgth777
INFO: Compiling list of local files...
INFO: Running stat() and reading/calculating MD5 values on 15957 files, this may take some time...
INFO: [1000/15957]
INFO: [2000/15957]
INFO: [3000/15957]
INFO: [4000/15957]
INFO: [5000/15957]
INFO: [6000/15957]
INFO: [7000/15957]
INFO: [8000/15957]
INFO: [9000/15957]
INFO: [10000/15957]
INFO: [11000/15957]
INFO: [12000/15957]
INFO: [13000/15957]
INFO: [14000/15957]
INFO: [15000/15957]
I tested the encryption with Transmit GUI S3 Client and didn't get plain text files.
But I see the original filename. I wish to change the filename to a random value, but have local the original filename (mapping?). How can I do this?
What are downsides doing so if I need to restore the files? I use Amazon S3 only as a backup, in addition to my TimeMachine backup.
If you use "random" names, then it isn't sync.
If your only record on the filenames/mapping is local, it will be impossible to restore your backup in case of a local failure.
If you don't need all versions of your files I'd suggest putting everything in a (possibly encrypted) compressed tarball before uploading it.
Otherwise, you will have to write a small script that lists all files and individually does an s3cmd put specifying a random destination, where the mapping is appended to a log file, which should be the first thing you s3cmd put to your server. I don't recommend this for something as crucial as storing your backups.
A skeleton showing how this could work:
# Save all files in backupX.sh where X is the version number
find /Users/$USERNAME/Downloads/ | awk '{print "s3cmd -e -v put "$0" s3://dgtrshitcrapola/"rand()*1000000}' > backupX.sh
# Upload the mapping file
s3cmd -e -v put backupX.sh s3://dgtrshitcrapola/
# Upload the actual files
sh backupX.sh
# Add cleanup code here
However, you will need to handle filename collisions, failed uploads, versioning clashes, ... why not use an existing tool that backs up to S3?