I created the aws_db_instance to provision the RDS MySQL database using Terraform configuration. Now my next question is to execute the SQL Script (CREATE TABLE and INSERT statements) on the RDS. I did the following but there is no effect. terraform plan cannot even see my changes on executing the sql. What did I miss here? Thanks.
resource "aws_db_instance" "mydb" {
# ...
provisioner "remote-exec" {
inline = [
"chmod +x script.sql",
"script.sql args",
]
}
}
Check out this post: How to apply SQL Scripts on RDS with Terraform
If you're just trying to setup user's and permissions (you shouldn't use the root pw you set when you generate the RDS) there is a terraform provider for that:
https://www.terraform.io/docs/providers/mysql/index.html
But you're looking for DB schema and seeding. That provider cannot do that.
If you're open to doing it another way, you may want to check out using ssm automation documents and/or lambda. I'd use lambda. Pick a language that you're comfortable with. Set the role of the lambda to have permissions to read the password it needs to do the work. You can save the password in ssm parameter store. Then script your DB work.
Then do a local exec in terraform that simply calls the lambda and pass it the ID of the RDS and the path to the secret in ssm parameter store. That will ensure that the DB operations are done from compute inside the VPC without having to setup an EC2 bastion just for that purpose.
Here's how javascript can get this done, for example:
https://www.w3schools.com/nodejs/nodejs_mysql_create_table.asp
Related
I am having issues deploying my docker images to aws ecr as part of a terraform deployment and I am trying to think through the best long term strategy.
At the moment I have a terraform remote backend in s3 and dynamodb on let's call it my master account. I then have dev/test etc environments in separate accounts. The terraform deployment is currently run off my local machine (mac) and uses the aws 'master' account and its credentials which in turn assumes a role in the target deployment account to create the resources as per:
provider "aws" { // tell terraform which SDK it needs to load
alias = "target"
region = var.region
assume_role {
role_arn = "arn:aws:iam::${var.deployment_account}:role/${var.provider_env_deployment_role_name}"
}
}
I am creating a number of ecs services with Fargate deployments. The container images are built in separate repos by GitHub Actions and saved as GitHub packages. These package names and versions are being deployed after the creation of the ecr and service (maybe that's not ideal thinking about it) and this is where the problems arise.
The process is to pull the image from GitHub Packages, retag it and upload to the ecr using multiple executions of a null_resource local-exec. Works fine stand alone but has problems as part of the terraform process. I think the reason is that the other resources use the above provider to get permissions but as null_resource does not accept a provider it cannot get the permissions this way. So I have been passing the aws creds values into the shell. Not convinced this is really secure but that's currently moot as it ain't working either. I get this error:
Error saving credentials: error storing credentials - err: exit status 1, out: `error storing credentials - err: exit status 1, out: `The specified item already exists in the keychain.``
Part of me thinks this is the wrong approach and that as I migrate to deploying via a Github action I can separate the infrastructure deployment via terraform from what is really the application deployment and just use GitHub secrets to reset the credentials values then run the script.
Alternatively, maybe the keychain thing just goes away and my process will work fine? Secure ??
That's fine for this scenario but it isn't really a generic approach for all my use cases.
I am shortly going to start deploying multiple aws lambda functions with docker containers. Haven't done it before but it looks like the process is going to be: create ecr, deploy container, deploy lambda function. This really implies that the container deployment should integral to the terraform deployment which loops back to my issue with the local-exec??
I found Actions to deploy to ECR which would imply splitting the deployments into multiple files but that seems inelegant and potentially brittle.
Maybe there is a simple solution, but given where I am trying to go with this, what is my best approach?
I know this isn't a complete answer, but you should be pulling your AWS creds from environment variables. I don't really understand if you need credentials for different accounts, but if you do then swap them during the progress of your action. See https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html . Terraform should pick these up and automatically use them for AWS access.
Instead of those hard coded access key/secret access keys I'd suggest making use of Github/AWS's ability to assume role through temporary credentials with OIDC https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-amazon-web-services
You'd likely only define one initial role that you'd authenticate into and from there assume into the other accounts you're deploying into.
These the assume role credentials are only good for an hour and do not have the operation overhead of having to rotate them.
As suggested by Kevin Buchs answer...
My primary issue was related to deploying from a mac and the use of the keychain. As this was not on the critical path I went round it and set up a GitHub Action.
The Action loaded environmental variables from GitHub secrets for my 'master' aws account credentials.
AWS_ACCESS_KEY_ID: ${{ secrets.NK_AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.NK_AWS_SECRET_ACCESS_KEY }}
I also loaded the target accounts credentials into environmental variables in the same way BUT with the prefix TF_VAR.
TF_VAR_DEVELOP_AWS_ACCESS_KEY_ID: ${{ secrets.DEVELOP_AWS_ACCESS_KEY_ID }}
TF_VAR_DEVELOP_AWS_SECRET_ACCESS_KEY: ${{ secrets.DEVELOP_AWS_SECRET_ACCESS_KEY }}
I then declare terraform variables which will be automatically populated from the environment variables.
variable "DEVELOP_AWS_ACCESS_KEY_ID" {
description = "access key for the dev account"
type = string
}
variable "DEVELOP_AWS_SECRET_ACCESS_KEY" {
description = "secret access key for the dev account"
type = string
}
And when I run a shell script with a local exec:
resource "null_resource" "image-upload-to-importcsv-ecr" {
provisioner "local-exec" {
command = "./ecr-push.sh ${var.DEVELOP_AWS_ACCESS_KEY_ID} ${var.DEVELOP_AWS_SECRET_ACCESS_KEY} "
}
}
Within the script I can then use these arguments to set the credentials eg
AWS_ACCESS=$1
AWS_SECRET=$1
.....
export AWS_ACCESS_KEY_ID=${AWS_ACCESS}
export AWS_SECRET_ACCESS_KEY=${AWS_SECRET}
and the script now has credentials to do whatever.
I am looking for a way to provision an instance with a configuration file that contains the endpoints to connect to a database cluster in an automatic way, using terraform. I am using a aws_rds_cluster resource, from which I can get the endpoint using the expression aws_rds_cluster.my-cluster.endpoint. Then, I would like to provision machines instantiated with an aws_instance resource so that the value of that expression is stored in the file /DBConfig.sh.
The content of the DBConfig.sh file would look like this :
#!/bin/bash
ENDPOINT=<$aws_rds_cluster.my-cluster.endpoint$>
READER_ENDPOINT=<$aws_rds_cluster.my-cluster.reader_endpoint$>
Truth be told, once I successfully reach that point, I'd like to be able to do the same thing for machines created by a aws_launch_configuration resource.
Is this something that can be done with terraform? If not, what other tools can I use to achieve this kind of automation? Thanks for your help!
There are few ways which can achieve that. I think all of the would involve user_data.
For example, you could have aws_instance with the user_data as follows:
resource "aws_instance" "web" {
# other atrributes
user_data = <<-EOL
#!/bin/bash
cat >./DBConfig.sh <<-EOL2
#!/bin/bash
ENDPOINT=${aws_rds_cluster.my-cluster.endpoint}
READER_ENDPOINT=${aws_rds_cluster.my-cluster.reader_endpoint}
EOL2
chmod +x ./DBConfig.sh
EOL
}
The above will launch an istance which will have DBConfig.sh with resolved values of the endpoints in its root (/) directory.
I have a IAM user that is created by Terraform. Keys are stored in Hashicrop Vault and apps read them from there.
I have developed ansible code/bash scripts to rotate the keys periodically successfully.
But the issue is terraform doesn't like when the keys are rotated. Whenever we try to run terraform it tries to re-create the key
Is there any way to manage the key rotation via terraform? or can we ignore this in terraform. Any help with examples would be really helpful.
Key rotation in terraform is possible by using terraform apply -replace=<resource address>, which replaces the resource immediately or terraform taint <resource address>, which replaces the resource on the next apply for version below v0.15.2. See https://www.terraform.io/docs/cli/commands/taint.html for more information.
When using these commands it makes sense to set the lifecycle of the resources to replace to create_before_destroy to avoid downtime. So in the case of AWS access keys, that would be
resource "aws_iam_access_key" "my_user" {
user = "my_user_name"
lifecycle {
create_before_destroy = true
}
}
Given this configuration one can simply run terraform apply -replace=aws_iam_access_key.my_user to rotate the keys. One only has to make sure that downstream applications that use the keys take notice of the changes and are restarted if necessary to make use of the new keys.
We have managed to solve the problem by removing key generation initially via terraform when the user is created.
We are using some ansible and bash scripts to now generate and rotate keys and then vault api to update secrets in Vault.
I have the AWS CLI installed on my Windows computer, and running this command "works" exactly like I want it to.
aws ec2 describe-images
I get the following output, which is exactly what I want to see, because although I have access to AWS through my corporation (e.g. to check code into CodeCommit), I can see in the AWS web console for EC2 that I don't have permission to list running instances:
An error occurred (UnauthorizedOperation) when calling the DescribeImages operation: You are not authorized to perform this operation.
I've put terraform.exe onto my computer as well, and I've created a file "example.tf" that contains the following:
provider "aws" {
region = "us-east-1"
}
I'd like to issue some sort of Terraform command that would yell at me, explaining that my AWS account is not allowed to list Amazon instances.
Most Hello World examples involve using terraform plan against a resource to do an "almost-write" against AWS.
Personally, however, I always feel more comfortable knowing that things are behaving as expected with something a bit more "truly read-only." That way, I really know the round-trip to AWS worked but I didn't modify any of my corporation's state.
There's a bunch of stuff on the internet about "data sources" and their "aws_ami" or "aws_instances" flavors, but I can't find anything that tells me how to actually use it with a Terraform command for a simple print()-type interaction (the way it's obvious that, say, "resources" go with the "terraform plan" and "terraform apply" commands).
Is there something I can do with Terraform commands to "hello world" an attempt at listing all my organization's EC2 servers and, accordingly, watching AWS tell me to buzz off because I'm not authorized?
You can use the data source for AWS instances. You create a data source similar to the below:
data "aws_instances" "test" {
instance_tags = {
Role = "HardWorker"
}
filter {
name = "instance.group-id"
values = ["sg-12345678"]
}
instance_state_names = ["running", "stopped"]
}
This will attempt to perform a read action listing your EC2 instances designated by the filter you put in the config. This will also utilize the IAM associated with the Terraform user you are performing the terraform plan with. This will result in the error you described regarding lack of authorization, which is your stated goal. You should modify the filter to target your organization's EC2 instances.
If I have a bash script sitting in an EC2 instance, is there a way that lambda could trigger it?
The trigger for lambda would be coming from RDS. So a table in mysql gets updated and a specific column in that table gets updated to "Ready", Lambda would have to pull the ID of that row with a "Ready" status and send that ID to the bash script.
Let's assume some things. First, you know how to set up a "trigger" using sns (see here) and how to hang a lambda script off of said trigger. Secondly, you know a little about python (Lambda's syntax offerings are Node, Java, and Python) because this example will be in Python. Additionally, I will not cover how to query a database with mysql. You did not mention whether your RDS instance was MySQL, Postgress, or otherwise. Lastly, you need to understand how to allow permission across AWS resources with IAM roles and policies.
The following script will at least outline the method of firing a script to your instance (you'll have to figure out how to query for relevant information or pass that information into the SNS topic), and then run the shell command on an instance you specify.
import boto3
def lambda_handler(event, context):
#query RDS to get ID or get from SNS topic
id = *queryresult*
command = 'sh /path/to/scriptoninstance' + id
ssm = boto3.client('ssm')
ssmresponse = ssm.send_command(InstanceIds=['i-instanceid'], DocumentName='AWS-RunShellScript', Parameters= { 'commands': [command] } )
I would probably have two flags for the RDS row. One that says 'ready' and one that says 'identified'. So SNS topic triggers lambda script, lambda script looks for rows with 'ready' = true and 'identified' = false, change 'identified' to true (to make sure other lambda scripts that could be running at the same time aren't going to pick it up), then fire script. If script doesn't run successfully, change 'identified' back to false to make sure your data stays valid.
Using Amazon EC2 Simple Systems Manager, you can configure an SSM document to run a script on an instance, and pass that script a parameter. The Lambda instance would need to run the SSM send-command, targeting the instance by its instance id.
Sample SSM document:
run_my_example.json:
{
"schemaVersion": "1.2",
"description": "Run shell script to launch.",
"parameters": {
"taskId":{
"type":"String",
"default":"",
"description":"(Required) the Id of the task to run",
"maxChars":16
}
},
"runtimeConfig": {
"aws:runShellScript": {
"properties": [
{
"id": "0.aws:runShellScript",
"runCommand": ["run_my_example.sh"]
}
]
}
}
}
The above SSM document accepts taskId as a parameter.
Save this document as a JSON file, and call create-document using the AWS CLI:
aws ssm create-document --content file:///tmp/run_my_example.json --name "run_my_example"
You can review the description of the SSM document by calling describe-document:
aws ssm describe-document --name "run_my_example"
You can specify the taskId parameter and run the command by using the document name with the send-command
aws ssm send-command --instance-ids i-12345678 --document-name "run_my_example" --parameters --taskid=123456
NOTES
Instances must be running the latest version of the SSM agent.
You will need to have some logic in the Lambda script to identify the instance ids of the server EG look up the instance id of a specifically tagged instance.
I think you could use the new EC2 Run Command feature to accomplish this.
There are few things to consider one of them is:
Security. As of today lambda can't run in VPC. Which means your EC2 has to have a wide open inbound security group.
I would suggest take a look at the messaging queue (say SQS ). This would solve a lot of headache.
That's how it might work:
Lambda. Get message; send to SQS
EC2. Cron job that gets trigger N number of minutes. pull message from sqs; process message.