Hi I have a question while using AWS Lambda <-> AWS RDS.
Now I have a connection with one lambda function and one RDS db schema.
In the Lambda function, I have config.py with this json codes.
dbinfo = {
'db' = ~
'user' = ~
'passwd'= ~
'schema' = ~
}
Lambda function works for connecting in to the RDS schema , and calling store procedure.
Right now, there is only one schema that I have to connect with Lambda function.
But soon that schema will be changed to a sharding structure.
(ex. game1 schema -> game1 , game2 schemas)
That stored procedure will be stored at both schemas (game1, game2).
How to connect two schemas in one Lambda function?
Structure your config.py file something like:
dbinfo = {
'shard1' = {
'db' = ~
'user' = ~
'passwd'= ~
'schema' = ~
},
'shard2' = {
'db' = ~
'user' = ~
'passwd'= ~
'schema' = ~
},
}
and inside of your Lambda, create multiple RDS clients for each shard.
And FWIW, you might want to consider storing your password outside of code in AWS ParameterStore.
Related
I am using terraform aws_rds_cluster and aws_rds_cluster_instances modules to provision the AWS RDS cluster (mysql). This creates the cluster with one writer and two read replicas. In the output.tf, I need to get the endpoint of the RDS writer instance.
output "rds_writer_instance_endpoint {
value = aws_rds_cluster.instances.*.endpoint
}
I got all the endpoints for the three instances "aws_rds_cluster.instances.*.endpoint". How to retrieve only the writer endpoint?
your instance objects will be returning with an attribute of writer. https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/rds_cluster_instance#attributes-reference
writer – Boolean indicating if this instance is writable. False indicates this instance is a read replica.
You can use this attribute to filter your list and return only objects that have this value set to true. Assuming that only one instance can be the writter I then use the one function of terraform to set the output as the value of the list.
output "rds_writer_instance_endpoint {
value = one([for instance in aws_rds_cluster.instances]: var.endpoint if var.writer])
}
As an example of this in action, I have created a list of variables that represent like your instances resource and apply the above pattern to it.
variable "instances" {
type = list(object({
endpoint = string
writer = bool
}))
default = [
{
endpoint = "http:localhost:8080"
writer = false
},
{
endpoint = "http:localhost:8081"
writer = true
},
{
endpoint = "http:localhost:8082"
writer = false
}
]
}
output "all_out" {
value = var.instances.*.endpoint
}
output "writer" {
value = one([for instance in var.instances: instance.endpoint if instance.writer])
}
OUTPUT
Outputs:
all_out = tolist([
"http:localhost:8080",
"http:localhost:8081",
"http:localhost:8082",
])
writer = "http:localhost:8081"
I am trying to write a query to return results of any referenced security group not owned by the current account.
This means I am trying to show security groups that are being used as part of a peering connection from another VPC.
There are a couple of restrictions.
Show the entire security group details (security group id, description)
Only show security groups where IpPermissions.UserIdGroupPairs has a Value and where that value is not equal to the owner of the security group
I am trying to write this using a single AWS CLI cmd vs a bash script or python script.
Any thoughts?
Heres what I have so far.
aws ec2 describe-security-groups --query "SecurityGroups[?IpPermissions.UserIdGroupPairs[*].UserId != '`aws sts get-caller-identity --query 'Account' --output text`']"
Following is Python 3.8 based AWS Lambda, but you can change a bit to use as python script file to execute on any supported host machine.
import boto3
import ast
config_service = boto3.client('config')
# Refactor to extract out duplicate code as a seperate def
def lambda_handler(event, context):
results = get_resource_details()
for resource in results:
if "configuration" in resource:
config=ast.literal_eval(resource)["configuration"]
if "ipPermissionsEgress" in config:
ipPermissionsEgress=config["ipPermissionsEgress"]
for data in ipPermissionsEgress:
for userIdGroupPair in data["userIdGroupPairs"]:
if userIdGroupPair["userId"] != "123456789111":
print(userIdGroupPair["groupId"])
elif "ipPermissions" in config:
ipPermissions=config["ipPermissions"]
for data in ipPermissions:
for userIdGroupPair in data["userIdGroupPairs"]:
if userIdGroupPair["userId"] != "123456789111":
print(userIdGroupPair["groupId"])
def get_resource_details():
query = "SELECT configuration.ipPermissions.userIdGroupPairs.groupId,configuration.ipPermissionsEgress.userIdGroupPairs.groupId,configuration.ipPermissionsEgress.userIdGroupPairs.userId,configuration.ipPermissions.userIdGroupPairs.userId WHERE resourceType = 'AWS::EC2::SecurityGroup' AND configuration <> 'null'"
results = config_service.select_resource_config(
Expression=query,
Limit=100
) # you might need to refacor to add support huge list of records using NextToken
return results["Results"]
I'm forwarding DNS requests sent to a list of internal domains (on premise) by using AWS Route53 resolver. By terraform, I want to share the rules I created to other accounts of the company, so I have the following:
# I create as much share endpoint as domain I have, so If I have 30 domains, I'll make 30 endpoint RAM:
resource "aws_ram_resource_share" "endpoint_share" {
count = length(var.forward_domain)
name = "route53-${var.forward_domain[count.index]}-share"
allow_external_principals = false
}
# Here I share every single endpoint with all the AWS ACcount we have
resource "aws_ram_principal_association" "endpoint_ram_principal" {
count = length(var.resource_share_accounts)
principal = var.resource_share_accounts[count.index]
resource_share_arn = {
for item in aws_ram_resource_share.endpoint_share[*]:
item.arn
}
}
The last block, calls the arn output of the first one which is a list.
Now, this last block doesn't work, I don't know how to use multiple counts, when I run this, I get the following error:
Error: Invalid 'for' expression
line 37: Key expression is required when building an object.
Any idea how to make this work?
Terraform version: 0.12.23
Use square brackets in resource_share_arn, like this:
resource_share_arn = [
for item in aws_ram_resource_share.endpoint_share[*]:
item.arn
]
Amazon Athena reads data from input Amazon S3 buckets using the IAM credentials of the user who submitted the query; query results are stored in a separate S3 bucket.
Here is the script in Hashicorp site https://www.terraform.io/docs/providers/aws/r/athena_database.html
resource "aws_s3_bucket" "hoge" {
bucket = "hoge"
}
resource "aws_athena_database" "hoge" {
name = "database_name"
bucket = "${aws_s3_bucket.hoge.bucket}"
}
Where it says
bucket - (Required) Name of s3 bucket to save the results of the query execution.
How can I specify the input S3 bucket in the terraform script?
You would use the storage_descriptor argument in the aws_glue_catalog_table resource:
https://www.terraform.io/docs/providers/aws/r/glue_catalog_table.html#parquet-table-for-athena
Here is an example of creating a table using CSV file(s):
resource "aws_glue_catalog_table" "aws_glue_catalog_table" {
name = "your_table_name"
database_name = "${aws_athena_database.your_athena_database.name}"
table_type = "EXTERNAL_TABLE"
parameters = {
EXTERNAL = "TRUE"
}
storage_descriptor {
location = "s3://<your-s3-bucket>/your/file/location/"
input_format = "org.apache.hadoop.mapred.TextInputFormat"
output_format = "org.apache.hadoop.mapred.TextInputFormat"
ser_de_info {
name = "my-serde"
serialization_library = "org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe"
parameters = {
"field.delim" = ","
"skip.header.line.count" = "1"
}
}
columns {
name = "column1"
type = "string"
}
columns {
name = "column2"
type = "string"
}
}
}
The input S3 bucket is specified in each table you create in the database, as such, there's no global definition for it.
As of today, the AWS API doesn't have much provision for Athena management, as such, neither does the aws CLI command, and nor does Terraform. There's no 'proper' way to create a table via these means.
In theory, you could create a named query to create your table, and then execute that query (for which there is API functionality, but not yet Terraform). It seems a bit messy to me, but it would probably work if/when TF gets the StartQuery functionality. The asynchronous nature of Athena makes it tricky to know when that table has actually been created though, and so I can imagine TF won't fully support table creation directly.
TF code that covers the currently available functionality is here: https://github.com/terraform-providers/terraform-provider-aws/tree/master/aws
API doco for Athena functions is here: https://docs.aws.amazon.com/athena/latest/APIReference/API_Operations.html
Terraform allows you to define Postgres master user and password with the options username and password. But there is no option to set up an application postgres user, how would you do that?
The AWS RDS resource is only used for creating/updating/deleting the RDS resource itself using the AWS APIs.
To create users or databases on the RDS instance itself you'd either want to use another tool (such as psql - the official command line tool or a configuration management tool such as Ansible) or use Terraform's Postgresql provider.
Assuming you've already created your RDS instance you would then connect to the instance as the master user and then create the application user with something like this:
provider "postgresql" {
host = "postgres_server_ip1"
username = "postgres_user"
password = "postgres_password"
}
resource "postgresql_role" "application_role" {
name = "application"
login = true
password = "application-password"
encrypted = true
}
addition to #ydaetskcoR answer, here is the full example for RDS PostgreSQL;
provider "postgresql" {
scheme = "awspostgres"
host = "db.domain.name"
port = "5432"
username = "db_username"
password = "db_password"
superuser = false
}
resource "postgresql_role" "new_db_role" {
name = "new_db_role"
login = true
password = "db_password"
encrypted_password = true
}
resource "postgresql_database" "new_db" {
name = "new_db"
owner = postgresql_role.new_db_role.name
template = "template0"
lc_collate = "C"
connection_limit = -1
allow_connections = true
}
The above two answers requires the host that runs the terraform has direct access to the RDS database, and usually you do not. I propose to code what you need to do in a lambda function (optionally with secrets manager for retrieving the master password):
resource "aws_lambda_function" "terraform_lambda_func" {
filename = "${path.module}/lambda_function/lambda_function.zip"
...
}
and then use the following data source (example) to call the lambda function.
data "aws_lambda_invocation" "create_app_user" {
function_name = aws_lambda_function.terraform_lambda_func.function_name
input = <<-JSON
{
"step": "create_app_user"
}
JSON
depends_on = [aws_lambda_function.terraform_lambda_func]
provider = aws.primary
}
This solution id generic. It can do what a lambda function can do with AWS API can do, which is basically limitless.