I have developed the application using Java and I also used the Amazon PostgreSQL database for data management. I hosted the application in Elastic beanstalk. Now, Someone suggested me to use the Amazon CloudFormation. So I created the Infrastructure code in JSON Format that also include Amazon RDS but I have some doubts.
When I use CloudFormation then that will automatically creates the new DB instance for my application but I specified another DB instance name in Java code then how it will communicate?
Please help me to clarify the doubts.
Thanks in advance...
You can configure DB URL in outputs section of CFN so that you get the required URL
CFN outputs
To get endpoint url for your AWS::RDS::DBInstance is returned using Return values:
Endpoint.Address The connection endpoint for the database. For example: mystack-mydb-1apw1j4phylrk.cg034hpkmmjt.us-east-2.rds.amazonaws.com
Endpoint.Port The port number on which the database accepts connections. For example: 3306
To get the Endpoint.Address out of your stack, you have to add Outputs section to your template. En example would be:
"Outputs": {
"DBEndpoint": {
"Description": "Endpoint for my RDS Instance",
"Value": {
"Fn::GetAtt" : [ "MyDB", "Endpoint.Address" ]}
}
}
}
Then using AWS SDK for Java you can query the Outputs of your CFN Stack to use in your Java application.
Related
How I can create an Athena data source in AWS CDK which is a JDBC connection to a MySQL database using the AthenaJdbcConnector?
I believe I can use aws-sam's CfnApplication to create the AthenaJdbcConnector Lambda, but how can I connect it to Athena?
I notice a lot of Glue support in CDK which would transfer to Athena (data catalog), and there are several CfnDataSource types in other modules such as QuickSight, but I'm not seeing anything under Athena in CDK.
See the image and references below.
References:
https://docs.aws.amazon.com/athena/latest/ug/athena-prebuilt-data-connectors-jdbc.html
https://github.com/awslabs/aws-athena-query-federation/tree/master/athena-jdbc
https://serverlessrepo.aws.amazon.com/applications/us-east-1/292517598671/AthenaJdbcConnector
I have been playing with the same issue. Here is what I did to create the Lambda for federated queries (Typescript):
const vpc = ec2.Vpc.fromLookup(this, 'my-project-vpc', {
vpcId: props.vpcId
});
const cluster = new rds.ServerlessCluster(this, 'AuroraCluster', {
engine: rds.DatabaseClusterEngine.AURORA_POSTGRESQL,
parameterGroup: rds.ParameterGroup.fromParameterGroupName(this, 'ParameterGroup', 'default.aurora-postgresql10'),
defaultDatabaseName: 'MyDB',
vpc,
vpcSubnets: {
onePerAz: true
},
scaling: {autoPause: cdk.Duration.seconds(0)} // Optional. If not set, then instance will pause after 5 minutes
});
let password = cluster.secret!.secretValueFromJson('password').toString()
let spillBucket = new Bucket(this, "AthenaFederatedSpill")
let lambdaApp = new CfnApplication(this, "MyDB", {
location: {
applicationId: "arn:aws:serverlessrepo:us-east-1:292517598671:applications/AthenaJdbcConnector",
semanticVersion: "2021.42.1"
},
parameters: {
DefaultConnectionString: `postgres://jdbc:postgresql://${cluster.clusterEndpoint.hostname}/MyDB?user=postgres&password=${password}`,
LambdaFunctionName: "crossref_federation",
SecretNamePrefix: `${cluster.secret?.secretName}`,
SecurityGroupIds: `${cluster.connections.securityGroups.map(value => value.securityGroupId).join(",")}`,
SpillBucket: spillBucket.bucketName,
SubnetIds: vpc.privateSubnets[0].subnetId
}
})
This creates the lambda with a default connection string like you would have it, if you used the AWS Console wizard in Athena to connect to a DataSource. Unfortunately it is NOT possible to add a Athena-catalog specific connection string via CDK. It should be set as an Environment Variable on the Lambda, and I found no way to do that. The Application template simply don't allow it, so this is a post-process by hand. I would sure like to hear from anybody if they have a solution for that!
Also notice that I add the user/password in the jdbc URL directly. I wanted to use SecretsManager, but because the Lambda is deployed in a VPC, it simply refuses to connect to the secretsmanager. I think this might be solvable by added a private VPN connection to SSM. Again - I would like to hear from anybody have tried that.
We're trying to enable Data API for our Aurora Serverless Cluster which has been set up to be in our private subnets in our VPC. We've tried enabling it via the console (i.e. Modify > Select Data API > Apply Immediately). There weren't any errors on after submitting, but when we get back to the "Modify" page, the "Data API" checkbox is still unselected.
We have also tried enabling Data API as described on the documentation https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/data-api.html , but the response still shows us that the HttpEndpointEnabled is still false.
My guess is that this could be related to Private Subnets or Security Groups. Any feedback is appreciated!
$ aws rds modify-db-cluster --db-cluster-identifier our-database --enable-http-endpoint
{
"DBCluster": {
"Capacity": 8,
"MasterUsername": "admin",
"HttpEndpointEnabled": false,
"ReadReplicaIdentifiers": [],
"VpcSecurityGroups": [
{
"Status": "active",
"VpcSecurityGroupId": "sg-0a66b09ade97573f0"
}
],
The database was created with CloudFormation if that matters
Data API is available only for serverless db. Make sure you have selected "serverless" option while creating database.
I had the same problem when I was trying to create aurora-postgresql variant that doesn't have data api option. Then I selected aurora-mysql with serverless option and aws gave me option to enable http endpoint.
I'm quite bad at using AWS but I'm trying to automate the set up of an ECS cluster with private DNS names in route53, using the new service discovery mechanism. I am able to click my way through the AWS UI to accomplish a DNS entry showing up in a private hosted zone but I cannot figure out the JSON parameters to add to the json for the command below to accomplish the same thing.
aws ecs create-service --cli-input-json file://aws/createService.json
and below is the approximate contents of the createService.json
referenced above
"cluster": "clustername",
"serviceName": "servicename",
"taskDefinition": "taskname",
"desiredCount": 1,
// here is where I'm guessing there should be some DNS config referencing some
// namespace or similar that I cannot figure out...
"networkConfiguration": {
"awsvpcConfiguration": {
"subnets": [
"subnet-11111111"
],
"securityGroups": [
"sg-111111111"
],
"assignPublicIp": "DISABLED"
}
}
I'd be grateful for any ideas since my googling skills apparently aren't good enough for this problem as it seems. Many thanks!
To automatically have an ECS service register instances into a servicediscovery service you can use the serviceRegistries attribute. Add the following to the ECS service definition json:
{
...
"serviceRegistries": [
{
"registryArn": "arn:aws:servicediscovery:region:aws_account_id:service/srv-utcrh6wavdkggqtk"
}
]
}
The attribute contains a list of autodiscovery services that should be updated by ECS when it creates or destroys a task as part of the service. Each registry is referenced using the ARN of the autodiscovery service.
To get the Arn use the AWS cli command aws servicediscovery list-services
Strangely the documentation of the ECS service definition does not contain information about this attribute. However this tutorial about service discovery does.
As it turns out there is no support in ecs create service for adding it to the service registry, i.e. the route53 private hosted zone. Instead I had to use aws servicediscovery create-service and then servicediscovery register-instance to finally get an entry in my private hosted zone.
This became a quite complicated solution so I'll instead give Terraform a shot at it since I found they recently added support for ECS service discovery and see where that takes me...
I have a bunch of EC2 instances that I've spun up using CloudFormation. I need to programmatically get the AWS instance_id for each of these hosts and would ideally like to do so without having to ssh into each of the hosts and gather that information. Is there an AWS API that will provide me with this functionality? Thanks!
There are several ways. The one I like best is putting an output section in your CloudFormation template with entries for each EC2 instance created by the template. Then you can fetch those results when you create the stack, either from the program output if you create the stack with a command line tool, or with the CloudFormation API.
You can also use the CloudFormation API to fetch a list of all created resources from a stack, or the EC2 API to get a list of all instances in a region and then filter them, perhaps by launch time.
Suppose your CloudFormation stack creates EC2 instances called Tom and Jerry. Then you would add an Outputs section something like the following:
"Outputs": {
"TomId": {
"Description": "Tom's instance id",
"Value": {"Ref": "Tom"}
},
"JerryId": {
"Description": "Jerry's instance id",
"Value": {"Ref": "Jerry"}
}
}
If you create the stack from the console, there will be a tab for the stack's outputs, and that will have a table with the two instance IDs included in it. Most command-line tools for creating a stack will also have some way of including the outputs.
But you asked for how to do it programmatically. Here's an example in Python, using the excellent boto library:
import boto.cloudformation
cf = boto.cloudformation.connect_to_region('us-east-1', aws_access_key_id='MyAccessKey', aws_secret_access_key='MySecretKey')
stacks = cf.describe_stacks('MyStackName')
stack = stacks[0]
for output in stack.outputs:
print "%s: %s" % (output.key, output.value)
Note that the describe_stacks method returns an array of matching stacks. There should be only one, but you can check that.
I am creating an RDS instance using CloudFormation using this:
"Resources": {
"myDB": {
"Type": "AWS::RDS::DBInstance",
"Properties": {
"AllocatedStorage": "5",
"DBInstanceClass": "db.m1.small",
"Engine": "MySQL",
"EngineVersion": "5.5",
"DBName": "mydb",
"MasterUsername": {
"Ref": "DBUser"
},
"MasterUserPassword": {
"Ref": "DBPassword"
},
"DBParameterGroupName": {
"Ref": "myRDSParamGroup"
}
}
}
and it all works. But I need to run initial SQL on the DB when its created, to setup my apps schema. My current approach is to have the app self migrating, but I'd like to do it in the CloudFormation definition. Is this possible?
No, it's not possible. However, you could have an EC2 instance connect to your RDS instance to do it. I'd probably store a .sql file in S3 and use a cloud-init script on the EC2 instance to download the file and execute it.
It would also be possible to create a CloudFormation custom resource. There is a good discussion about how to build one using SNS here; it is also possible to build one using Lambda. Custom resources are essentially just RPCs, so it wouldn't be difficult to create one to initialize a database with a schema, for example.
CloudFormation still doesn't hold any solutions for us, but hopefully they will add Database Migration Service support soon.
In the meantime, there is great solution if you're using CodePipeline: create a migration stage that invokes a Lambda function to run your migration. I stumbled across this guide for invoking Lambda from CodePipeline that may be helpful for those unfamiliar.
Another option is to use DBSnapshotIdentifier property for AWS::RDS::DBInstance resource. The only catch is that you need to have a DB loaded in AWS to create the snapshot in the first place. From then on, you can automate your cloudformation stack to be using it though.
DBSnapshotIdentifier:
Name (ARN) of the DB snapshot that's used to restore the DB instance.
If the property contains a value (other than an empty string), AWS CloudFormation creates a database from the specified snapshot.
After you restore a DB instance with a DBSnapshotIdentifier property, you must specify the same DBSnapshotIdentifier property for any future updates to the DB instance. When you specify this property for an update, the DB instance is not restored from the DB snapshot again, and the data in the database is not changed. However, if you don't specify the DBSnapshotIdentifier property, an empty DB instance is created, and the original DB instance is deleted.
Look in the doc from more info:
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-rds-database-instance.html#cfn-rds-dbinstance-dbsnapshotidentifier