Related
I am trying to setup the AWS environment for my SpringBoot application which uses Postgres.
I decided to use CloudFormation to configure all the AWS components needed for my application.
Following is the cloud formation template app.json:
{
"AWSTemplateFormatVersion": "2010-09-09",
"Parameters": {
"S3BucketName": {
"Description": "S3 BucketName",
"Type": "String"
},
"S3FileName": {
"Description": "Name of the jar/war file",
"Type": "String"
},
"DBName":{
"Default":"mytododb",
"Description":"The database name",
"Type":"String",
"MinLength":"1",
"MaxLength":"64",
"AllowedPattern":"[a-zA-Z][a-zA-Z0-9]*",
"ConstraintDescription":"must begin with a letter and contain only alphanumeric characters."
},
"DBUser":{
"Description":"The database admin account username",
"Type":"String",
"MinLength":"1",
"MaxLength":"16",
"AllowedPattern":"[a-zA-Z][a-zA-Z0-9]*",
"ConstraintDescription":"must begin with a letter and contain only alphanumeric characters."
},
"DBPassword":{
"NoEcho":"true",
"Description":"The database admin account password",
"Type":"String",
"MinLength":"8",
"MaxLength":"41",
"AllowedPattern":"[a-zA-Z0-9]*",
"ConstraintDescription":"must contain only alphanumeric characters."
},
"DBAllocatedStorage":{
"Default":"5",
"Description":"The size of the database (Gb)",
"Type":"Number",
"MinValue":"5",
"MaxValue":"1024",
"ConstraintDescription":"must be between 5 and 1024Gb."
},
"DBInstanceClass":{
"Default":"db.t2.micro",
"Description":"The database instance type",
"Type":"String",
"ConstraintDescription":"must select a valid database instance type."
},
"MultiAZDatabase":{
"Default":"false",
"Description":"Create a multi-AZ RDS database instance",
"Type":"String",
"AllowedValues":[
"true",
"false"
],
"ConstraintDescription":"must be either true or false."
}
},
"Resources": {
"SpringBootApplication": {
"Type": "AWS::ElasticBeanstalk::Application",
"Properties": {
"Description":"Spring boot and elastic beanstalk"
}
},
"SpringBootApplicationVersion": {
"Type": "AWS::ElasticBeanstalk::ApplicationVersion",
"Properties": {
"ApplicationName":{"Ref":"SpringBootApplication"},
"SourceBundle": {
"S3Bucket": {
"Ref": "S3BucketName"
},
"S3Key": {
"Ref": "S3FileName"
}
}
}
},
"SpringBootBeanStalkConfigurationTemplate": {
"Type": "AWS::ElasticBeanstalk::ConfigurationTemplate",
"Properties": {
"ApplicationName": {"Ref":"SpringBootApplication"},
"Description":"A display of speed boot application",
"OptionSettings": [
{
"Namespace": "aws:rds:dbinstance",
"OptionName": "DBAllocatedStorage",
"Value": {
"Ref": "DBAllocatedStorage"
}
},
{
"Namespace": "aws:rds:dbinstance",
"OptionName": "DBDeletionPolicy",
"Value": "Delete"
},
{
"Namespace": "aws:rds:dbinstance",
"OptionName": "DBEngine",
"Value": "postgres"
},
{
"Namespace": "aws:rds:dbinstance",
"OptionName": "DBEngineVersion",
"Value": "10.4"
},
{
"Namespace": "aws:rds:dbinstance",
"OptionName": "DBInstanceClass",
"Value": { "Ref": "DBInstanceClass" }
},
{
"OptionName": "DBPassword",
"Namespace": "aws:rds:dbinstance",
"Value": { "Ref": "DBPassword" }
},
{
"Namespace": "aws:rds:dbinstance",
"OptionName": "DBUser",
"Value": { "Ref": "DBUser" }
},
{
"Namespace": "aws:rds:dbinstance",
"OptionName": "MultiAZDatabase",
"Value": { "Ref": "MultiAZDatabase" }
}
],
"SolutionStackName": "64bit Amazon Linux 2018.03 v2.9.2 running Java 8"
}
},
"SpringBootBeanstalkEnvironment": {
"Type": "AWS::ElasticBeanstalk::Environment",
"Properties": {
"ApplicationName": {"Ref":"SpringBootApplication"},
"EnvironmentName":"TodoAppEnvironment",
"TemplateName": {"Ref":"SpringBootBeanStalkConfigurationTemplate"},
"VersionLabel": {"Ref": "SpringBootApplicationVersion"}
}
}
},
"Outputs": {
"DevURL": {
"Description": "The URL of the DEV Elastic Beanstalk environment",
"Value": {
"Fn::Join": [
"",
[
{
"Fn::GetAtt": [
"SpringBootBeanstalkEnvironment",
"EndpointURL"
]
}
]
]
},
"Export": {
"Name": {
"Fn::Sub": "${AWS::StackName}-EndpointURL"
}
}
}
}
}
And the parameters file parameters.json is:
[
{
"ParameterKey": "DBUser",
"ParameterValue": "myuser"
},
{
"ParameterKey": "DBPassword",
"ParameterValue": "secret"
},
{
"ParameterKey": "S3BucketName",
"ParameterValue": "todoapp"
},
{
"ParameterKey": "S3FileName",
"ParameterValue": "todo-api-spring-boot-0.0.1.jar"
}
]
I am creating the CloudFormation stack using
aws cloudformation create-stack --template-body file://app.json --parameters file://parameters.json --stack-name=todo-stack
Problem:
The stack is getting created, however the database (RDS) instance is not getting created and when I see in ElasticBeanstalk configuration for the app in Management Console there is no Database configuration associated with the application.
My expectation/assumption is by configuring various aws:rds:dbinstance properties in OptionSettings an RDS instance will be provisioned and associates with ElasticBeanstalk application.
Is my understanding wrong or am I missing any other setting?
PS:
The idea is to use integrated RDS so that I can use RDS_HOST, RDS_PORT etc properties in my application to connect to database.
I am able to configure a separate RDS Resource and connect to it by specifying the connection parameters. But I am expecting aws:rds:dbinstance properties in OptionSettings would create the RDS and associate it to Elasticbeanstalk application. If that is not the case then what is the purpose of configuring those parameters in OptionSettings?
Ref:
https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/command-options-general.html#command-options-general-rdsdbinstance
https://github.com/metabase/metabase/blob/master/bin/aws-eb-docker/cloudformation-elasticbeanstalk.json.template
RDS creation is not done due to the Namespace Approach. Basically Namespace is used to override the default properties of the resources which you have created.
So the solution is to include the RDS creation template before your "SpringBootBeanStalkConfigurationTemplate"
"Resources": {
"RDS Creation": {
{
"Type" : "AWS::RDS::DBInstance",
"Properties" : {
"AllocatedStorage" : String,
"AllowMajorVersionUpgrade" : Boolean,
"AssociatedRoles" : [ DBInstanceRole, ... ],
"AutoMinorVersionUpgrade" : Boolean,
"AvailabilityZone" : String,
"BackupRetentionPeriod" : Integer,
"CharacterSetName" : String,
"CopyTagsToSnapshot" : Boolean,
"DBClusterIdentifier" : String,
"DBInstanceClass" : String,
"DBInstanceIdentifier" : String,
"DBName" : String,
"DBParameterGroupName" : String,
"DBSecurityGroups" : [ String, ... ],
"DBSnapshotIdentifier" : String,
"DBSubnetGroupName" : String,
"DeleteAutomatedBackups" : Boolean,
"DeletionProtection" : Boolean,
"Domain" : String,
"DomainIAMRoleName" : String,
"EnableCloudwatchLogsExports" : [ String, ... ],
"EnableIAMDatabaseAuthentication" : Boolean,
"EnablePerformanceInsights" : Boolean,
"Engine" : String,
"EngineVersion" : String,
"Iops" : Integer,
"KmsKeyId" : String,
"LicenseModel" : String,
"MasterUsername" : String,
"MasterUserPassword" : String,
"MonitoringInterval" : Integer,
"MonitoringRoleArn" : String,
"MultiAZ" : Boolean,
"OptionGroupName" : String,
"PerformanceInsightsKMSKeyId" : String,
"PerformanceInsightsRetentionPeriod" : Integer,
"Port" : String,
"PreferredBackupWindow" : String,
"PreferredMaintenanceWindow" : String,
"ProcessorFeatures" : [ ProcessorFeature, ... ],
"PromotionTier" : Integer,
"PubliclyAccessible" : Boolean,
"SourceDBInstanceIdentifier" : String,
"SourceRegion" : String,
"StorageEncrypted" : Boolean,
"StorageType" : String,
"Tags" : [ Tag, ... ],
"Timezone" : String,
"UseDefaultProcessorFeatures" : Boolean,
"VPCSecurityGroups" : [ String, ... ]
}
}
}
}
After this, you can include the RDS parameters in your other Resources as it will be created first.
I was stumbling across the same question and found some sort of an answer here: https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features.managing.db.html
In that document they say:
You can configure your environment's database instance using configuration files. Use the options in the aws:rds:dbinstance namespace. The following example modifies the allocated database storage size to 100 GB.
So the aws:rds:dbinstance properties in OptionSettings are only meant to modify an existing database instance, not to create one in the first place.
I am trying to create a cloud formation template to create EC2 instance. I would like to use Security Group Name instead of Security Group Id. When I use that I am getting error "Encountered unsupported property SecurityGroup". How can I use Security Group Name while creating EC2 Instance through Cloud Formation
"Resources":
{
"EC2Instance":
{
"Type" : "AWS::EC2::Instance",
"Properties":
{
"InstanceType":
{
"Ref": "InstanceType"
},
"SecurityGroup":
[
{
"Ref" : "InstanceSecurityGroup"
}
],
"KeyName":
{
"Ref" : "AWS::Region"
},
"ImageId":
{
"Ref": "AMI"
}
}
},
"InstanceSecurityGroup":
{
"Type":"AWS::EC2::SecurityGroup",
"Properties":
{
"GroupDescription": "Enable SSH access via port 22",
"GroupName":
{
"Fn::FindInMap":
[
"EnvironmentConfig",
{
"Ref": "Environment"
},
"SGGroupName"
]
},
"SecurityGroupIngress":
[
{
"IpProtocol": "tcp",
"FromPort": 22,
"ToPort": 22,
"CidrIp":"10.252.0.0/16"
},
{
"IpProtocol": "tcp",
"FromPort": 22,
"ToPort": 22,
"CidrIp":"10.251.0.0/16"
}
],
"VpcId":
{
"Fn::FindInMap":
[
"EnvironmentConfig",
{
"Ref":"Environment"
},
"VPC"
]
}
}
}
It should be SecurityGroups instead of SecurityGroup, i.e.
"SecurityGroups" : [{ "Ref" : "InstanceSecurityGroup" }]
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/quickref-ec2.html
I have a VPC setup to that my lambda function can talk to my RDS server. This is working. I also need my lambda functions to have access to the internet. To this end I'm trying to setup an internet gateway and the routes to allow it. I'm failing.
The VPC routes and gateway are created as the following
"VPC": {
"Type": "AWS::EC2::VPC",
"Properties": {
"CidrBlock": "10.0.0.0/16",
"EnableDnsSupport": "true",
"EnableDnsHostnames": "true",
"InstanceTenancy": "default",
"Tags" : [{ "Key": "Name", "Value": { "Ref": "DomainName" } }]
}
},
"VPCRouteTable" : {
"Type" : "AWS::EC2::RouteTable",
"Properties" : {
"VpcId" : { "Ref" : "VPC" },
"Tags" : [{ "Key": "Name", "Value": { "Ref": "DomainName" } }]
}
},
"InternetGateway" : {
"Type" : "AWS::EC2::InternetGateway",
"Properties" : {
"Tags" : [{ "Key": "Name", "Value": { "Ref": "DomainName" } }]
}
},
"AttachGateway": {
"Type" : "AWS::EC2::VPCGatewayAttachment",
"Properties" : {
"VpcId" : { "Ref" : "VPC" },
"InternetGatewayId" : { "Ref" : "InternetGateway" }
}
},
"InternetRoute" : {
"Type" : "AWS::EC2::Route",
"DependsOn" : "InternetGateway",
"Properties" : {
"RouteTableId" : { "Ref" : "VPCRouteTable" },
"DestinationCidrBlock" : "0.0.0.0/0",
"GatewayId" : { "Ref" : "InternetGateway" }
}
},
I create the subnets and associate them with the route table
"SubnetA": {
"Type": "AWS::EC2::Subnet",
"Properties": {
"VpcId": { "Ref": "VPC" },
"CidrBlock": "10.0.0.0/24",
"AvailabilityZone": { "Fn::Select": [ "0", { "Fn::GetAZs": { "Ref": "AWS::Region" } }]},
"Tags" : [{ "Key": "Name", "Value": { "Ref": "DomainName" } }]
}
},
"SubnetB": {
"Type": "AWS::EC2::Subnet",
"Properties": {
"VpcId": { "Ref": "VPC" },
"CidrBlock": "10.0.1.0/24",
"AvailabilityZone": { "Fn::Select": [ "1", { "Fn::GetAZs": { "Ref": "AWS::Region" } }]},
"Tags" : [{ "Key": "Name", "Value": { "Ref": "DomainName" } }]
}
},
"SubnetARouteTableAssociation" : {
"Type" : "AWS::EC2::SubnetRouteTableAssociation",
"Properties" : {
"SubnetId" : { "Ref" : "SubnetA" },
"RouteTableId" : { "Ref" : "VPCRouteTable" }
}
},
"SubnetBRouteTableAssociation" : {
"Type" : "AWS::EC2::SubnetRouteTableAssociation",
"Properties" : {
"SubnetId" : { "Ref" : "SubnetB" },
"RouteTableId" : { "Ref" : "VPCRouteTable" }
}
},
I have the database security groups
"DBSubnetGroup": {
"Type": "AWS::RDS::DBSubnetGroup",
"Properties": {
"DBSubnetGroupDescription": "Database Access",
"SubnetIds" : [{ "Ref": "SubnetA" }, { "Ref": "SubnetB" }],
"Tags" : [{ "Key": "Name", "Value": { "Ref": "DomainName" } }]
}
},
"DBEC2SecurityGroup": {
"Type": "AWS::EC2::SecurityGroup",
"Properties": {
"GroupDescription": "Security group for RDS DB Instance",
"VpcId": {"Ref": "VPC"},
"SecurityGroupIngress" : [{
"IpProtocol": "tcp",
"FromPort": "3306",
"ToPort": "3306",
"CidrIp": "10.0.0.0/16"
}],
"Tags" : [{ "Key": "Name", "Value": { "Ref": "DomainName" } }]
}
},
and the lambda security group
"LambdaSecurityGroup": {
"Type": "AWS::EC2::SecurityGroup",
"Properties": {
"GroupDescription": "Security group for Lambda",
"VpcId": {"Ref": "VPC"},
"Tags" : [{ "Key": "Name", "Value": { "Ref": "DomainName" } }]
}
},
So As it stands now, my lambda's can talk to the database just fine. but they can't reach the internet. What am I missing?
If your lambda function needs to have access to both your VPC resources and Internet, then create 2 subnets: public and private. Put your lambda in private subnet and configure NAT in public subnet.
From http://docs.aws.amazon.com/lambda/latest/dg/vpc.html
Therefore, if your Lambda function requires Internet access (for
example, to access AWS services that don't have VPC endpoints, such as
Amazon Kinesis), you can configure a NAT instance inside your VPC or
you can use the Amazon VPC NAT gateway. For more information, see NAT
Gateways in the Amazon VPC User Guide.
I have created a simple JSON template for launching an EC2 instance using CloudFormation. The JSON code works for RedHat ami-2051294a (in us-east-1 region) but does not work for Amazon Linux ami-0b33d91d.
What could be the issue?
"Resources": {
"Ec2Instance" :{
"Type" : "AWS::EC2::Instance",
"Properties" : {
"Tags": [
{ "Key" : "Name" , "Value" : "BIP-Spark" }
],
"KeyName": { "Ref" : "KeyName" },
"InstanceType" : { "Ref" : "BipDevInstanceType" },
"ImageId" : { "Ref" : "NATAMI" },
"IamInstanceProfile" : { "Ref": "RoleName" },
"BlockDeviceMappings" : [ {
"DeviceName" : "/dev/sda1",
"Ebs" : { "VolumeSize" : "30", "VolumeType": "gp2" }
},
{
"DeviceName" : "/dev/sdb",
"Ebs" : { "VolumeSize" : "30", "VolumeType": "gp2" }
}
],
"NetworkInterfaces" : [ {
"GroupSet": [ "sg-***" ],
"SubnetId": { "Ref" : "SubnetID" },
"AssociatePublicIpAddress": "true",
"DeleteOnTermination": "true",
"DeviceIndex":"0" }
]
}
}
}
}
Both AMIs launched perfectly well for me.
Here is the minimal version of the template that I used to successfully launch an Amazon EC2 instance with Amazon Linux AMI ami-0b33d91d:
{
"AWSTemplateFormatVersion":"2010-09-09",
"Resources": {
"Ec2Instance" :{
"Type" : "AWS::EC2::Instance",
"Properties" : {
"InstanceType" : "t2.micro",
"ImageId" : "ami-0b33d91d",
"NetworkInterfaces" : [ {
"GroupSet": [ "sg-xxxxxxxx" ],
"SubnetId": "subnet-xxxxxxxx",
"AssociatePublicIpAddress": "true",
"DeleteOnTermination": "true",
"DeviceIndex":"0" }
]
}
}
}
}
The instance successfully entered the Running state and kept running correctly.
It is likely that one of your manually-entered parameters was incorrect.
When I launch a RDS instance manually I'm able to assign what VPC I want it to be part of. I'm trying to create a stack using AWS cloudformation, however I do not see an API to be able to do that. I can create my VPC in the stack and then reference it for security groups both EC2 and DB security groups and they both end up been part of the VPC however the RDS instance itself does not. Is there a way to assign the VPC to the RDS instance?
Below is my template:
{
"AWSTemplateFormatVersion": "2010-09-09",
"Metadata": {
"AWS::CloudFormation::Designer": {
"30e03bfc-b61a-4d6c-89db-1b62b258a305": {
"size": {
"width": 80,
"height": 80
},
"position": {
"x": 700,
"y": 170
},
"z": 0,
"embeds": []
}
}
},
"Parameters": {
"DBPreferredBkupWindow": {
"Description" : "The daily time range (in UTC) during which automated backups are created, ideally off peak-hours.",
"Type" : "String",
"MinLength" : "1",
"MaxLength" : "11",
"AllowedPattern" : "\\d[0-23]:\\d[0-59]-\\d[0-23]:\\d[0-59]",
"Default" : "01:00-02:00"
}
},
"Resources": {
"VPC": {
"Type": "AWS::EC2::VPC",
"Properties": {
"CidrBlock" : "172.16.0.0/16",
"EnableDnsSupport" : true
}
},
"DB": {
"Type": "AWS::RDS::DBInstance",
"Properties": {
"DBName" : "ems",
"Engine" : "postgres",
"EngineVersion" : "9.4.7",
"DBInstanceClass" : "db.t1.micro",
"DBInstanceIdentifier" : "rltdb",
"MasterUsername" : "pgadmin",
"MasterUserPassword" : "pgadmin1",
"AllocatedStorage" : "100",
"Iops" : "1000",
"BackupRetentionPeriod" : "7",
"PreferredBackupWindow" : { "Ref" : "DBPreferredBkupWindow" },
"MultiAZ" : true,
"PubliclyAccessible" : false,
"AutoMinorVersionUpgrade" : false,
"VPCSecurityGroups" : [{ "Ref" : "SecurityGroup" } ]
},
"Metadata": {
"AWS::CloudFormation::Designer": {
"id": "30e03bfc-b61a-4d6c-89db-1b62b258a305"
}
}
},
"DBSecurityGroup": {
"Type": "AWS::RDS::DBSecurityGroup",
"Properties": {
"EC2VpcId" : { "Ref" : "VPC" },
"DBSecurityGroupIngress" : { "EC2SecurityGroupName": { "Ref": "SecurityGroup"} },
"GroupDescription" : "Database Access"
}
},
"SecurityGroup" : {
"Type" : "AWS::EC2::SecurityGroup",
"Properties" : {
"VpcId" : { "Ref" : "VPC" },
"GroupDescription" : "Enable database access for application",
"SecurityGroupIngress" : [
{"IpProtocol" : "tcp", "FromPort" : "5432", "ToPort" : "5432", "CidrIp" : "0.0.0.0/0"}
]
}
}
}
}
You have to create a DBSubnetGroup and at least two subnets in your CloudFormation template.
"subnet-1" : {
"Type" : "AWS::EC2::Subnet",
"Properties" : {
"CidrBlock" : "172.16.1.0/24",
"VpcId" : { "Ref" : "VPC" }
}
},
"subnet-2" : {
"Type" : "AWS::EC2::Subnet",
"Properties" : {
"CidrBlock" : "172.16.2.0/24",
"VpcId" : { "Ref" : "VPC" }
}
},
"DBSubnetGroup" : {
"Type" : "AWS::RDS::DBSubnetGroup",
"Properties" : {
"SubnetIds" : [
{ "Ref" : "subnet-1" },
{ "Ref" : "subnet-2" }
],
}
},
and in last you have to include DBSubnetGroup in your "DB" Object.
"DBSubnetGroupName": { "Ref": "DBSubnetGroup" }
You need to include the DBSubnetGroupName:
A DB subnet group to associate with the DB instance.
If there is no DB subnet group, then it is a non-VPC DB instance.
Create a DBSubnetGroup resource using subnets in your VPC, then tie that to your DBInstance:
"DBSubnetGroupName": { "Ref": "MySubnetGroup" }