I need to create Elastic Beanstalk Environment from boto3.
For which I guess the API sequence should be:
create_application()
From this API we get the "Application Name":
create_environment(kwargs)
Here i am passing below json as kwargs to api
{
"ApplicationName": "APP-NAME",
"EnvironmentName": "ABC-Nodejs",
"CNAMEPrefix": "ABC-Neptune",
"SolutionStackName": "64bit Amazon Linux 2016.03 v2.1.1 running Node.js"
}
Questions:
How to mention that the Environment EC2 should be attached to which
VPC and subnet
In which Subnet its ELB should be created
Any sample code will be helpful
Please Note: I have one public and one private subnet, we can control the creation of EC2 and ELB creation through subnet IDs
To set up dependent resources with your Environment you would have to use the Elastic Beanstalk Option Settings. Specifically for VPCs you can use the aws:ec2:vpc namespace, I've linked the documentation for those settings with that.
The code example would be something like this:
{
ApplicationName: "APP-NAME",
EnvironmentName: "ABC-Nodejs",
CNAMEPrefix: "ABC-Neptune",
SolutionStackName: "64bit Amazon Linux 2016.03 v2.1.1 running Node.js"
OptionSettings=[
{
'Namespace': 'aws:ec2:vpc',
'OptionName': 'VPCId',
'Value': 'vpc-12345678'
},
{
'Namespace': 'aws:ec2:vpc',
'OptionName': 'ELBSubnets',
'Value': 'subnet-11111111,subnet-22222222'
},
],
}
Thanks nbalas,
I am using below code to create EB.
Despite giving already created security group names in "aws:elb:loadbalancer" and "aws:autoscaling:launchconfiguration" it is creating new security groups and attaching them to EC2 instance and load balancer. So now both of the security groups new and old ones are attached to the resources. I don't want to create new security groups at all and want to use old ones only.
kwargs={
"ApplicationName": "Test",
"EnvironmentName": "ABC-Nodejs",
"CNAMEPrefix": "ABC-ABC",
"SolutionStackName": "64bit Amazon Linux 2016.03 v2.1.1 running Node.js",
"OptionSettings": [
{
"Namespace": "aws:ec2:vpc",
"OptionName": "Subnets",
"Value": "subnet-*******0"
},
{
"Namespace": "aws:ec2:vpc",
"OptionName": "ELBSubnets",
"Value": "subnet-********1"
},
{
"Namespace": "aws:elb:loadbalancer",
"OptionName": "SecurityGroups",
"Value": "sg-*********2"
},
{
"Namespace": "aws:autoscaling:launchconfiguration",
"OptionName": "SecurityGroups",
"Value": "sg-**********3"
}
]
}
response = client.create_environment(**kwargs)
Related
According the AWS docs at here and here I should be able to automate a certificate creation and validation using cloudformation. Apparently when you specify a HostedZoneId in the DomainValidationOptions, it is supposed to create the required DNS record to complete the validation (at least that is what it seems from the very vague documentation). My CF template for the cert looks like this:
Resources:
MyAPICert:
Type: AWS::CertificateManager::Certificate
Properties:
DomainName: xxxx.dev.mydomain.io
DomainValidationOptions:
- DomainName: mydomain.io
HostedZoneId: /hostedzone/Z03XXXXXXXXXXXX
ValidationMethod: DNS
'mydomain.io' (changed of course) was registered using AWS as registrar as the documents say must be the case for automated validation to work.
This template above is included in a serverless.yml as a resource. However, when I deploy, the stack creation is just stuck waiting for the DNS record - i.e. it does not add the required CNAME entry as I understand it is supposed to do and as such the stack is stuck.
Has anyone gotten this feature to work?
And, yes, I know about the 3rd party custom resources that try to do the same thing, I don't want to use them if CF is supposed to do this natively now.
I hit the same issue. You need to specify the full domain name including the host in the DomainValidationOptions DomainName parameter, and just specify the hosted zone id:
Resources:
MyAPICert:
Type: AWS::CertificateManager::Certificate
Properties:
DomainName: xxxx.dev.mydomain.io
DomainValidationOptions:
- DomainName: xxxx.dev.mydomain.io
HostedZoneId: Z03XXXXXXXXXXXX
ValidationMethod: DNS
In my testing, the Route53 validation record was added about a minute after running the stack, and the domain successfully validated itslef after about 15 minutes.
If this is stuck as in progress for a long time, it could be that you are using a Private Hosted Zone when you need to use the Public one. Probably you don't use a private CA.
That process should take 2-3 minutes, not more than that.
I just deployed the below template to CloudFormation and it successfully created the validation DNS records and authorised the certificate.
If you were to pass the parameters SiteDnsZoneName=mydomain.io. and SiteDnsZoneId=ABCDEFGHIJKLMNOPQRSTU it would create a SAN cert that covers both mydomain.io and *.mydomain.io
{
"Description": "Deploy wildcard SAN cert inc bare domain. (Must deploy cert to us-east-1 for CloudFront)",
"Parameters": {
"SiteDnsZoneName": {
"Type": "String",
"MinLength": 4,
"Description": "DNS Zone",
"Default": "example.com"
},
"SiteDnsZoneId": {
"Type": "String",
"MinLength": 8,
"Description": "DNS Zone Id",
"Default": "ABCDEFGHIJKLMNOPQRSTU"
}
},
"Resources": {
"SiteCertificate": {
"Type": "AWS::CertificateManager::Certificate",
"Properties": {
"DomainName": {
"Fn::Join": [
".",
[
"*",
{
"Ref": "SiteDnsZoneName"
}
]
]
},
"SubjectAlternativeNames": [
{
"Ref": "SiteDnsZoneName"
}
],
"DomainValidationOptions": [
{
"DomainName": {
"Ref": "SiteDnsZoneName"
},
"HostedZoneId": {
"Ref": "SiteDnsZoneId"
}
}
],
"ValidationMethod": "DNS"
}
}
}
}
Note: If you want to use a cert in CloudFront you have to deploy the cert in us-east-1.
Note 2: Route53 needs to be hosting your DNS Zone, but theres no requirement on AWS being the registrar. Your domain can be registered with any provider, so long as you use the AWS name servers provided by Route53 when you add the zone.
At my AWS account, I have few VPC. I'm trying to find a way to list all resources that located under a certain VPC.
Thanks!!
You can do it in three ways: AWS CLI, AWS console or code.
AWS CLI
You can use AWS CLI to list all ENIs associated with the VPC and prettify the output using the --query parameter to get a resource list with the desired fields (AZ, instance-id, etc.).
aws ec2 describe-network-interfaces --filters Name=vpc-id,Values=<vpc-id> --query 'NetworkInterfaces[*].[AvailabilityZone, OwnerId, Attachment.InstanceId, PrivateIpAddresses[*].Association.PublicIp]'
aws ec2 describe-network-interfaces --filters Name=vpc-id,Values=<vpc-id> --query 'NetworkInterfaces[*].[RequesterId,Description]'
A sample of the raw output (only one instance on the VPC):
"NetworkInterfaces": [
{
"Association": {
"IpOwnerId": "amazon",
"PublicDnsName": "ec2-54-196-57-169.compute-1.amazonaws.com",
"PublicIp": "54.196.57.169"
},
"Attachment": {
"AttachTime": "2020-08-24T10:59:16+00:00",
"AttachmentId": "eni-attach-047e562690aabbffd",
"DeleteOnTermination": true,
"DeviceIndex": 0,
"InstanceId": "i-0fe495a6c17bd0f82",
"InstanceOwnerId": "570398916848",
"Status": "attached"
},
"AvailabilityZone": "us-east-1d",
"Description": "",
"Groups": [
{
"GroupName": "launch-wizard-1",
"GroupId": "sg-0aa7d8257bb487e1b"
}
],
"InterfaceType": "interface",
"Ipv6Addresses": [],
"MacAddress": "0e:58:38:33:9a:31",
"NetworkInterfaceId": "eni-0b20855178d276783",
"OwnerId": "570398916848",
"PrivateDnsName": "ip-172-31-34-30.ec2.internal",
"PrivateIpAddress": "172.31.34.30",
"PrivateIpAddresses": [
{
"Association": {
"IpOwnerId": "amazon",
"PublicDnsName": "ec2-54-196-57-169.compute-1.amazonaws.com",
"PublicIp": "54.196.57.169"
},
"Primary": true,
"PrivateDnsName": "ip-172-31-34-30.ec2.internal",
"PrivateIpAddress": "172.31.34.30"
}
],
"RequesterManaged": false,
"SourceDestCheck": true,
"Status": "in-use",
"SubnetId": "subnet-e2bc5fbd",
"TagSet": [],
"VpcId": "vpc-6ad2e110"
}
]
And now filtered:
For the first --query
[
"us-east-1d",
"57039816848",
"i-0fe495a6c17bd0f82",
[
"44.196.57.169"
]
]
And for the second --query (another VPC):
[
"amazon-elasticache",
"ElastiCache alon-001"
],
[
"amazon-elasticache",
"ElastiCache alon-002"
],
[
"975289786086",
"arn:aws:ecs:us-east-2:57039916848:attachment/22a90802-fae7-4afb-9a7e-43e6f4be8ca4"
],
[
"074689309192",
"Interface for NAT Gateway nat-069344579d8bda20"
],
[
"amazon-elb",
"ELB app/EC2Co-EcsEl-YX74WCWEGOK/0b6d7bc60b540b1"
],
[
"amazon-elb",
"ELB app/EC2Co-EcsEl-YX74WCWGGOK/0b6bd7c60b540b1"
],
[
"amazon-elasticache",
"ElastiCache alon-003"
]
AWS Console
You can do the same using the AWS console.
Under EC2->Network Interfaces, search for the desired vpc-id in the search bar.
Code
Using a python script called vpc-inside.py you can describe all of your VPC resources.
usage: vpc-inside.py [-h] -v VPC [-r REGION] [-p PROFILE]
optional arguments:
-h, --help show this help message and exit
-v VPC, --vpc VPC The VPC to annihilate
-r REGION, --region REGION AWS region that the VPC resides in
-p PROFILE, --profile PROFILE AWS profile
And the output will look like this:
EKSs in VPC vpc-07ef7f777429cfd82:
Omikron
--------------------------------------------
ASGs in VPC vpc-07ef7f777429cfd82:
eks-pooks-9ebf225b-70a9-a026-034f-c7431df9b7ba resides in vpc-07ef7f777429cfd82
eks-pooks-9ebf225b-70a9-a026-034f-c7431df9b7ba
--------------------------------------------
RDSs in VPC vpc-07ef7f777429cfd82:
--------------------------------------------
EC2s in VPC vpc-07ef7f777429cfd82:
i-0c63874d77ea2ba78
i-043740f224015e69e
--------------------------------------------
Lambdas in VPC vpc-07ef7f777429cfd82:
--------------------------------------------
Classic ELBs in VPC vpc-07ef7f777429cfd82:
--------------------------------------------
ELBs V2 in VPC vpc-07ef7f777429cfd82:
--------------------------------------------
NAT GWs in VPC vpc-07ef7f777429cfd82:
--------------------------------------------
VPC EndPoints in VPC vpc-07ef7f777429cfd82:
--------------------------------------------
IGWs in VPC vpc-07ef7f777429cfd82:
--------------------------------------------
ENIs in VPC vpc-07ef7f777429cfd82:
eni-079231232dc136305
eni-05ff227eca8341a08
eni-0c01b2871887ac3f7
eni-00e11d4f9590161b4
--------------------------------------------
Security Groups in VPC vpc-07ef7f777429cfd82:
sg-0b4554a65e1560745
sg-0f93574d6b180b263
--------------------------------------------
Routing tables in VPC vpc-07ef7f777429cfd82:
rtb-0694bdbdd696b2bed
rtb-072ec82a18d8a04ba
--------------------------------------------
ACLs in VPC vpc-07ef7f777429cfd82:
acl-0c0087eabf9335940
--------------------------------------------
Subnets in VPC vpc-07ef7f777429cfd82:
subnet-0b8cc1132727e5b5d
subnet-0e47ee92a9ca80280
subnet-0c25990d9a138616b
--------------------------------------------
You can try in AWS Config > Advanced queries and run below query :
All resources:
SELECT
resourceId,
resourceName,
resourceType
Resources directly associated to VPC:
SELECT
resourceId,
resourceName,
resourceType
WHERE
relationships.resourceId = 'vpc-02368dae78f1387e5'
Query can be further enhanced, see some example of preconfigured query.
VPCs mostly contain EC2 instances, RDS instances, Load Balancers and Lambda functions. Plus, things that use EC2 underneath, like Elasticache. These are the types of resources that connect into a VPC.
Some people suggest using the Tag Editor to find resources: Is there a way to list all resources in AWS.
I also like aws inventory, which simply runs in your browser and does a great job of showing resources. Just give it an Access Key and Secret Key to run.
There's no built in service to easily do this.
The best hope you'd have of find all resources is programatically looping over resources that support:
SubnetId
VpcId
Today when I launch an app using kubernetes over aws it exposes a publicly visible LoadBalancer Ingress URL, however to link that to my domain to make the app accessible to the public, I need to manually go into the aws route53 console in a browser on every launch. Can I update the aws route53 Resource Type A to match the latest Kubernetes LoadBalancer Ingress URL from the command line ?
Kubernetes over gcloud shares this challenge of having to either predefine a Static IP which is used in launch config or manually do a browser based domain linkage post launch. On aws I was hoping I could use something similar to this from the command line
aws route53domains update-domain-nameservers ???
__ OR __ can I predefine an aws kubernetes LoadBalancer Ingress similar to doing a predefined Static IP when over gcloud ?
to show the deployed app's LoadBalancer Ingress URL issue
kubectl describe svc
... output
Name: aaa-deployment-407
Namespace: ruptureofthemundaneplane
Labels: app=bbb
pod-template-hash=4076262206
Selector: app=bbb,pod-template-hash=4076262206
Type: LoadBalancer
IP: 10.0.51.82
LoadBalancer Ingress: a244bodhisattva79c17cf7-61619.us-east-1.elb.amazonaws.com
Port: port-1 80/TCP
NodePort: port-1 32547/TCP
Endpoints: 10.201.0.3:80
Port: port-2 443/TCP
NodePort: port-2 31248/TCP
Endpoints: 10.201.0.3:443
Session Affinity: None
No events.
UPDATE:
Getting error trying new command line technique (hat tip to #error2007s comment) ... issue this
aws route53 list-hosted-zones
... outputs
{
"HostedZones": [
{
"ResourceRecordSetCount": 6,
"CallerReference": "2D58A764-1FAC-DEB4-8AC7-AD37E74B94E6",
"Config": {
"PrivateZone": false
},
"Id": "/hostedzone/Z3II3949ZDMDXV",
"Name": "chainsawhaircut.com."
}
]
}
Important bit used below : hostedzone Z3II3949ZDMDXV
now I craft following using this Doc (and this Doc as well) as file /change-resource-record-sets.json (NOTE I can successfully change Type A using a similar cli call ... however I need to change Type A with an Alias Target of LoadBalancer Ingress URL)
{
"Comment": "Update record to reflect new IP address of fresh deploy",
"Changes": [{
"Action": "UPSERT",
"ResourceRecordSet": {
"Name": "chainsawhaircut.com.",
"Type": "A",
"TTL": 60,
"AliasTarget": {
"HostedZoneId": "Z3II3949ZDMDXV",
"DNSName": "a244bodhisattva79c17cf7-61619.us-east-1.elb.amazonaws.com",
"EvaluateTargetHealth": false
}
}
}]
}
on command line I then issue
aws route53 change-resource-record-sets --hosted-zone-id Z3II3949ZDMDXV --change-batch file:///change-resource-record-sets.json
which give this error message
An error occurred (InvalidInput) when calling the ChangeResourceRecordSets operation: Invalid request
Any insights ?
Here is the logic needed to update aws route53 Resource Record Type A with value from freshly minted kubernetes LoadBalancer Ingress URL
step 1 - identify your hostedzone Id by issuing
aws route53 list-hosted-zones
... from output here is clip for my domain
"Id": "/hostedzone/Z3II3949ZDMDXV",
... importantly never populate json with hostedzone Z3II3949ZDMDXV its only used as a cli parm ... there is a second similarly named token HostedZoneId which is entirely different
step 2 - see current value of your route53 domain record ... issue :
aws route53 list-resource-record-sets --hosted-zone-id Z3II3949ZDMDXV --query "ResourceRecordSets[?Name == 'scottstensland.com.']"
... output
[
{
"AliasTarget": {
"HostedZoneId": "Z35SXDOTRQ7X7K",
"EvaluateTargetHealth": false,
"DNSName": "dualstack.asomepriorvalue39e7db-1867261689.us-east-1.elb.amazonaws.com."
},
"Type": "A",
"Name": "scottstensland.com."
},
{
"ResourceRecords": [
{
"Value": "ns-1238.awsdns-26.org."
},
{
"Value": "ns-201.awsdns-25.com."
},
{
"Value": "ns-969.awsdns-57.net."
},
{
"Value": "ns-1823.awsdns-35.co.uk."
}
],
"Type": "NS",
"Name": "scottstensland.com.",
"TTL": 172800
},
{
"ResourceRecords": [
{
"Value": "ns-1238.awsdns-26.org. awsdns-hostmaster.amazon.com. 1 7200 900 1209600 86400"
}
],
"Type": "SOA",
"Name": "scottstensland.com.",
"TTL": 900
}
]
... in above notice value of
"HostedZoneId": "Z35SXDOTRQ7X7K",
which is the second similarly name token Do NOT use wrong Hosted Zone ID
step 3 - put below into your change file aws_route53_type_A.json (for syntax Doc see link mentioned in comment above)
{
"Comment": "Update record to reflect new DNSName of fresh deploy",
"Changes": [
{
"Action": "UPSERT",
"ResourceRecordSet": {
"AliasTarget": {
"HostedZoneId": "Z35SXDOTRQ7X7K",
"EvaluateTargetHealth": false,
"DNSName": "dualstack.a0b82c81f47d011e6b98a0a28439e7db-1867261689.us-east-1.elb.amazonaws.com."
},
"Type": "A",
"Name": "scottstensland.com."
}
}
]
}
To identify value for above field "DNSName" ... after the kubernetes app deploy on aws it responds with a LoadBalancer Ingress as shown in output of cli command :
kubectl describe svc --namespace=ruptureofthemundaneplane
... as in
LoadBalancer Ingress: a0b82c81f47d011e6b98a0a28439e7db-1867261689.us-east-1.elb.amazonaws.com
... even though my goal is to execute a command line call I can do this manually by getting into the aws console browser ... pull up my domain on route53 ...
... In this browser picklist editable text box (circled in green) I noticed the URL gets magically prepended with : dualstack. Previously I was missing that magic string ... so json key "DNSName" wants this
dualstack.a0b82c81f47d011e6b98a0a28439e7db-1867261689.us-east-1.elb.amazonaws.com.
finally execute the change request
aws route53 change-resource-record-sets --hosted-zone-id Z3II3949ZDMDXV --change-batch file://./aws_route53_type_A.json
... output
{
"ChangeInfo": {
"Status": "PENDING",
"Comment": "Update record to reflect new DNSName of fresh deploy",
"SubmittedAt": "2016-07-13T14:53:02.789Z",
"Id": "/change/CFUX5R9XKGE1C"
}
}
.... now to confirm change is live run this to show record
aws route53 list-resource-record-sets --hosted-zone-id Z3II3949ZDMDXV
You can also use external-dns project.
AWS specific setup can be found here
After installation it can be used with an annotation e.g.: external-dns.alpha.kubernetes.io/hostname: nginx.external-dns-test.my-org.com.
Note the IAM permissions needs to be set properly.
My goal is to have a Cloud Formation template which not only automatically creates a VPC with a NAT host and bastion host, but which deploys a .NET app pulled from S3 into an Elastic Beanstalk which is load balanced and more importantly only allows access to the app from my office, NOT the whole internet. It seems that even though the app might be in a VPC in a Private subnet and the ELB is in the Public subnet, that the Network ACL on the Public subnet is irrelevant. If I lock down the Public subnet to only my office, connections from outside the office can still come into the ELB and hit the application.
What seems to work is applying a Security group to the ELB, but I do not see any way to force the creation of a specific ELB with a specific SG inside a "AWS::ElasticBeanstalk::Environment" object. The ELB and ELB SG are created automatically by beanstalk and must be manually altered after CF runs. I don't want that. I want a way to create everything in CF in an automated way with no manual steps after the fact. yes, I've tried Cloud Former on a manually created stakc. No, it doesn't give me what I want.
Here's an excerpt from my CF Template:
"MyWebApp": {
"Type": "AWS::ElasticBeanstalk::Application",
"Properties": {
"ApplicationName" : "AlmDemoWebApp",
"Description": "MyWebapp"
}
},
"MyWebAppVersion": {
"Type": "AWS::ElasticBeanstalk::ApplicationVersion",
"Properties": {
"ApplicationName": {"Ref": "MyWebApp"},
"SourceBundle": {
"S3Bucket": "mywebapp",
"S3Key": {"Fn::Join" : ["", ["MyWebApp.", {"Ref":"Version"}, ".zip"]]}
}
}
},
"MyWebAppEnvironment" : {
"DependsOn" : ["MyWebApp", "MyWebAppVersion", "BastionSecurityGroup", "BeanstalkSecurityGroup", "VPC", "EBLoadBalancer", "EBLoadBalancerSecurityGroup", "PrivateSubnet", "PublicSubnet"],
"Type" : "AWS::ElasticBeanstalk::Environment",
"Properties" : {
"ApplicationName" : { "Ref" : "MyWebApp" },
"Description" : "MyWebApp Target Environment",
"SolutionStackName": "64bit Windows Server 2012 R2 running IIS 8.5",
"OptionSettings" : [
{"Namespace" : "aws:autoscaling:launchconfiguration", "OptionName" : "SecurityGroups", "Value" : { "Ref" : "BeanstalkSecurityGroup" }},
{"Namespace" : "aws:autoscaling:launchconfiguration", "OptionName" : "EC2KeyName", "Value" : { "Ref" : "InstanceKeyName" }},
{"Namespace" : "aws:ec2:vpc", "OptionName" : "VPCId", "Value" : { "Ref" : "VPC" }},
{"Namespace" : "aws:ec2:vpc", "OptionName" : "Subnets", "Value" : { "Ref" : "PrivateSubnet" }},
{"Namespace" : "aws:ec2:vpc", "OptionName" : "ELBSubnets", "Value" : { "Ref" : "PublicSubnet" }}],
"VersionLabel": {"Ref": "MyWebAppVersion"}
}
}
Is there some mysterious and poorly documented option that I can put in the AWS::ElasticBeanstalk::Environment -> Properties -> OptionSettings that will force the Elastic Beanstalk to use a specific ELB configured previously in the CF template ("EBLoadBalancer") rather than automatically create one with a random name? Applying ingress rules to the "BeanstalkSecurityGroup" doesn't seem to help. The SG rules have to be on the ELB to actually work, apparently.
Sounds like you need to create an internal ELB for your Elastic Beanstalk stack. For this, create a property as below.
"ELBScheme":{
"Type":"String",
"AllowedValues":[
"internal"
],
"Default":"internal",
"Description":"Internal load balancer in VPC so that your Elastic Beanstalk application cannot be accessed from outside your VPC."
}
And refer it as a namespace in your AWS::ElasticBeanstalk::Environment "MyWebAppEnvironment"
{
"Namespace":"aws:ec2:vpc",
"OptionName":"ELBScheme",
"Value":"internal"
}
How can I create an RDS instance with the create-environment or another subcommand of aws elasticbeanstalk? I've tried several combinations of parameters to no avail. Below is an example.
APP_NAME="randall-railsapp"
aws s3api create-bucket --bucket "$APP_NAME"
APP_VERSION="$(git describe --always)"
APP_FILE="deploy-$APP_NAME-$APP_VERSION.zip"
git archive -o "$APP_FILE" HEAD
aws s3 cp "$APP_FILE" "s3://$APP_NAME/$APP_FILE"
aws --region us-east-1 elasticbeanstalk create-application-version \
--auto-create-application \
--application-name "$APP_NAME" \
--version-label "$APP_VERSION" \
--source-bundle S3Bucket="$APP_NAME",S3Key="$APP_FILE"
aws --region us-east-1 elasticbeanstalk create-environment \
--application-name "$APP_NAME" \
--version-label "$APP_VERSION" \
--environment-name "$APP_NAME-env" \
--description "randall's rails app environment" \
--solution-stack-name "64bit Amazon Linux 2014.03 v1.0.0 running Ruby 2.1 (Puma)" \
--cname-prefix "$APP_NAME-test" \
--option-settings file://test.json
And the contents of test.json:
[
{
"OptionName": "EC2KeyName",
"Namespace": "aws:autoscaling:launchconfiguration",
"Value": "a-key-is-here"
},
{
"OptionName": "EnvironmentType",
"Namespace": "aws:elasticbeanstalk:environment",
"Value": "SingleInstance"
},
{
"OptionName": "SECRET_KEY_BASE",
"Namespace": "aws:elasticbeanstalk:application:environment",
"Value": "HAHAHAHAHAHA"
},
{
"OptionName": "DBPassword",
"Namespace": "aws:rds:dbinstance",
"Value": "hunter2"
},
{
"OptionName": "DBUser",
"Namespace": "aws:rds:dbinstance",
"Value": "random"
},
{
"OptionName": "DBEngineVersion",
"Namespace": "aws:rds:dbinstance",
"Value": "9.3"
},
{
"OptionName": "DBEngine",
"Namespace": "aws:rds:dbinstance",
"Value": "postgres"
}
]
Anyone know why this is failing? Anything I specify with a aws:rds:dbinstance namespace seems to get removed from the configuration.
Just setting the aws:rds:dbinstance options does not create an RDS database.
Currently you can create an RDS instance using one of the following techniques:
Create using AWS Console
Use eb cli
Use Resources section of ebextensions to create an RDS resource
The first two approaches are the most convenient as they do all the heavy lifting for you but for the third one you have to do some extra work. The third approach is what you would want to use if you are not using the console or eb CLI.
You can create an RDS resource for your beanstalk environment using the following ebextension snippet. Create a file called 01-rds.config in the .ebextensions directory of your app source.
Resources:
AWSEBRDSDatabase:
Type: AWS::RDS::DBInstance
Properties:
AllocatedStorage: 5
DBInstanceClass: db.t2.micro
DBName: myawesomeapp
Engine: postgres
EngineVersion: 9.3
MasterUsername: myAwesomeUsername
MasterUserPassword: myCrazyPassword
This file is in YAML format so indentation is important. You could also use JSON if you like.
These are not option settings so you cannot pass it as --option-settings test.json. You just need to bundle this file with your app source.
Read more about what properties you can configure on your RDS database here. On this page you can also find what properties are required and what properties are optional.
Let me know if the above does not work for you or if you have any further questions.
As of December 2017 we use the following ebextensions
$ cat .ebextensions/rds.config
Resources:
AWSEBRDSDBSecurityGroup:
Type: AWS::RDS::DBSecurityGroup
Properties:
EC2VpcId:
Fn::GetOptionSetting:
OptionName: "VpcId"
GroupDescription: RDS DB VPC Security Group
DBSecurityGroupIngress:
- EC2SecurityGroupId:
Ref: AWSEBSecurityGroup
AWSEBRDSDBSubnetGroup:
Type: AWS::RDS::DBSubnetGroup
Properties:
DBSubnetGroupDescription: RDS DB Subnet Group
SubnetIds:
Fn::Split:
- ","
- Fn::GetOptionSetting:
OptionName: DBSubnets
AWSEBRDSDatabase:
Type: AWS::RDS::DBInstance
DeletionPolicy: Delete
Properties:
PubliclyAccessible: true
MultiAZ: false
Engine: mysql
EngineVersion: 5.7
BackupRetentionPeriod: 0
DBName: test
MasterUsername: toor
MasterUserPassword: 123456789
AllocatedStorage: 10
DBInstanceClass: db.t2.micro
DBSecurityGroups:
- Ref: AWSEBRDSDBSecurityGroup
DBSubnetGroupName:
Ref: AWSEBRDSDBSubnetGroup
Outputs:
RDSId:
Description: "RDS instance identifier"
Value:
Ref: "AWSEBRDSDatabase"
RDSEndpointAddress:
Description: "RDS endpoint address"
Value:
Fn::GetAtt: ["AWSEBRDSDatabase", "Endpoint.Address"]
RDSEndpointPort:
Description: "RDS endpoint port"
Value:
Fn::GetAtt: ["AWSEBRDSDatabase", "Endpoint.Port"]
AWSEBRDSDatabaseProperties:
Description: Properties associated with the RDS database instance
Value:
Fn::Join:
- ","
- - Ref: AWSEBRDSDatabase
- Fn::GetAtt: ["AWSEBRDSDatabase", "Endpoint.Address"]
- Fn::GetAtt: ["AWSEBRDSDatabase", "Endpoint.Port"]
With such custom options
$ cat .ebextensions/custom-options.config
option_settings:
"aws:elasticbeanstalk:customoption":
DBSubnets: subnet-1234567,subnet-7654321
VpcId: vpc-1234567
The only things - you have to explicitly pass RDS_* env variables to your application.
The other answers did not work in my environment as of Sept 2015. After much trial and error, the following worked for me:
config template snippet (YAML):
aws:rds:dbinstance:
DBAllocatedStorage: '5'
DBDeletionPolicy: Delete
DBEngine: postgres
DBEngineVersion: 9.3.9
DBInstanceClass: db.t2.micro
DBPassword: PASSWORD_HERE
DBUser: USERNAME_HERE
MultiAZDatabase: false
.ebextensions/rds.config file (JSON):
{
"Parameters": {
"AWSEBDBUser": {
"NoEcho": "true",
"Description": "The name of master user for the client DB Instance.",
"Default": "ebroot",
"Type": "String",
"ConstraintDescription": "must begin with a letter and contain only alphanumeric characters"
},
"AWSEBDBPassword": {
"NoEcho": "true",
"Description": "The master password for the DB instance.",
"Type": "String",
"ConstraintDescription": "must contain only alphanumeric characters"
},
"AWSEBDBName": {
"NoEcho": "true",
"Description": "The DB Name of the RDS instance",
"Default": "ebdb",
"Type": "String",
"ConstraintDescription": "must contain only alphanumeric characters"
}
},
"Resources": {
"AWSEBAutoScalingGroup": {
"Metadata": {
"AWS::ElasticBeanstalk::Ext": {
"_ParameterTriggers": {
"_TriggerConfigDeployment": {
"CmpFn::Insert": {
"values": [
{
"CmpFn::Ref": "Parameter.AWSEBDBUser"
},
{
"CmpFn::Ref": "Parameter.AWSEBDBPassword"
},
{
"CmpFn::Ref": "Parameter.AWSEBDBName"
}
]
}
}
},
"_ContainerConfigFileContent": {
"plugins": {
"rds": {
"Description": "RDS Environment variables",
"env": {
"RDS_USERNAME": {
"Ref": {
"CmpFn::Ref": "Parameter.AWSEBDBUser"
}
},
"RDS_PASSWORD": {
"Ref": {
"CmpFn::Ref": "Parameter.AWSEBDBPassword"
}
},
"RDS_DB_NAME": {
"Ref": {
"CmpFn::Ref": "Parameter.AWSEBDBName"
}
},
"RDS_HOSTNAME": {
"Fn::GetAtt": [
"AWSEBRDSDatabase",
"Endpoint.Address"
]
},
"RDS_PORT": {
"Fn::GetAtt": [
"AWSEBRDSDatabase",
"Endpoint.Port"
]
}
}
}
}
}
}
}
},
"AWSEBRDSDatabase": {
"Type": "AWS::RDS::DBInstance",
"DeletionPolicy": "Delete",
"Properties": {
"DBName": {
"Ref": {
"CmpFn::Ref": "Parameter.AWSEBDBName"
}
},
"AllocatedStorage": "5",
"DBInstanceClass": "db.t2.micro",
"Engine": "postgres",
"DBSecurityGroups": [
{
"Ref": "AWSEBRDSDBSecurityGroup"
}
],
"MasterUsername": {
"Ref": {
"CmpFn::Ref": "Parameter.AWSEBDBUser"
}
},
"MasterUserPassword": {
"Ref": {
"CmpFn::Ref": "Parameter.AWSEBDBPassword"
}
},
"MultiAZ": false
}
},
"AWSEBRDSDBSecurityGroup": {
"Type": "AWS::RDS::DBSecurityGroup",
"Properties": {
"DBSecurityGroupIngress": {
"EC2SecurityGroupName": {
"Ref": "AWSEBSecurityGroup"
}
},
"GroupDescription": "Enable database access to Beanstalk application"
}
}
}
}
I had the same problem, couldn't get it to work via .ebextensions, and I don't like the EB CLI tool.
EB CLI uses an undocumented API feature, and a customized version of the botocore library ('eb_botocore') to make this work. :(
So I went ahead and forked botocore, and merged in the API data file used by eb_botocore: https://github.com/boto/botocore/pull/396
Then I ran 'python setup.py install' on both my modified botocore and aws-cli (both at master), and aws-cli now accepts a --template-specification option on the 'aws elasticbeanstalk create-environment' command. Hooray!
Example usage:
aws elasticbeanstalk create-environment\
...various options...\
--option-settings file://option-settings.json
--template-specification file://rds.us-west-2.json
where rds.us-west-2.json is:
{
"TemplateSnippets": [{
"SnippetName": "RdsExtensionEB",
"Order": 10000,
"SourceUrl":
"https://s3.amazonaws.com/elasticbeanstalk-env-resources-us-west-2/eb_snippets/rds/rds.json"
}]
}
(it appears you must select a snippet specific to your EB region).
and option-settings.json contains RDS-related settings similar to ones listed in the question (DBEngine, DBInstanceClass, DBAllocatedStorage, DBPassword).
It works great. I hope the AWS CLI team allows us to use this feature in the official tool in the future. I'm guessing it's not a trivial change or they would have done it already, but it's a pretty major omission functionality-wise from the Elastic Beanstalk API and AWS CLI tool, so hopefully they take a crack at it.