I have one VPC with an RDS instance in it. They are both located in the same region.
I want to use the RDS instance in another VPC, that is in another region on another AWS account (we have multiple AWS accounts). If that's not complicated enough the 2nd VPC comes up via CloudFormation (i.e. dynamic). Whenever I am bringing up a CloudFormation stack I want to attach the RDS instance automatically.
I have looked at:
exposing the RDS instance on the public internet :(
an ELB w/ TCP transport to put the database instance behind
VPC peering but the different regions and the approval workflow in the AWS console make little sense in the case we are using CloudFormation
All of these seem suboptimal to me and was wondering if somebody already did this before. If yes, please share what you did and what the though process behind it was.
Use a VPN tunnel from one VPC to the other. You could build your own or look at Vyatta. Ideally the two VPCs do not have overlapping CIDRs. Note that you cannot use VPC peering inter-region.
For anyone who stumbles around here, it looks like AWS VPC Peering can now be done cross region: https://docs.aws.amazon.com/vpc/latest/peering/what-is-vpc-peering.html
Related
I'm kind of confused on the concept of VPC.
I understand that when I create a EC2 instance(s), it automatically resides in a VPC, as EC2 is an virtual server instance. Actually, I understand how VPC settings work (I read all the official tutorial documents for VPC)!
But what about other services or resources? S3, Lambda, RDS, ... Don't they have to be in a VPC either?
Probably, I don't clearly understand what are VPCs for. Or, I have confusions about other managed services AWS provides. I want to get clear about what can be in VPC and what can't be.
Currently, building a serverless app that use DynamoDb and Elasticsearch is quite easy
Using serverless, you just declare everything in serverless.yml and you are good to go
Problems (quickly) arise when you need to use RDS or Elasticache because you have all kind of troubles with VPCs...which then simply defeats the serverless paradigm (developper should only focus on code)
The quickest solution is then to use a 3rd party solution (like RedisLabs or ClearDb)
My question is : why RDS and Elasticache require the VPC mode ? why aren't they usable directly like a 3rd party service ?
EDIT : as noted in the comments, you can place DynamoDB and Elasticsearch behind a VPC .
The problem then becomes : how to efficiently access them (RDS, elasticache, dynamodb, elasticsearch) from a lambda function ?
You need to configure the VPC of the lambda function to access all the other VPCs as described in https://docs.aws.amazon.com/lambda/latest/dg/vpc.html
Also, consider that the lamba also needs to access 3rd party services on the internet (ex : sendgrid, onesignal, ...) so I think that you still need a NAT somewhere
The distinction here is where the resources are actually running. Both Elasticsearch and DynamoDB are managed services running outside of your AWS account. RDS and Elasticache are different - they are launched into your AWS account, hence the need to tell AWS where you want to run them.
By the way, RDS doesn't require VPC. You can optionally run it in EC2-Classic or in EC2-VPC. And those are the only options to run compute on AWS (either in VPC, or not in VPC), so you are not actually being constrained here. You are simply being asked which you prefer.
The solution for access to private resources in your VPC (like RDS databases) is to configure the Lambda function to run in that VPC. Now the Lambda function is essentially inside the VPC, so it is constrained by the VPC's networking configuration. For the Lambda function to reach external websites, it needs a route to the public internet. Typically the way you do this in VPC is to configure an IGW and some form of NAT (roll-your-own, or managed NAT from AWS). This is all normal VPC behavior, and not specific to Lambda.
The best article I found on the subject : http://blog.brianz.bz/post/accessing-vpc-resources-with-lambda/
It explains nearly everything about VPC access from lambda (from what is a VPC, why you need it, how to access from lamdba and to configure from serverless)
I have two VPCs:
VPC1 which holds our RDS instance.
VPC2 which holds our cluster of EC2 instances.
We have successfully setup a VPC peering connection, routes and security groups to allow appropriate communication.
In order to resolve the RDS instance AZ-appropriate local IP address from it's hostname, we need to follow these instructions and set --requester-peering-connection-options AllowDnsResolutionFromRemoteVpc=true.
If I do this manually through the AWS Console or the AWS CLI it all works fine, however I'm creating the cluster of EC2 instances through CloudFormation and the option is missing from the CloudFormation documentation.
The effect of this is that my stack starts up and fails because the services themselves cannot connect to the database.
Am I doing something obvious wrong, or is this just Amazon being incomplete?
Thanks!
Due to the frequency of updates, there are many times where an AWS feature isn't available in CloudFormation (ALB targeting Lambda used to be) - you end up having to create a custom resource to manage it. It's not too bad, just make sure that your lambda responds with success or failure in all scenarios, including exceptions, otherwise your stack will be 'in progress' for hours.
I am faced with a chicken and egg problem. I currently have a server in EC2 classic, as well as an RDS instance -- in EC2 classic as well. The EC2 instances also interact with Cassandra cluster, which also resides in EC2 classic.
However, I need to move RDS into the VPC. Now, in an ideal world, I'd have all of my stuff in VPC at this point. However, that presents a major migration challenge and I'd like to minimize impact on users and keep steps to minimum -- this is mainly because of the Cassandra cluster.
It turns out that I cannot create security group rules between VPC and Non-VPC security groups.
So, how can I have RDS in VPC that my EC2 instances can access w/o having to open up my RDS to the entire world ?
Any help is greatly appreciated.
UPDATE: So, one idea I had is to assign elastic IPs to my EC2 instances and add IPs explicitly to the security group for RDS within VPC. Would that work ? (trying it now using https://github.com/skymill/aws-ec2-assign-elastic-ip)
Yes, unfortunately that's the only way to do it. You cannot use DNS in security groups, so you're stuck with IP address.
So, I ended up solving it exactly like I described -- assign elastic IPs to my EC2 instances and add IPs explicitly to the security group for RDS within VPC. It ended up working great.
We have couple of RDS that are not added under VPC, so we need to bring them under VPC. Please let me know the steps and downtime expected. Also let me know if there need to be any changes in the webservers, so that everything works fine after RDS is under VPC.
You must have a VPC created before hand that have subnet in atleast two different regions.
After this go and create a "subnet group" for RDS and add two existing subnet in that.
Next take a snapshot of your RDS instance and start a new RDS instance from snapshot in VPC.
That should be it.
http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_VPC.html#USER_VPC.Non-VPC2VPC - official documentation from Amazon.
Depending on your configuration and if the RDS needs to be accessible from the Internet you will have to check the option "Publicly Accessible" in the creation wizard and (in addition to the subnets mentioned in other answers) ensure that the Security has port for the DB properly enabled (and maybe from 0.0.0.0/0).