I am trying to build a graphQL API with Serverless Framework on AWS-Lambda using Apollo-Server-Lambda etc. I need to use a not publicly available PostgreSQL RDS instance.
I can get lambdas up and running and sending back requests when not in VPC. I can get a Postgres RDS Database up and running and connected to PgAdmin (when in publicly available mode).
The problem is once I make the RDS Postgres 'non public' and try to get all these pieces talking together I fail.
I have tried multitude of different approaches.
This is regularly portrayed as magic. It is wonderfully written https://gist.github.com/reggi/dc5f2620b7b4f515e68e46255ac042a7
I could not get access to Postgres with my lambdas using this. So my first question.
Do I need a NAT gateway for incoming (ingress) api calls to lambdas in a VPC?
My current understanding is that maybe I only need a NAT gateway for my lambdas to make outgoing calls to other api's out of aws or things like S3. Is this correct?
Next up. I have made a security group for my lambdas and have added this security group to the inbound list for the security group that was created for RDS. My understanding is this is how the lambdas should gain access to RDS. I have not had such luck. Maybe this is related to public or non public subnets? Maybe this is related to my understanding of the necessity of a NAT?
Basically the only visibility I have been able to get is Lambdas timing out after 20 or 30 seconds depending on my limit when they are trying to connect to postgres in private. Cloudwatch logs reveal nothing else.
Lastly, for now 😂, what is the best way to connect my dev machine to Postgres once it is 'not public'? I have my machines IP listed for inbound TCP/IP to port 5432 to postgres in the RDS security group but that does not seem to give me the access I was hoping for. Do I really need a VPN connected to VPC? Whats the simplest way?
I have done this tutorial with basic alterations for Postgres https://docs.aws.amazon.com/lambda/latest/dg/vpc-rds.html
I have read and considered answers from this question & more
Allow AWS Lambda to access RDS Database
I have had many success-full deployments with serverless framework with many variations on serverless.yml config to try these options or else I would show a specific one I thought was failing but this is more broadly that I cant seem to grasp exactly how all these VPC, security groups, routing tables etc are supposed to interact.
Any help greatly appreciated!
Obviously, Lambda needs to be setup to run inside the same VPC, but I'm assuming you already got that.
You need to:
Create a security group (SG) and associate it with the Lambda function.
Now, open the SG associated with the RDS instance (not the one you created above).
Inside the RDS SG, go to "Inbound" tab and click "Edit"
Select "PostgreSQL" in the Type column. In the Source column, select "Custom" in the select dropdown and enter the Lambda SG ID in the input text (if you start typing "sg-", it will show you all your SGs).
Does it work?
Make sure your Lambda function is in the VPC, and the security group allows connections from IP addresses within the subnet of the VPC. The amount of available IP addresses is going to affect how many lambda functions can be run concurrently. Also make sure that the Lambda function's role has the ability to describe the VPC (the AWSLambdaVPCAccessExecutionRole policy should do the job for you).
Related
Is there an alternative to AWS's security groups in the Google Cloud Platform?
Following is the situation which I have:
A Basic Node.js server running in Cloud Run as a docker image.
A Postgres SQL database at GCP.
A Redis instance at GCP.
What I want to do is make a 'security group' sort of so that my Postgres SQL DB and Redis instance can only be accessed from my Node.js server and nowhere else. I don't want them to be publically accessible via an IP.
What we do in AWS is, that only services part of a security group can access each other.
I'm not very sure but I guess in GCP I need to make use of Firewall rules (not sure at all).
If I'm correct could someone please guide me as to how to go about this? And if I'm wrong could someone suggest the correct method?
GCP has firewall rules for its VPC that work similar to AWS Security Groups. More details can be found here. You can place your PostgreSQL database, Redis instance and Node.js server inside GCP VPC.
Make Node.js server available to the public via DNS.
Set default-allow-internal rule, so that only the services present in VPC can access each other (halting public access of DB and Redis)
As an alternative approach, you may also keep all three servers public and only allow Node.js IP address to access DB and Redis servers, but the above solution is recommended.
Security groups inside AWS are instance-attached firewall-like components. So for example, you can have a SG on an instance level, similar to configuring IP-tables on regular Linux.
On the other hand, Google Firewall rules are more on a Network level. I guess, for the level of "granularity", I'd say that Security Groups can be replaced to instance-level granularity, so then your alternatives are to use one of the following:
firewalld
nftables
iptables
The thing is that in AWS you can also attach security groups to subnets. So SG's when attached to subnets, are also kind of similar to google firewalls, still, security groups provide a bit more granularity since you can have different security groups per subnet, while in GCP you need to have a firewall per Network. At this level, protection should come from firewalls in subnets.
Thanks #amsh for the solution to the problem. But there were a few more things that were required to be done so I guess it'll be better if I list them out here if anyone needs in the future:
Create a VPC network and add a subnet for a particular region (Eg: us-central1).
Create a VPC connector from the Serverless VPC Access section for the created VPC network in the same region.
In Cloud Run add the created VPC connector in the Connection section.
Create the PostgreSQL and Redis instance in the same region as that of the created VPC network.
In the Private IP section of these instances, select the created VPC network. This will create a Private IP for the respective instances in the region of the created VPC network.
Use this Private IP in the Node.js server to connect to the instance and it'll be good to go.
Common Problems you might face:
Error while creating the VPC Connector: Ensure the IP range of the VPC connector and the VPC network do not overlap.
Different regions: Ensure all instances are in the same region of the VPC network, else they won't connect via the Private IP.
Avoid changing the firewall rules: The firewall rules must not be changed unless you need them to perform differently than they normally do.
Instances in different regions: If the instances are spread across different regions, use VPC network peering to establish a connection between them.
I am running a DB cluster with two instances at Amazon RDS Aurora. One instance is the master, the other instance is a read-only replica. The purpose of the replica is to allow a third party application access to certain tables of the database for reporting. Therefore, the reporting tool accesses the read-only cluster endpoint, which works perfectly fine. In order to achieve zero-downtime maintenance, AWS promotes the "replica" to be the "master" at any time. That's pretty cool and does not affect the reporting tool, because it accesses the cluster-ro endpoint, which always routes the traffic to the correct (read-only) replica.
However, this means I have to enable the "Publicly accessible: Yes" flag on both instances, so that the reporting tool (which is located outside the VPC) has access to all instances, because I can not predict which instance becomes the master or replica, correct?
I'd prefer, that the "master" instance (whatever instance that is) can only be accessed from inside the VPC. How can I achieve that?
My understand is that every change I do on the "master" instance, is automatically done on the replica(s), including adding/removing security groups for example. So if I open the firewall to allow access to the replica(s) for the reporting tool, the same IP addresses can also access the normal cluster endpoint and instance (not only the cluster-ro endpoint). How can I prevent that?
You will need to build something custom for this unfortunately. Few options to consider from a design point of view are as follows:
Aurora cluster shares the security group settings across all instance like you called out. If you want to have custom settings, then what you could consider is making your whole cluster VPC only, and then have either ALBs or EC2 proxies that forward requests to your DB instance(s). You can then have multiple of these "proxies" and associate separate security groups for each of them.
One big callout with this sort of an architecture is that you need to make sure that you take care of failovers cleanly. Your proxies should always talk to the cluster endpoints and never to the instance endpoints, as instances can change from READER to WRITER behind the scenes. For example, ALBs do no let you create listeners that forward requests to a DNS, they only work with IPs. This means that you would need additional infrastructure to monitor the IPs of readers and writers and keep the ALB updated.
EC2 proxies are definitely a better option for such a design, with the caveat of added cost. I can go into more details if you have specific questions around this setup. This is definitely a summary of the approach, and not prod ready.
On the same note, why can't you use read restricted db users instead and keep the cluster public (with ssl enabled of course)?
I have two VPCs:
VPC1 which holds our RDS instance.
VPC2 which holds our cluster of EC2 instances.
We have successfully setup a VPC peering connection, routes and security groups to allow appropriate communication.
In order to resolve the RDS instance AZ-appropriate local IP address from it's hostname, we need to follow these instructions and set --requester-peering-connection-options AllowDnsResolutionFromRemoteVpc=true.
If I do this manually through the AWS Console or the AWS CLI it all works fine, however I'm creating the cluster of EC2 instances through CloudFormation and the option is missing from the CloudFormation documentation.
The effect of this is that my stack starts up and fails because the services themselves cannot connect to the database.
Am I doing something obvious wrong, or is this just Amazon being incomplete?
Thanks!
Due to the frequency of updates, there are many times where an AWS feature isn't available in CloudFormation (ALB targeting Lambda used to be) - you end up having to create a custom resource to manage it. It's not too bad, just make sure that your lambda responds with success or failure in all scenarios, including exceptions, otherwise your stack will be 'in progress' for hours.
I have created a free-tier PostgreSQL RDS instance and everything appears to looks good on the portal. However, I am unable to get to the instance.
Going through the troubleshooting steps, they mention it could a firewall issue on my end. However, a quick ping from an external site reveals the same timeout issue.
Is there a step that I've missed?
Ping is typically disabled in AWS Security Groups. It is not recommended as a method of checking network connections.
The best method would be to use an SQL client to connect to the database via JDBC or ODBC.
Things to check:
Your RDS instance was launched as Publicly Accessible
Your RDS instance was launched in a Public Subnet (Definition: The subnet's Route Table points to an Internet Gateway)
The Security Group permits connections on the database port (this is also where you could permit PING access, but no guarantee that it would work with an RDS instance)
Check the associated Security Groups that you have tagged. Security groups hold the firewall rules. Either you may have to tweak the group that you have selected or try changing / modifying the group that you (or your profile) have access to.
I am trying to access a Redshift database from a Lambda function. When I add 0.0.0.0/0 to the security group connections in the Redshift interface (as suggested by this article), I am able to connect successfully. From a security perspective, however, I don't feel comfortable using 0.0.0.0/0.
Is there a way to only allow Lambda to access Redshift without opening it up to the public internet?
I have tried adding the AWS IP ranges, however, this didn't work (as it only allows a limited number of security group connections).
Enable VPC access for your Lambda function, and open the Redshift Security Group to the SG assigned to the Lambda function, just like you would open it to an EC2 server.
I was having the same problem and the answer wasn't helping. It would work with I added a 0.0.0.0/0 to my security group, but that is not something I could work with long term.
I ended up creating a new VPC with a public (10.0.1.0) and private(10.0.2.0) subnet and a NAT on the public subnet. Running the lambda function on the private subnet, but still not able to access the redshift db. Moved the Redshift db to the public (10.0.1.0) subnet. I put Redshift on that subnet because the load script runs externally and there is a security group that lets through my two work/home IPs
on port 5439 and it also lets in the private 10.0.2.0/24 subnet where the lambda
script is running.
That said it still wasn't working and it took a while to figure out that I couldn't use the full dns name to access Redshift. Even with a working NAT in place it still wasn't connecting to redshift. I tested this with two EC2 instances and a simple script to connect and run a query. This was faster than working in lambda just to solve the connectivity. This link on managing clusters and this one on cluster node ip solved the issue for me.
Solution: Go to the cluster and click on it. Down in the lower right corner of all the config information are the SSH ingestion settings and the private IP. I used that IP instead of the url an it solved all my connectivity issues. Seems basic now, but spent a lot of time searching and could not find the answer I was looking for.
Good Luck,
Wood