I've spend about a full day trying to solve this, but have no luck so far. I'm also open to alternative suggestions than my current setup.
I have an RDS instance inside of a VPC. I am trying to make CodeBuild be able to access this RDS instance for a testing step.
Currently, I setup a VPC endpoint for the CodeBuild service, with all 3 subnets of the VPC. I know that if I allow all inbound traffic for the security group on the RDS, it works. I don't want to allow all inbound traffic though- and given this, have been unsuccessful.
I have tried the following to no avail:
Taking the private IPv4's of the ENI's created by the VPCE, adding them as inbound rules to the security group on the RDS
Creating a separate VPC for CodeBuild, and setup VPC peering (this seemed overly complex, and I'm not sure if the peering would even allow CodeBuild traffic to hit an RDS; it also makes things complicated down the road for CodeDeploy).
Putting CodeBuild inside the VPC of the RDS instance. When doing this, I created a new subnet in the VPC, assigned it to a NAT in the routes table (and this NAT was on the VPC of the RDS instance); put CodeBuild kept telling me it had no internet access.
setup a VPC endpoint for the CodeBuild service,
VPC endpoints are not used for inbound traffic from CB to VPC. They are used for your applications in VPC to interact with CB service without the internet.
Putting CodeBuild inside the VPC of the RDS instance.
This is the correct way. Sadly you haven't provided any details of your VPC, subents, NAT, route tables, security groups, NACLs setup, thus its difficult to speculate why it does not work.
Thanks Marcin for pointing me in the right direction to make CodeBuild in the same VPC. When I was able to focus on that, I saw this post again:
CodeBuild cannot find the 0.0.0.0/0 destination for the target internet gateway
which I had the same issue; my NAT was also on the private subnet. Now, it's on the public subnet, and it's working.
Related
This question has been asked several times, and I've been reading the answer for the past hours, so I'm going to summarise everything here.
To get the connection between RDS and lambda, the normal steps are:
Attach the lambda to the same VPC as the database
Create a security group for the lambda
Modify / add a security group for the RDS, in which you allow the VPC's security group as inbound providing the connection port.
Add policy AWSLambdaVPCAccessExecutionRole to lambda function
That's supposed to be it, however it is not working for me.
I have checked other possible errors:
DNS of database not found. Solved by setting an internet gateway for the VPC, checked that the DNS is properly resolved
Inbound and Outbound rules of the lambda functions. Solved by setting literally all traffic for both directions.
Outbound rule for DB's security group is set to all traffic and destinations
Note: I can connect to the database via a VPN using another security group, meaning it's not a credential error.
I really don't understand what else I'm missing at this point
Finally found the answer, the error was due to the fact that the RDS was on a public subnet which had an internet gateway. In this case, the solution is as follows:
Create a public NAT gateway (attached to a public subnet, with an elastic ip)
Create one (or more) private subnets in the same VPC
Create a route with the “Destination” as “0.0.0.0”, “Target” as the Internet Gateway, and save your settings. Associate this route with the public subnets
Create a route in this table with the “Destination” as “0.0.0.0”, “Target” as the NAT Gateway created previously, and save your settings. Associate this route with the private subnets
Finally, connect the lambda to the private subnet(s) only
I am trying to use CodeDeploy to deploy my revisions on private subnet instances using VPC endpoint.
VPC endpoint is having required subnet configured and security group applied on VPC endpoint is having all inbound and outbound allowed. Still my deployment is failing and I do not have any way to check logs on EC2 instance since it is private. Any help or guidance regarding this?
I am trying to follow the below link:
https://aws.amazon.com/about-aws/whats-new/2020/08/aws-codedeploy-now-supports-deployments-to-virtual-private-cloud-endpoints/
I have read most of the Stackoverflow questions and heaps of docs on the internet but for some reason can't get lambda to connect to Redshift when Redshift is in a different VPC and different AWS account.
I have two subnets associated with the VPC and an internet gateway and NAT gateway are attached to the VPC. This is the VPC that the Lambda function is associated with. I added the NAT elastic ip to the security group of Redshift security group. The NAT is in the route table pointing to Redshift Elastic ip.
I following methods do work:
taking lambda out of the VPC and opening Redshift to public (0.0.0.0) which is not ideal
closing Redshift to the public and making the query from an EC2 instance in the VPC where the Lambda function is (put EC2 ip in Redshift security group)
Any idea how to get lambda to use the elastic ip of the NAT or something along those lines? Does the NAT in the route table need to point to elastic ip of Redshift or to a CIDR block or something similar? What am I missing?
As you said opening your Redshift to internet is really not the best answer, you should really consider using it only in an internet way.
So you will find below what I recommend to you:
- Bind you Lambda VPC to Redshift VPC using a VPC Peering (https://docs.aws.amazon.com/vpc/latest/peering/what-is-vpc-peering.html). Be careful to follow all steps for route table.
- Once your VPC peering has been made, you will be able to declare directly you Lambda Security Group to your Redshift Security group to allow inbound access.
You will then be able to remove the internet-facing of your Redshift Cluster and only use it with an internal way.
Then if you still want to use your way be careful of:
- Be careful to deploy your Lambda into your private subnets (to be able to use NAT Gateway EIP) and be sure to have all your routes configured (https://docs.aws.amazon.com/vpc/latest/userguide/vpc-nat-gateway.html). If you deploy your lambda into your pubic subnets, it's gonna be harder to configure your Redshift Security group.
- Then allow your Lambda VPC EIP to Redshift security group (inbound rule)
I figured it out using the following gist: https://gist.github.com/reggi/dc5f2620b7b4f515e68e46255ac042a7
Essentially, what I was doing wrong was using one route table. I also was pointing the NAT to the elastic ip of Redshift.
All I had to do was delete the NAT from the original route table and create a new additional route table. I then add the NAT into the new route table and associate that route table with the other subnet that is in the VPC. I point the NAT to 0.0.0.0 instead of the Redshift elastic ip.
This allowed me to get rid of the "open to public" issue in the Redshift security group. I also believe the NAT helps the Lambda stay secure from public connections but would need clarification on that.
I have an AWS lambda function that makes a request to the internet. When it makes the request with NO VPC, it's ok, but when I add it to the VPC, it stops working. I've attached an Internet Gateway to the VPC and created a NAT Gateway with RT to use outbound 0.0.0.0/0, but it stills not working.
With the 15seconds timeout, it's always throwing TO.
Could you please help me?
I've already followed these sites:
https://gist.github.com/reggi/dc5f2620b7b4f515e68e46255ac042a7
http://derpturkey.com/lambda-vpc-and-internet-access-configuration/
Everything seems to be well configured.
Lambda configuration
Route Table configuration
IGW attached to VPC
There are three ways to grant Internet access to an AWS Lambda function:
Do not assign it a VPC, or
Assign it to a VPC and attach an Elastic IP Address to the ENI (Elastic Network Interface) that is created in the VPC, or
Assign it to a private subnet in a VPC and use a NAT Gateway to grant the private subnet access to the Internet
If you have already verified the network configurtions like VPC setup, Subnets, Route Table, IG/NAG gate way. And associated the Lambda with current subnet and security groups.
Then the last thing to verify would be:
For Lambda functions to work into custom or User Defined VPC, Adding Lambda to the VPC would require AWSLambdaVPCAccessExecutionRole in addition to AWSLambdaBasicExecutionRole.
Ensure that the associated role has the above permissions.
Source:
https://docs.aws.amazon.com/lambda/latest/dg/vpc-rds-create-iam-role.html
I've created a new environment for my beanstalk app under a vpc.
I created a vpc with private and public subnets.
I've configured the security groups as outlined here:
http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/AWSHowTo-vpc-basic.html
The environment seemed to setup correctly but the status is red.
Clicking on monitor reveals no data for latency or requests.
The website url returns error code 503.
Not sure what to do
This is almost certainly an issue with not being able to communicate with the NAT. The documentation is a little ambiguous when it comes to configuring your .ebextensions config files.
Don't mess with the Network ACLs for the VPC. The infrastructure needs to be able to talk to itself.
The NAT instance & Beanstalk ELB need to be in the Public subnet (10.0.0.0/24)
The Beanstalk instances themselves need to be in the Private subnet (10.0.1.0/24)
The Security Group that you apply to the instances needs to also apply to the NAT. That Security Group also needs to be on its own Inbound list so that everything inside that SG can talk to each other.
Using curl, try to make outbound calls. First on the Bastion Host, then on the NAT, then on a Beanstalk instance. Are any of them unable to make outbound calls?
I just recently spent a couple of weeks trying to develop a CloudFormation template that handles this stuff. I think that most of it is still in my brain. I think.
If these troubleshooting steps don't help, let me know and I'll dig a little deeper.
When you created the VPC, I assume you did it using the wizard. In this situation, the NAT is configured to accept calls from "default" security group.
When you created the Elastic beanstalk environment, did you specify that you want the new instances to be on "default" security group?
Check if the new instance belongs to the VPC's default security group in EC2 management area. If it is not a member of default Security group, add this security group to your NAT's inbound connections.
There is a good AWS CloudFormation template example to create VPC with public and private subnets and the NATs:
https://github.com/awsdocs/elastic-beanstalk-samples/blob/9720e38e9da155752dce132a31d8e13a27364b83/cfn-templates/vpc-privatepublic.yaml
May be very useful for those having issues with such VPC configuration.
To deploy it to your AWS using cli:
aws cloudformation deploy --stack-name mystackname --template-file path/to/vpc-privatepublic.yaml
Or you can deploy from CloudFormation UI dashboard.