I have a private subnet inside a VPC, that cannot route to the internet. I'm trying to access amazon ECR, but getting a timeout. My guess is that ECR requires internet connection, however I cannot find any documentation that says that.
Does ECR require internet connection? Is there a way to use it from within a private subnet?
Update 2020
Interface VPC Endpoints are now supported for ECR; meaning now we can configure an endpoint from our private subnet to ECR without a NAT Gateway and still be able to pull images from it.
Documentation: Amazon ECS interface VPC endpoints (AWS PrivateLink)
A private subnet is truly private and only in/out traffic that you specify will be allowed. S3 has VPC Endpoints that allow you to connect to S3 (http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/vpc-endpoints.html) without routing through the public internet. VPC endpoint functionality for AWS ECR has been requested (https://forums.aws.amazon.com/thread.jspa?threadID=222124) but to the best of my knowledge it is not yet currently available.
An VPC endpoint for ECR is not available, but requested as the first issue on AWS' container roadmap (created 2018-11-28), implemented as a PrivateLink.
It's in state "Coming soon".
It will cost minimum around 22$/month (PrivateLink costs for 3 availability zones in us-east, without traffic costs), if they don't state it otherwise.
Related
I have an issue regarding to AWS VPC networking, I want to access external API from my ECS task, I've configured ECS in a Private subnet and the only way to access this ECS is trough an load balancer, in some services like Lambda is working (I can access external resources) but in ECS I can't access to them, I tried modifying the security group rules + modify ACL rules but isn't working, If anyone know how I can do, I be very grateful, thanks.
ps: I created the VPC on the UI that AWS has.
For resources in a private subnet to access the Internet, the only option is to send that traffic through a NAT Gateway.
You will have to create an AWS NAT Gateway in at least one of the public subnets of your VPC, and then add a route in each of the private subnets to that NAT Gateway.
I am trying to understand the concept of how VPC endpoints work and I am not sure that I understand the AWS documentation. For example, I have a private S3 bucket and I have an EKS cluster. So if my bucket is private I believe that traffic from the EKS cluster to S3 does not go through the internet, but only through the AWS network. But in a case my s3 bucket was public, then probably I will need to set up the VPC endpoint, so traffic will not leave the AWS. The same logic I would expect with ECR, if it is private you load images to your EKS through AWS network.
So what is the exact case when you need to use VPC endpoint within your AWS account (not from on-prem or another VPC)?
VPC endpoints are typically used with public AWS services (such as S3, DynamoDB, ECR, etc.) when the client applications are hosted inside your VPC and you do not want to route traffic via public Internet, which would otherwise result in a number of hops to reach the AWS service.
Imagine a situation when you have an app running on an EC2 instance, which is deployed to a private subnet of your VPC (i.e. a Pod in your EKS cluster). This app reads/writes data from/to AWS S3. If you do not use a VPC endpoint, your traffic will first reach your NAT gateway, then your VPC's Internet gateway out to the public Internet. Eventually, it will hit AWS S3. The response will travel back via the same route.
Same thing with ECR (i.e. a new instance of your Kubernetes Pod started by the kubelet). It's better (i.e. quicker) to pick the shortest route to download a Docker image from ECR rather than traverse a number of switches/routers. With a VPC endpoint your traffic will first hit the VPC endpoint (without leaving your private subnet) and then reach e.g. ECR directly (traffic does not leave the Amazon network).
As correctly mentioned by #jarmod, one should differentiate between routing (Layer 3 in the OSI model) and authentication/authorization (Layer 7). For example, you can use a VPC endpoint to reach AWS S3, but not be authorized (or even unauthenticated) to e.g. read a file from an S3 bucket.
Hope this clarifies the idea behind using VPC endpoints.
I have a docker image which is just an Java application. The java application reads data from DynamoDB and S3 buckets and outputs something (its a test app). I have hosted the docker images onto public docker-hub repo.
In AWS, i have created private subnet which is hosting an EC2 via AWS ECS. Now to have security high; i am using VPC Endpoints for DynamoDB and S3 bucket operations for the containers.
And i have used NAT Gateway to allow EC2 to pull docker images from docker-hub.
Problem:
When i remove VPC Endpoint, the application is able to read DynamoDB and S3 via NAT. Which means the traffic is going through public network.
Thoughts:
Can not whitelist the Ip addresses of Dockerhub as it can change.
Since AWS ECS handles all the docker pull etc tasks, i do not have control to customize.
I do not want to use AWS container registry. I prefer dockerhub.
DynamoDB/S3 private addresses are not known
Question:
How to make sure that traffic for docker hub should only be allowed via NAT?
How to make sure that the DynamoDB and S3 access should be via Endpoints only?
Thanks for your help
IF you want to restrict outbound traffic over your NAT (by DNS hostname) to DockerHub only you will need a third party solution that can allow or deny outbound traffic before it traverses the internet.
You would install this appliance in a separate subnet which has NAT Gateway access. Then in your existing subnet(s) for ECS you would update the route table to have the 0.0.0.0/0 route speak to this appliance (by specifying its ENI). If you check the AWS marketplace there may be a solution already in place to fulfil the domain filter.
Alternatively you could automate a tool that is able scrape the whitelisted IP addresses for DockerHub, and then have it add these as allow all traffic rules with a NACL. This NACL would only be applied to the subnets that the NAT Gateway resides in.
Regarding your second question, from the VPC point of view by adding the prefix list of the S3 and DynamoDB endpoints to the route table it will forward any requests that hit these API endpoints through the private route.
At this time DynamoDB does not have the ability to prevent public routed interaction, however S3 does. By adding a condition of the VPCE to its bucket policy you can deny any access that tries to interact outside of the listed VPC Endpoint. Be careful not to block yourself access from the console however, by blocking only the specific verbs that you don't want allowed.
I have deployed multiple applications(docker containers) by ECS on private subnet. NAT Gateway is used to allow internet access. Due to active deployment, the images download from ECR by nat gateway is costing too much due to their size.
I am planning on using AWS PrivateLink. But will AWS PrivateLink allow access to internet from my apps or can I use PrivateLink only for ECR image downloads from ECR and natgateway for occasional internet connectivity for my applications.
Thanks for your Valuable time.
You cannot access the internet outside of AWS services in a private instance if you're only using AWS PrivateLink.
You can download ECR images through VPC endpoints instead of a NAT gateway. The following endpoints are required:
com.amazonaws.region.ecr.dkr
com.amazonaws.region.ecr.api
com.amazonaws.region.ecr.s3
Reference: https://docs.aws.amazon.com/AmazonECR/latest/userguide/vpc-endpoints.html
I need to connect dynamoDb and elasticache from aws-lambda (otherthan using NAT Gateway).
ElastiCache provides essential caching methods along with help in making the Lambda state-ful. The concern is that for Lambda to work nice with DynamoDB it should be set to NoVPC.
If we have to use ElastiCache, Lambda and both have to be in the same VPC.TO use Both ElastiCache and DynamoDB together is quite a challenge specially with Lambda. Given the VPC challenges.Do you have any suggestions to make this easier?
A Lambda function would have to have VPC access to connect to ElastiCache, and it would have to have access to resources outside the VPC to access DynamoDB so it would require a NAT gateway. There is no way to provide access to both of those services to a single Lambda function without enabling VPC access and setting up a NAT gateway.
If you just need a Redis server and aren't required specifically to use ElasiCache, then you could use a RedisLabs instance which wouldn't require you to enable VPC access on your Lambda function.
There is now a relatively easy solution for DynamoDb access from a VPC: VPC Endpoints.
"Previously, if you wanted your EC2 (elroy: or lambda) instances in your VPC to be able to access DynamoDB, you had two options. You could use an Internet Gateway (with a NAT Gateway or assigning your instances public IPs) or you could route all of your traffic to your local infrastructure via VPN or AWS Direct Connect and then back to DynamoDB."
"A VPC endpoint for DynamoDB enables Amazon EC2 instances in your VPC to use their private IP addresses to access DynamoDB with no exposure to the public Internet...Your EC2 instances do not require public IP addresses, and you do not need an Internet gateway, a NAT device, or a virtual private gateway in your VPC. You use endpoint policies to control access to DynamoDB. Traffic between your VPC and the AWS service does not leave the Amazon network. "
The above quotes come from the links below. Note the the references to "EC2 instances" apply to lambda contexts as well.
See https://aws.amazon.com/blogs/aws/new-vpc-endpoints-for-dynamodb/
and
http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/vpc-endpoints-dynamodb.html