When to use aws direct connect and aws storage gateway. My question is these services seems to be similer, so what are use cases to use these two services.
AWS Direct Connect is a network connection between AWS and on on-premises network. The physical connection is an optical fiber link organised through a Telco, while Direct Connect provisions the physical port where the fiber connects in an AWS transit center.
AWS Storage Gateway is a storage service that provisions a virtual tape drive, virtual S3 drive or virtual disk that is stored in AWS. It typically runs across a Direct Connect connection.
AWS direct connect connect the in premisses resources with any services, while AWS storage gateway used to connects to S3 services including AWS S3 Glacier only.
This is one of the difference.
"AWS Direct Connect is a network service that provides an alternative to using the Internet to connect customer's on-premise sites to AWS" (AWS Docs).
"AWS Storage Gateway is a hybrid cloud storage service that gives you on-premises access to virtually unlimited cloud storage" (AWS Docs).
Direct Connect creates a private network connection btwn AWS and on-prem resources while Storage Gateway enables you to store and retrieve Amazon S3 objects through standard file storage protocol.
Storage Gateway - As the name suggests, this service is used to connect on-premises infra with STORAGE services (specifically S3, FSx, EBS)
Direct Connect - This service is used to connect on-premises infra with any AWS resources (in any region)
Related
I have a Python application deployed on EKS (Elastic Kubernetes Service). This application saves large files inside an S3 bucket using the AWS SDK for Python (boto3). Both the EKS cluster and the S3 bucket are in the same region.
My question is, how is communication between the two services (EKS and S3) handled by default?
Do both services communicate directly and internally through the Amazon network, or do they communicate externally via the Internet?
If they communicate via the internet, is there a step by step guide on how to establish a direct internal connection between both services?
how is communication between the two services (EKS and S3) handled by default?
By default the network topology of your EKS offers route to the public AWS S3 endpoints.
Do both services communicate directly and internally through the Amazon network, or do they communicate externally via the Internet?
Your cluster needs to have network access to the said public AWS S3 endpoints. Example, worker nodes running in public subnet or the use of NAT gateway in private subnet.
...is there a step by step guide on how to establish a direct internal connection between both services?
You create VPC endpoints for S3 in the VPC that your EKS runs to ensure network communication with S3 stay within AWS network. VPC endpoints for S3 support both interface and gateway type. Try this article to learn about the basic of S3 endpoints, you can use the same method to create endpoints in the VPC where your EKS runs. Request to S3 from your pods will then use the endpoint to reach out to S3 within AWS network.
You can add S3 access to your EKS node IAM role, this link shows you how to add ECR registry access to EKS node IAM role, but it is the same for S3.
The other way is to make environment variables available in your container, see this link, though I would recommend the first way.
I'm working with AWS and need some support please.
My team provisioned Direct Connect and we can now enjoy private connectivity from our corporate network to VPC on AWS.
Management is asking if it's possible that aws cli commands are executed through Direct Connect and not through the public internet. Indeed, we have a lot of scripts with a lot of commands like aws ec2 describe-instances and so on. I guess these calls the public REST API of EC2 service that AWS exposes.
They're asking if it's possible that these calls do not go through the public internet.
I've seen VPC endpoints? Are they the solution?
See How can I access my Amazon S3 bucket over Direct Connect? for how to do this with S3.
Basically:
After BGP is up and established, the Direct Connect router advertises all global public IP prefixes, including Amazon S3 prefixes. Traffic heading to Amazon S3 is routed through the Direct Connect public virtual interface. The public virtual interface is routed through a private network connection between AWS and your data center or corporate network.
You can extend this to other Amazon services, per the AWS Direct Connect FAQs:
All AWS services, including Amazon Elastic Compute Cloud (EC2), Amazon Virtual Private Cloud (VPC), Amazon Simple Storage Service (S3), and Amazon DynamoDB can be used with Direct Connect.
Refer to #jarmod's answer below for the answer to the question but read on for why I think this sounds like an XY problem.
There is no reason at all why management should be concerned.
Third-party auditors assess the security and compliance of AWS services as part of multiple AWS compliance programs. Using the AWS CLI to access a service does not alter that service's compliance - AWS has compliance programs which pretty much cover every IT compliance framework out there globally.
Compliance aside, the AWS CLI does not store any customer data (there should be no data protection concerns) & transmits data securely (unless you manually override this).
The user guide highlights this:
The AWS CLI does not itself store any customer data other than the credentials it needs to interact with the AWS services on the user's behalf.
By default, all data transmitted from the client computer running the AWS CLI and AWS service endpoints is encrypted by sending everything through a HTTPS/TLS connection.
You don't need to do anything to enable the use of HTTPS/TLS. It is always enabled unless you explicitly disable it for an individual command by using the --no-verify-ssl command line option.
As if that's not enough, you can also add increased security when communicating with AWS services by enforcing a minimum version of TLS 1.2 to be used by the CLI.
There should be targeting of much much bigger attack vectors, like:
The physical accessibility of the device storing the credentials
Permanent access tokens vs. temporary credentials
IAM policies associated with the credentials
The AWS CLI is secure.
I'm trying to lower egress fees from multiple S3 buckets in one AWS account (several Terabytes per month) to our US datacenter.
I thought of setting up a VPC on our AWS account, and using a Gateway Endpoint for S3 and then having a Direct Connect always active from our datacenter to our AWS VPC (and pay the usual hourly + GB transferred reduced fees).
After reading the documentation it is my understanding that I will not pay any traffic to use the Gateway Endpoint for S3 since traffic never leaves AWS until it reaches our AWS VPC. To transfer it to our datacenter the usual hourly + GB transferred reduced fees for the Direct Connect is then billed.
Is this correct? Will we still be able to initiate a GET from our datacenter applications (through the Direct Connect to this VPC Gateway Endpoint for S3 so that we can pull S3 files from the 3rd party AWS account that endpoint is linked to? (requests would only originate from our datacenter servers to get or sometimes put S3 files).
You will not be able to route to the gateway endpoint via direct connect. However, you can configure a public VIF on your direct connect to route all traffic to the Amazon IP spaces via the direct connect.
From: Which type of virtual interface should I use to connect different resources in AWS?
To connect to AWS resources that are reachable by a public IP address
(such as an Amazon Simple Storage Service bucket) or AWS public
endpoints, use a public virtual interface.
Alternatively you can use the recently released interface endpoint for S3 (AWS Announcement - Amazon S3 now supports AWS PrivateLink), to get this working you need to configure your application to use this endpoint. However this can maybe against your needs, because you will be charged a small fee for each GB processed by the interface endpoint. Therefore the AWS Direct Connect public VIF should be the right choice for you.
My organization has an AWS presence, but no VPN nor Direct Connect to and from our on-premises data center. We would still like to leverage DynamoDB in the short-term without having DirectConnect or a VPN connection in place. We will not be using any EC2 instances for our web services. Is it possible for an on--prem host to talk to DynamoDB without any AWS networking infrastructure in place....basically a call direct to the DynamoDB service without VPN or Direct Connect?
All you need is an Internet connection to access DynamoDB. Your on-premis servers will need to have access to make calls to the AWS API, which is publicly accessible over the Internet.
You can use an VPC endpoint gateway to connect your server to Dynamo Db using amazon network
https://docs.aws.amazon.com/it_it/vpc/latest/privatelink/vpc-endpoints.html
If I am downloading an S3 object from an EC2 instance, does this request leave the Amazon network, or is the request made via Internet?
It depends whether the EC2 instance and S3 bucket are in the same region or not.
All communication between regions is across the public Internet.
You can read more about AWS regions and availability zones here. Communication within the same region happens over low-latency private links:
Availability Zones are connected to each other with fast, private
fiber-optic networking.
See AWS Global Infrastructure.
EDIT
Although data transfer happens over private links within the same region, accessing the API endpoints using the SDK or CLI still requires Internet access. See AWS Regions and Endpoints.
If you're concerned about security in Java SDK, the default client configuration is to use HTTPS for all requests for increased security. (Although individual clients can also override this setting by explicitly including the protocol as part of the endpoint URL when calling AmazonWebServiceClient.setEndpoint(String))
If you're concerned about data transfer cost, all inbound traffic from S3 to EC2 is free of charge.