How does AWS Direct Connect work?
From AWS docs:
AWS Direct Connect establishes a dedicated network connection between your on-premises network and AWS ... bypassing your internet service provider.
Please explain what it actually means in practical sense? And why would it be any better?
NOTE: I know there are AWS docs (worst docs I've ever seen) and some
other articles and answers on SO, but none of those explain what it
actually means in practice or miss some important notes for understanding for somebody who's only used to public internet and can't imagine how it's possible to "bybass public internet". So I decided
to create this post from my collective knowledge and also provided a real example from our company case. Hope it will be
useful for somebody.
So, from AWS docs:
AWS Direct Connect establishes a dedicated network connection between your on-premises network and AWS ... bypassing your internet service provider.
But to understand what it actually means, lets refresh some knowledge.
Public Internet
Internet cable goes from your PC to your ISP (Internet Service Provider) switch located somewhere in your apartments building, usually.
And then... it gets connected to another switch, and it goes on and on like that, travelling through other cables and switches until it reaches the target PC you wanted to reach. Each switch knows where to send the signal next (how: it's a different topic).
So the signal first travels to some main switch of your ISP (where it does all the pricing/monitoring etc). ISP itself is then connected to another bigger ISP (they have some sort of partnership where your ISP pays that another ISP for using its cables, just like you do with your own ISP). In total, lots of physical cables literally lay down around the world going undersea/underground/over-the-air allowing the whole world to be connected to each other.
Problem
There are millions of internet users. You can imagine how many switches and cables is needed to support all that big network, and how much traffic (hello TikTok) is traveling. Because of that:
Your signal doesn't travel via the most optimal route through switches, when it needs to go from your PC to another target machine (some AWS machine in our case). Especially when you're on different continents.
Big traffic also makes your ping fluctuate, depending on the load on each individual switch.
There are all sorts of switches, we can't really trust them. Moreover, ISPs aren't required to be compliant with PCI security standard, or any other. If you want a secure connection, you have to use VPN (IPSec, OSI layer 3), but it costs in terms of performance and ping speed.
AWS Direct Connect
AWS own internet
AWS came here and said - "let me create my own internet around the world. I literally lay down my own physical cables and I'll connect them to only selected (partner) data centers. So you can use those instead of public internet" AWS may still probably lease someone's cables (especially underseas one's), but most of the cables/switches are their own ones. Benefits:
It's a much smaller net. Less switches and they're more reliable. Cables go almost straight to AWS. Therefore it's faster.
AWS implements MACsec OSI layer 2 security at hardware level. So no VPN required (though you could still use it). It's faster.
How to connect
Obviously, AWS can't connect to each PC in the world just like public ISPs, otherwise it would be the same public internet network, sort of speaking, and wouldn't make sense.
AWS has a partnership with selected data centers around the world to which AWS did the physical connection and put their own switch there ("AWS Direct Connect Cage"). So if you put your server in such data center (or at least your server connects to such data center from other nearest location via public internet) you can quickly enter AWS network where your signal will travel much faster. So you don't even need a public ISP in such case.
You do this to:
Reduce latency and improve stability into your AWS environment.
Even reduce latency between non-AWS endpoints. When both endpoints use public internet to only connect to the nearest AWS cage, while then cage-to-cage traffics goes through AWS internal network. And voila!
Results
In our company case, we managed to decrease the latency around 5 times (i.e. 0.5 seconds vs 2.5 seconds) for non-AWS connectivity.
Related
I've watched hours upon hours of tutorials and have read until my eyes were about to bleed, but I just cannot seem to grasp how Amazon VPCs are working. I've created and deleted entire VPC environments with EC2 instances various times following tutorials, but as soon as I go to create one w/out the tutorial, I'm lost.
I'm trying to come up with an analogy to help me to better understand. What I have so far is something like this:
A VPC is like a Club. At the front of the club, you have an
Entrance, the IGW. Inside the Club, you have different areas; the General Area which would be the public subnet and the
Management Area which is the private subnet.
Within the General Area you would have a Dance Floor/Bar which
would equate to an EC2 Instance and a Receiving Bay where management
can receive deliveries and whatnot from the outside world, the NAT.
Then in the Management Area you'd have an Office, another EC2
Instance, and your Inventory which is like your RDS.
I think that's a somewhat accurate analogy so far, but once I start to try and work in the SGs, NACLs, RTs, etc, I realize that I'm just not grasping it all.
Can anyone help me with finishing this analogy or supply a better analogy? I'm at my wits' end.
Rather than using analogies, let's use the network you already have at home.
Within your home, you probably have a Router and various devices connected to the router. They might be directly connected via ethernet cables (eg a PC), or they might be connected via wifi (eg tablets, phones, Alexa). Your home network is like a VPC. Your various devices connect to the network and all of the devices can talk to each other.
You also have some sort of box that connects your router to the Internet. This might be a cable modem, or a fibre router or (in the old days) a telephone connection. These boxes connect your network (VPC) to the Internet and are similar in function to an Internet Gateway. Without these boxes, your network would not be able to communicate with the Internet. Similarly, without an Internet Gateway, a VPC cannot communicate with the Internet.
Some home routers allow you to broadcast a Guest network in addition to your normal network. This is a network where you can give guests a password, but they can't access your whole network -- this is good for security, since they can't snoop around your network to try and steal your data. This is similar in concept to having a separate subnet -- there are two networks, but routing rules (NACLs) block the traffic between them to improve security.
A home router typically blocks incoming access to your devices. This means that people on the Internet cannot access your computer, printer, devices, etc. This is good, since there are many bots on the Internet always trying to hack into devices on your network. However, the home router allows outbound requests from your devices to the Internet (eg a website) and it is smart enough to allow the responses to come back into the network. This is equivalent to a Security Group, which has rules that determine what Inbound and Outbound requests are permitted. Security Groups are stateful, which means they automatically allow return traffic even if it is not specifically listed. The difference is that the router is acting as the Security Group, whereas in an Amazon VPC it is possible to assign a Security Group to each individual resource (like having a router on each resource).
That doesn't cover all the capabilities of an Amazon VPC, but it should give you an idea of how the network actually behaves.
I have created a Site to Site VPN connection between VPC of Google cloud Platform and AWS in North Virginia region for both the VPCs. But the problem is I have been getting a very high ping and low bandwidth while communicating between the instances. Can any one tell me the reason for this?
image showing the ping data
The ping is very high considering they are in a very close region. Please help.
Multiple reason behind the cause :
1) verify gcp network performance by gcping
2) verify the tcp size and rtt for bandwidth
3) verify with iperf or tcpdump for throughput
https://cloud.google.com/community/tutorials/network-throughput
Be aware that any VPN will be traversing the internet, so even though they are relatively close to each other there will be multiple hops before the instances are connected together.
Remember that from the instance it will need to route outside of AWS networks, then to any hops on the internet to GCP and finally routed to the instance and back again to return the response
In addition there is some variation in performance as the line will not be dedicated.
If you want dedicated performance, without traversing the internet you would need to look at AWS Direct Connect. However, this might limit your project because of cost.
One of the many limits to TCP throughout is:
Throughput <= EffectiveWindowSize / RoundTripTime
If your goal is indeed higher throughput, then you can consider tweaking the TCP window size limits. The default TCP window size under Linux is ~3MB. However, there is more to EffectiveWindowSize than that. There is also the congestion window, which will depend on factors such as packet losses and congestion control heuristics being used (eg cubic vs bbr).
As far as sanity checking the ping RTTs you are seeing, you can compare with ping times you see between an instance in AWS us-east-1 and GCP us-east4 when you are not using a VPN.
I am testing Erlang and have a few questions related to Security of the Distribution. (There is a lot of mixed information out there) These type of questions come with lots of opinions related to situations, and depends on personal comfort level on the type of data you are dealing with. For the sake of this question, lets assume it is a simple chat server where users can connect to and chat together.
Example Diagram:
The cluster will be behind a private subnet VPC with elastic-load-balancing directing all connections to these nodes (to and from). The elastic-load-balancing will be the only direct path to these nodes (there would be no way to connect to a node via name#privatesubnet).
My question is the following:
Based on this question and answer: Distributed erlang security how to?
There are two different types of inner-communication that can take place. Either, directly connecting nodes using built in functionality, or doing everything over a TCP connection with a custom protocol. The first is the most easiest, but I believe it comes with a few security issues, and I was wondering based on the above diagram if It would be good enough (Er, okay, Good Enough is not always good when dealing with sensitive information, but there can always be better ways to do everything ...)
How do you secure and Erlang cluster behind a private subnet? I would like to hide the nodes, and manually connect them, and of course use cookies on them. Is there any flaws with this approach? And since a custom protocol using TCP would be the best option, what type of impact does that have on performance? I want to know the potential security flaws(As I said, there is a lot of mixed information out there on how to do this).
I would be interested in hearing from people who have used Erlang in this manner!
On AWS, with your EC2 nodes in a private subnet, you are pretty safe from unwanted connections to your nodes. You can verify this by trying to connect (in any way) to the machines running your code: if you're using a private subnet you will be unable to do so because the instances are not even addressable outside the subnet.
Your load-balancer should not be forwarding Erlang node traffic.
You can do a little better than the above using some security-group rules. Configure your nodes to use some range of ports. Then make a group "erlang" that allows connections to that port range from the "erlang" group and denies the connection otherwise. Finally, assign that security-group to all your Erlang-running instances. This prevents instances that don't need to talk to Erlang from being able to do so.
I think you have a very "classic" setup over there.
You aren't going to connect to the cluster from the Internet ― "outside" the ELB. Assuming the "private" sub-net is shared for something else, you can allow only certain IPs (or ranges) to connect via EPMD.
In any case, some machines must be "trusted" to connect to via EPMD and some other(s) can only establish a connection to some other port(s)...otherwise anything that's running your Erlang cluster is useless.
Something to think about is: you might want to (and indeed you will have to) connect to the cluster for doing some "administrative task(s)", either from the Internet or from somewhere else. I've seen this done via SSH; Erlang support that out-of-the-box.
A final word on doing everything over a TCP connection with a custom protocol, please don't, you will end-up implementing something on your own that hardly have what Erlang offers, and it's really awesome at. In the end, you'll have the same constraints.
We have a number of 3rd party systems which are not part of our AWS account and not under our control, each of these systems have an internal iis server set up with dns which is only available from the local computer. This iis server holds an API which we want to be able to utilise from our EC2 instances.
My idea is to set up some type of vpn connection between the ec2 instance and the 3rd party system so that the ec2 instance can use the same internal dns to call the api.
AWS provide direct connect, is the correct path go down in order to do this? If it is, can anyone provide any help on how to move forward, if its not, what is the correct route for this?
Basically we have a third party system, on this third party system is an IIS server running some software which contains an API. So from the local machine I can run http://<domain>/api/get and it returns a JSON lot of code. However in order to get on to the third party system, we are attached via a VPN on an individual laptop. We need our EC2 instance in AWS to be able to access this API, so need to connect to the third party via the same VPN connection. So I think I need within AWS a separate VPC.
The best answer depends on your budget, bandwidth and security requirements.
Direct Connect is excellent. This services provides a dedicated physical network connection from your point of presence to Amazon. Once Direct Connect is configured and running your will then configure a VPN (IPSEC) over this connection. Negative: long lead times to install the fibre and relatively expensive. Positives, high security and predicable network performance.
Probably for your situation, you will want to consider setting up a VPN over the public Internet. Depending on your requirements I would recommend installing Windows Server on both ends linked via a VPN. This will provide you with an easy to maintain system provided you have Windows networking skills available.
Another good option is OpenSwan installed on two Linux system. OpenSwan provides the VPN and routing between networks.
Setup times for Windows or Linux (OpenSwan) is easy. You could configure everything in a day or two.
Both Windows and OpenSwan support a hub architecture. One system in your VPC and one system in each of your data centers.
Depending on the routers installed in each data center, you may be able to use AWS Virtual Private Gateways. The routers are setup in each data center with connection information and then you connect the virtual private gateways to the routers. This is actually a very good setup if you have the correct hardware installed in your data centers (e.g. a router that Amazon supports, which is quite a few).
Note: You probably cannot use a VPN client as the client will not route two networks together, just a single system to a network.
You will probably need to setup a DNS Forwarder in your VPC to communicate back to your private DNS servers.
Maybe sshuttle can do, what you need. Technically you can open ssh tunnel between your EC2 and remote ssh host. It can also deal with resolving dns requests at remote side. That is not perfect solution, since typical VPN has fail over, but you can use it as starting point. Later, maybe as foll back, or for testing purposes.
I'm trying to determine if boost client & server endpoints will automatically use all available network interfaces.
I've read the two links that I've found on the subject, and it seems that it's possible, but which interface is chosen, if so?
Can a server endpoint accept connections on all interfaces? Will a client endpoint automatically seek the best connection, for example, where one internet connection connects to a local region with better connectivity to geographically nearer nodes available through one device yet to the internet at large with better connectivity to the rest of the world through another device?
If they do not automatically use all available interfaces, how can they be made to do so?
Can a server endpoint accept connections on all interfaces?
Yes.
Will a client endpoint automatically seek the best connection, for example, where one internet connection connects to a local region with better connectivity to geographically nearer nodes available through one device yet to the internet at large with better connectivity to the rest of the world through another device?
The client will choose the route that has the lowest cost as determined by its IP routing tables. Each intermediate router will do the same.