I want to run a dpdk experiment using Amazon EC2 service. But there are a great number of services in AWS. I don't know which one to choose.
My experiment need two servers connected together using 10Gbps network adpater supporting dpdk. I run pktgen-dpdk on one server to send packets towards the other server. And another dpdk application will run in the other server to deal with these packets.
I think I can rent servers such c4.8xlarge c4.4xlarge. But I don't know how to set up the local network between them. The local network should have low latency.
Any suggestions will be appreciated! Thank you!
You're looking for Virtual Private Cloud (VPC). An AWS EC2 "instance" like your c4.8xlarge is just a machine. The VPC and several other components allow you to set up a broader network, routing, security groups (basically, a firewall) and other networking capabilities, including in your case a Gateway, which would allow your dpkg system to look out onto the Internet to find dependencies.
The in-network latency is extremely low, < 1ms in our experience. I think most current EC2 instances support 10Gbps networking and other speedy network capabilities.
Related
There an external HTTP server (located somewhere in the US), which we must communicate with. We use AWS EC2 instances.
While we can buy a "bigger instance" to improve the internal network performance, is there a way to lessen (optimize?) the roundtrip time between our EC2 instance and the external server? Are therer any tools that could be useful?
You haven't specified what type of EC2 instance you use which is a big factor determining the network performance.
You also said
from my home network, it is much faster than when running on an AWS EC2 (regardless of where the ec2 is hosted)
I know nothing about your home network and your EC2 instance config so this is hard to judge but I'd expect, on average, the EC2 instance having faster network than what's available on the end user's site.
It's also not 100% clear what you are measuring. You said "round trip time" so you are only interested in end-to-end latency? Any particular throughput requirements?
That said, here's a useful cheat sheet which you can download and check your instance type: https://cloudonaut.io/ec2-network-performance-cheat-sheet/
Furthermore, you can use iperf (or iperf3) to perform some experiments on both sides of the connection:
https://www.tecmint.com/test-network-throughput-in-linux/
https://aws.amazon.com/premiumsupport/knowledge-center/network-throughput-benchmark-linux-ec2/
I got a new laptop and planning to dedicate the current laptop as a central log monitoring system for the server clusters already set up on AWS. AWS servers have static IP, while my personal laptop will be connected to Wifi. The clusters receive low to moderate traffic and there aren't many logs generated.
To use the laptop as a central log monitoring system, I can do one of these things:
Stream logs in realtime(Using streams to reduce reconnection overheads)
HTTP Long Polling(Can't push as my ISP doesn't allow me a static IP)
Make a VPN server and figure out some way to push/poll logs.
I think the 1st option(streaming logs) looks the most promising.
Is there some better way to this?
Also, how do I stream logs in this setup considering clients have static IP while my central server has dynamic IP?
Are there any open-source/existing services that achieves this already(Why re-invent the wheel when you have a start!)?
Thank you in advance!
I have created a Site to Site VPN connection between VPC of Google cloud Platform and AWS in North Virginia region for both the VPCs. But the problem is I have been getting a very high ping and low bandwidth while communicating between the instances. Can any one tell me the reason for this?
image showing the ping data
The ping is very high considering they are in a very close region. Please help.
Multiple reason behind the cause :
1) verify gcp network performance by gcping
2) verify the tcp size and rtt for bandwidth
3) verify with iperf or tcpdump for throughput
https://cloud.google.com/community/tutorials/network-throughput
Be aware that any VPN will be traversing the internet, so even though they are relatively close to each other there will be multiple hops before the instances are connected together.
Remember that from the instance it will need to route outside of AWS networks, then to any hops on the internet to GCP and finally routed to the instance and back again to return the response
In addition there is some variation in performance as the line will not be dedicated.
If you want dedicated performance, without traversing the internet you would need to look at AWS Direct Connect. However, this might limit your project because of cost.
One of the many limits to TCP throughout is:
Throughput <= EffectiveWindowSize / RoundTripTime
If your goal is indeed higher throughput, then you can consider tweaking the TCP window size limits. The default TCP window size under Linux is ~3MB. However, there is more to EffectiveWindowSize than that. There is also the congestion window, which will depend on factors such as packet losses and congestion control heuristics being used (eg cubic vs bbr).
As far as sanity checking the ping RTTs you are seeing, you can compare with ping times you see between an instance in AWS us-east-1 and GCP us-east4 when you are not using a VPN.
I need to host a service with rest-api on a server which does below listed tasks:
Download and upload files in s3 bucket
Run some cpu intensive computations
Return json response
I know an ec2 instance will be a better approach to host my service but given price differences between workspace and ec2 instance, I am exploring this route. Are there any limitations on amazon workspace that might prevent me from using them for my use case?
I came across ngrok which I believe can help me direct requests over the internet to my workspace local server.
Has anyone played around with it and could add some suggestion?
AWS terms of service do not allow you to do that I’m afraid. See section 36 on workspaces.
http://aws.amazon.com/service-terms/
36.3. You and End Users may only use the WorkSpaces Services for an End User’s personal or office productivity. WorkSpaces are not meant to accept inbound network connections, be used as server instances, or serve web traffic or your network traffic. You may not reconfigure the inbound network connections of your WorkSpaces. We may shut down WorkSpaces that are used in violation of this Section or other provisions of the Agreement.
I suggest you use an r5a.xlarge for the lowest cost 32GB RAM instance type (it’s AMD processor is cheaper than r5 on intel). Investigate whether spot instances would work if your state persists on S3 and not in the local instance, otherwise if you need it for at least a year reserved instances are discounted over on demand pricing.
We have a number of 3rd party systems which are not part of our AWS account and not under our control, each of these systems have an internal iis server set up with dns which is only available from the local computer. This iis server holds an API which we want to be able to utilise from our EC2 instances.
My idea is to set up some type of vpn connection between the ec2 instance and the 3rd party system so that the ec2 instance can use the same internal dns to call the api.
AWS provide direct connect, is the correct path go down in order to do this? If it is, can anyone provide any help on how to move forward, if its not, what is the correct route for this?
Basically we have a third party system, on this third party system is an IIS server running some software which contains an API. So from the local machine I can run http://<domain>/api/get and it returns a JSON lot of code. However in order to get on to the third party system, we are attached via a VPN on an individual laptop. We need our EC2 instance in AWS to be able to access this API, so need to connect to the third party via the same VPN connection. So I think I need within AWS a separate VPC.
The best answer depends on your budget, bandwidth and security requirements.
Direct Connect is excellent. This services provides a dedicated physical network connection from your point of presence to Amazon. Once Direct Connect is configured and running your will then configure a VPN (IPSEC) over this connection. Negative: long lead times to install the fibre and relatively expensive. Positives, high security and predicable network performance.
Probably for your situation, you will want to consider setting up a VPN over the public Internet. Depending on your requirements I would recommend installing Windows Server on both ends linked via a VPN. This will provide you with an easy to maintain system provided you have Windows networking skills available.
Another good option is OpenSwan installed on two Linux system. OpenSwan provides the VPN and routing between networks.
Setup times for Windows or Linux (OpenSwan) is easy. You could configure everything in a day or two.
Both Windows and OpenSwan support a hub architecture. One system in your VPC and one system in each of your data centers.
Depending on the routers installed in each data center, you may be able to use AWS Virtual Private Gateways. The routers are setup in each data center with connection information and then you connect the virtual private gateways to the routers. This is actually a very good setup if you have the correct hardware installed in your data centers (e.g. a router that Amazon supports, which is quite a few).
Note: You probably cannot use a VPN client as the client will not route two networks together, just a single system to a network.
You will probably need to setup a DNS Forwarder in your VPC to communicate back to your private DNS servers.
Maybe sshuttle can do, what you need. Technically you can open ssh tunnel between your EC2 and remote ssh host. It can also deal with resolving dns requests at remote side. That is not perfect solution, since typical VPN has fail over, but you can use it as starting point. Later, maybe as foll back, or for testing purposes.