Trying to work out TCO for Amazon AWS for network what is 400GB upload and download per month comparable to Mbit/s in the network calculator?
You cannot calculate raw network bandwidth (Mbits per second) of the data centre link from average download rate, or vice versa.
The download rate will depend all all sorts of other factors, including end-to-end network latency, raw bandwidth in the entire network path from client to server, network congestion, the ability of the end points to read / write the data from / to disk, etcetera.
Also, you need to decide if you want your download / uploads to happen quickly or slowly. What "user experience" do you want your users to have?
Related
We want to download objects from Google Storage via the application via the 5G network. But the download speed does not rise above 130 Mbps, despite the fact that the speed test on the device shows 400 Mbps. Can you please tell me if there are any restrictions on the download speed in this service?
Google Cloud Storage doesn't have any hard limit 1, but the network between your device and Google infrastructure may affect your download speeds.
Check your ping to GCP regions with this tool. If your data is stored in a location that has very high latency, try moving your storage bucket somewhere closer.
You can also take a look at this article to find out how you can improve your Google Cloud Storage performance.
5G implies a mobile device, and I do not know how to tune the TCP settings on a mobile device, but one of the fundamental limits to the performance of a TCP connection is:
Throughput <= WindowSize / RoundTripTime
If you cannot make the RoundTripTime smaller by accessing things more closely to you, you can seek to increase WindowSize.
I have tried many times to install the R server on an AWS instance using terminal commands without any luck. I can install it using http://www.louisaslett.com/RStudio_AMI/
and following a Youtube video but I cannot get the dropbox sync to stop "syncing". I have tried installing a fresh version using the terminal and Putty and other methods without much success.
What I wanted to use AWS for was to use the bandwidth / computing time.
I basically wanted to run an R script to download a bunch of documents which could take 2 weeks to download. I had hoped to save these on a large dropbox account I have access to but unfortunately library("RStudioAMI")
linkDropbox()
excludeSyncDropbox("*") doesn`t seem to work for me and the whole dropbox folder gets synced onto my AWS instance and I run out of space.
So basically... I think I will forget dropbox and just use AWS storage.
I want to download appox 500GB - or perhaps 1TB worth of data (running an R script to download documents and save them), it just connects to a website and downloads a document, so no ML or high computing power needed. Just a consistent connection. Once the documents are fully downloaded I would like to then just transfer them to an external hard drive I have for further analysis.
So my question is, "approximately" how much do you think this may cost, I don't care about paying 20-30$ I just don`t want to go in with inexperience/without knowledge and rack up hundreds$.
Additionally: What other instances/servers do you suggest I pay for, I feel like I dont need that much power just consistency.
Here is another SO question I opened:
Amazon AWS Dropbox link error: "No directories are being ignored."
There will be three main costs for your scenario:
Amazon EC2, which is charged hourly. You do not need much processing power, so a t3.small would probably be adequate if you're not doing any big computations. It's only about 2c/hour, which is $7 for 2 weeks.
An Amazon EBS disk volume attached to your Amazon EC2 instance for storing the data. A General Purpose volume is 10c/GB/month. So, 1TB for 2 weeks would be $50. If you configure it to use "Cold HDD (sc1)", then it's a quarter of that price.
Data Transfer for when you download from AWS. If you are using AWS in the USA, it is 9c/GB. So, 1TB = $90. This would be your major cost.
There might be some other minor costs, but they won't be significant compared to the above.
Or, given that your basic goal is to collect and download data, you could just do it on a computer at home.
If you are not strictly limited to EC2 ( which I think you are not, considering the requirement you stated and the AMI approach failed for you) , AWS Lightsail would be a much better solution
It has bundled data transfer package and acceptable performance
Here is the 1-month plan
512 MB Memory
1 Core Processor
20 GB SSD Disk
1 TB Transfer ( Data in will cost nothing, only data Out, Ex: From LightSail to your local PC )
Additional SSD - $10 for 1 TB
Average network performance for that instance I see is about 30 Megabyte per second. You can just shutdown everything and only billed for the hours you used in the month
I am trying to create a video sharing app for only one country (India). My bucket is in the India region. The size of videos will be of approximately 18MB each.
The video download takes too much time (~4mins) to download even via CloudFront. How can I accelerate my S3 download speed?
Apps that play streaming video do not download an entire video file before playing. Rather, they use a video protocol that only downloads a few seconds at a time. Amazon CloudFront can support these protocols, such Adobe's Real-Time Message Protocols and Microsoft Smooth Streaming.
See: How RTMP Distributions Work
Therefore, the time for the actual file to download is not important. Rather, the time to serve the first portion of the video is most important, followed by the ability to continue serving content as fast as the viewer consumes the content.
Connections speeds will, of course, vary depending upon each user's own Internet connection -- therefore, it is best to test the speed of your application's video playing ability from many different ISPs and types of Internet connections.
I have an application which does file upload and download. I also am able to limit upload/download speed to a desired level (CONFIGURABLE), so that my application does not consume the whole available bandwidth. I am able to achieve this using the libcurl (http) library.
But my question is, if I have to limit my upload speed to say 75% of the available upload bandwidth, how do I find out my available upload bandwidth programatically? preferably in C/C++. If it is pre-configured, I have no issues, but if it has to be learnt and adapted each time, like I said, 75% of the available upload limit, I do not know who to figure it out. Same is applicable to download. Any pointers would be of great help.
There's no way to determine the absolute network capacity between two points on a regular network.
The reason is that the traffic can be rerouted in between, other data streams appear or disappear or links can be severed.
What you can do is figure out what is the available bandwidth right now. One way to do it is to upload/download a chunk of data (say 1MB) as fast as possible (no artificial caps), and measure how long it takes. From there you can figure out what bandwidth is available now and go from there.
You could periodically measure the bandwidth again to make sure you're not too way off.
We are trying to run an ETL process in an High I/O Instance on Amazon EC2. The same process locally on a very well equipped laptop (with a SSD) take about 1/6th the time. This process is basically transforming data (30 million rows or so) from flat tables to a 3rd normal form schema in the same Oracle instance.
Any ideas on what might be slowing us down?
Or another option is to simply move off of AWS and rent beefy boxes (raw hardware) with SSDs in something like Rackspace.
We have moved most of our ETL processes off of AWS/EMR. We host most of it on Rackspace and getting a lot more CPU/Storage/Performance for the money. Don't get me wrong AWS is awesome but there comes a point where it's not cost effective. On top of that you never know how they are really managing/virtualizing the hardware that applies to your specific application.
My two cents.