I'm confused about some facets of the Amazon Web Services stuff. Here is what I want to do.
My site lets users enter equations and solve them. Some of the equations will deal with large data sets and math that is too computationally expensive for the browser.
My site will look at each equation and determine if it should be solved in the browser or on a server.
If it needs to be solved on the server, I want to do one of two things. First, either send the data and a function and have AWS run the code on that data. The other option is to have preset code with is given data.
AWS then runs the code and returns a JSON of the solution.
For example, lets say that a user has a numeric matrix of 1,000 by 1,000 and they want to take the inverse or do Gaussian elimination. My code would look at the size of the matrix and decide that it needs to be run on the server. The code would then call my function on AWS to solve this, send it the data, and AWS returns the answer.
As I read, I don't understand exactly how to set up EC2 to call a function from a server or from an ajax call. Does AWS not do what I think it does? Do I need to host my site on AWS to do this?
If it matters, I am running a LAMP stack on Hostmonster.
You can use Amazon EC2 to create a server (eg a web server) that is accessible on the Internet. What you load on the server, and how you use the server, is up to you.
There is no functionality provided by Amazon EC2 that would help you for your specific stated use case. Anything you would run on a "normal" server can be run on Amazon EC2, since it is just a virtual machine running an operating system and whatever software you configure.
From your description, you will need to develop a web app that runs mostly in the browser (eg with JavaScript), but also makes calls to a back-end server. How you do that is totally in your control.
Related
I am trying to find the best way to architect a low cost solution to provide an on-demand web server for a certain amount of time.
The context is as follows: I have some large amount of data sitting on S3. From time to time, users will want to consult that data. I've written a Flask app that can display the data in a nice way for them. Beign poorly written, it really only accepts a single user session at the time. Currently therefore they have to download the Flask app and run it on their own machine.
I would like to find a way for users to request a cloud-based web server that would run the Flask app (through a docker container for example) on-demand, and give them access to it quickly, without having to do much if anything on their own machine.
Every user wanting to view the data would have their own web server created on demand (to avoid multiple users sharing the same web server, which wouldn't work with my Flask app)
Critically, and in order to avoid cost, the web server would terminate itself automatically after some (configurable) idle time (possibly with the Flask app informing the user that it's about to shut down, so that they can "renew" the lease).
Initially I thought that maybe AWS Fargate would be good: it can run docker instances, is quite configurable in terms of CPU/disk it can get (my Flask app is resource-hungry), and at least on paper could be used in a way that there is zero cost when users are not consulting the data (bar S3 costs naturally). But it's when it comes to the detail that I'm not sure...
How to ensure that every new user gets their own Fargate instance?
How to shut-down the instance automatically after idle time?
Is Fargate quick enough in terms of boot time?
The closest I can think is AWS App Runner. It's built on top of Fargate and it provides an intelligent scale out mechanism (probably you are not interested in this) as well as a scale to (almost) 0 capability. The way it works is that when the endpoint is solicited and it's doing work you pay for the entire fargate task (cpu/memory) you have selected in the configuration. If the endpoint is doing nothing you only pay for the memory (note the memory cost is roughly 20% of the entire cost so it's not scale to 0 but "quasi"). Checkout the pricing examples at the bottom of this page.
Please note you can further optimize costs by pausing/starting the endpoint (when it's paused you pay nothing) but in that case you need to create the logic that pauses/restarts it.
Another option you may want to explore is using Lambda this way (which would allow you to use the same container image and benefit from the intrinsic scale to 0 of Lambda). But given your comment "Lambda doesn’t have enough power, and the timeout is not flexible" may be a show stopper.
Using AWS cloud services I would like to create a truly distributed application (ie where every node of the network is actually located in a different place) , based on a distributed consensus algorithm (similar to RAFT). Unfortunately I don't know AWS services very well and I only have in mind what I have to do, but I still have a lot of doubts about what services should I use (after having understood how they work).
In summary (as in the figure) what I would like to do is this:
1) User can use an API, called "main API" in the image (providing certain parameters). This API "starts the algorithm" by communicating directly with one or more remote machines.
2) Each machine has its own API (with its own HTTP address), and for this reason it is able to communicate with all the other machines. Each of these machines runs the same consensus algorithm.
3)At the end of the algorithm (ie when a machine chooses the value) it writes this value into a database (DB in the image). For example, there may be a database for every single machine, but it is not mandatory.
So my question is about which AWS services are right for this purpose and above all how to best use these services for my task.
What is the best service for creating your own machines and exposing them to the world via API? I only know (very poorly) the API Gateway service, but I don't know absolutely how to "own" remote machines and databases.
What is the best way to associate the algorithm code with a single machine? I was thinking of using AWS Lambda (a program is called in response to a certain HTTP call on the respective API). As for the output, I thought I was writing to the Dynamo DB database.
I would like to send to the main API parameters that specify the number of machines (and DBs) that will be used (maybe they could be "dynamically allocated"). Can you do this thing? For example I could tell the main API that I want to execute the algorithm with 3 machines and therefore only 3 remote machines (I don't care what) will work.
Thanks a lot!
Is it possible to open a website,like facebook.com for example, on an amazon web service?
My objective is to automate a certain task in a game and to do so without having to be online on my computer. The point is to spend less time on that game, but to not be left behind on the progress. (I'm building a bot to automate the daily tasks there, just need to know if i can now leave everything running on amazon)
Another project i want to do is to automate access to my email account and perform certain tasks depending on the emails i receive.
You get the point, i tried searching on google but i only find results about creating or hosting your own website in there and not about accessing existing websites and using automation in them.
It sounds like what you want is a virtual private server - basically a computer in the cloud that you control and is always on.
AWS have a service called LightSail for this kind of purpose. Under the hood lightsail just uses EC2, but lightsail takes away a lot of the options and configuration to provide a simpler 'click and go' kind of service.
Once you have a server you can schedule regular tasks. Depending on the complexity of your needs, you could look at using Cron as a scheduler and curl for you http requests.
For the specifics of any project you have I would suggest opening a new question with details of what you are trying to do, the reading you have done, and examples of any code you have tried.
Lately I see this buzzword phrase in job requirements in position descriptions of skills needed from an applicant (knowledge of):
"Horizontally scalable RESTful services"...
What exactly is it? I cannot google anything that would really explain the notion.
I expect that horizontal scaling is adding more servers to handle more load, rather than adding more memory and cpus to a server, as that is vertical scaling.
So, you could have a docker container that has your REST service, which should be stateless. There are many ways to scale in production.
On each connection you could then create a new container, and once that service is done you remove it, so every connection has its own server.
If you are running something like nodejs, which is very light, then you could get away with this, but if you are using a heavier web server then you may want to look at something like autoscaling, from AWS, and as the load on each container goes up, create a new one, so you don't overload any particular server.
You don't have to use Docker, but it wouldn't hurt for you to learn about it.
running multiple instance of application in multiple machines with a load balancer, We traditionally call it as web farming.
If your API is stateless, then you can directly go to load balancer without docker.
I have a web app running on php, mysql, apache on a virtual windows server. I want to redesign it so it is scalable (for fun so I can learn new things) on AWS.
I can see how to setup an EC2 and dump it all in there but I want to make it scalable and take advantage of all the cool features on AWS.
I've tried googling but just can't find a simple guide (note - I have no command line experience of Linux)
Can anyone direct me to detailed resources that can lead me through the steps and teach me? Or alternatively, summarise the steps in an answer so I can research based on what you say.
Thanks
AWS is growing and changing all the time, so there aren't a lot of books to help. Amazon offers training that's excellent. I took their three day class on Architecting with AWS that seems to be just what you're looking for.
Of course, not everyone can afford to spend the travel time and money to attend a class. The AWS re:Invent conference in November 2012 had a lot of sessions related to what you want, and most (maybe all) of the sessions have videos available online for free. Building Web Scale Applications With AWS is probably relevant (slides and video available), as is Dissecting an Internet-Scale Application (slides and video available).
A great way to understand these options better is by fiddling with your existing application on AWS. It will be easy to just move it to an EC2 instance in AWS, then start taking more advantage of what's available. The first thing I'd do is get rid of the MySql server on your own machine and use one offered with RDS. Once that's stable, create one or more read replicas in RDS, and change your application to read from them for most operations, reading from the main (writable) database only when you need completely current results.
Does your application keep any data on the web server, other than in the database? If so, get rid of all local storage by moving that data off the EC2 instance. Some of it might go to the database, some (like big files) might be suitable for S3. DynamoDB is a good place for things like session data.
All of the above reduces the load on the web server to just your application code, which helps with scalability. And now that you keep no state on the web server, you can use ELB and Auto-scaling to automatically run multiple web servers (and even automatically launch more as needed) to handle greater load.
Does the application have any long running, intensive operations that you now perform on demand from a web request? Consider not performing the operation when asked, but instead queueing the request using SQS, and just telling the user you'll get to it. Now have long running processes (or cron jobs or scheduled tasks) check the queue regularly, run the requested operation, and email the result (using SES) back to the user. To really scale up, you can move those jobs off your web server to dedicated machines, and again use auto-scaling if needed.
Do you need bigger machines, or perhaps can live with smaller ones? CloudWatch metrics can show you how much IO, memory, and CPU are used over time. You can use provisioned IOPS with EC2 or RDS instances to improve performance (at a cost) as needed, and use difference size instances for more memory or CPU.
All this AWS setup and configuration can be done with the AWS web console, or command-line tools, or SDKs available in many languages (Python's boto library is great). After learning the basics, look into CloudFormation to automate it better (I've written a couple of posts about that so far).
That's a bit of the 10,000 foot high view of one approach. You'll need to discover the details of each AWS service when you try to use them. AWS has good documentation about all of them.
Depending on how you look at it, this is more of a comment than it is an answer, but it was too long to write as a comment.
What you're asking for really can't be answered on SO--it's a huge, complex question. You're basically asking is "How to I design a highly-scalable, durable application that can be deployed on a cloud-based platform?" The answer depends largely on:
The specifics of your application--what does it do and how does it work?
Your tolerance for downtime balanced against your budget
Your present development and deployment workflow
The resources/skill sets you have on-staff to support the application
What your launch time frame looks like.
I run a software consulting company that specializes in consulting on Amazon Web Services architecture. About 80% of our business is investigating and answering these questions for our clients. It's a multi-week long project each time.
However, to get you pointed in the right direction, I'd recommend that you look at Elastic Beanstalk. It's a PaaS-like service that abstracts away the underlying AWS resources, making AWS easier to use for developers who don't have a lot of sysadmin experience. Think of it as "training wheels" for designing an autoscaling application on AWS.