Is it possible to open a website,like facebook.com for example, on an amazon web service?
My objective is to automate a certain task in a game and to do so without having to be online on my computer. The point is to spend less time on that game, but to not be left behind on the progress. (I'm building a bot to automate the daily tasks there, just need to know if i can now leave everything running on amazon)
Another project i want to do is to automate access to my email account and perform certain tasks depending on the emails i receive.
You get the point, i tried searching on google but i only find results about creating or hosting your own website in there and not about accessing existing websites and using automation in them.
It sounds like what you want is a virtual private server - basically a computer in the cloud that you control and is always on.
AWS have a service called LightSail for this kind of purpose. Under the hood lightsail just uses EC2, but lightsail takes away a lot of the options and configuration to provide a simpler 'click and go' kind of service.
Once you have a server you can schedule regular tasks. Depending on the complexity of your needs, you could look at using Cron as a scheduler and curl for you http requests.
For the specifics of any project you have I would suggest opening a new question with details of what you are trying to do, the reading you have done, and examples of any code you have tried.
Related
I have a website that I am trying to have the web form connect to a MySQL database running on Amazon's RDS to post and retrieve information. I'm an absolute beginner with code but have managed to get myself this far (creative3c.org). I've had coworkers and friends offer some help, but their knowledge doesn't extend to everything I was told I would need (AWS API Gateway, Lambda--anything else?).
I've been pulling my hair out for a week looking through tutorials, articles, and step-by-step guides but so many presume extensive knowledge on the viewer or they all talk about what I don't need (like phpmyadmin--and php won't work for S3 or Lambda).
Am I jumping too far into the really complex stuff? The person that told me to go the AWS route is certified and brilliant with code--but unfortunately they are fickle, busy, and aren't a good teacher to distill their knowledge. I don't know if I should have gone with something simpler. If you view the website, you'll probably understand how basic it is.
I'm stuck and really stressed about finishing this website and appreciate the help to get me in the right direction! I feel I'm so close! I'm really good at scaling up from a small example of exactly what I need--I just need that initial example!
I'm pleased to hear that you've learnt so quickly. All the terminology floating around can be very confusing. Just remember: AWS is just the platform you deploy to. It can be as simple and complicated as you want it to be
I'm not an AWS expert but here's my birdseye view
You could build an entire running website on your laptop then simply deploy that wholesale to a LAMP server that you've created up in AWS. Now you have a web application running in AWS, without using any of the AWS jargon (beanstalks, lambdas...)
Thats when you would follow this link to provision your server: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/install-LAMP.html
Or you could put the database piece of your application into RDS (a database on the cloud) then put the web application piece in a seperate web server then configure those two servers to talk to each other.
You have a web site but it's now running on two seperate machines
Or (I'm a bit hazy on this) for the web app you could instead deploy bits of your web code to lambda and stick them all together
In all cases you can apply 'elastic beanstalk' to automatically grow and shrink the computers running your site.
Like I said it can be as simple and complicated as you want it to be - and you don't need it to be complicated so the BlueHost option is fine.
I'm confused about some facets of the Amazon Web Services stuff. Here is what I want to do.
My site lets users enter equations and solve them. Some of the equations will deal with large data sets and math that is too computationally expensive for the browser.
My site will look at each equation and determine if it should be solved in the browser or on a server.
If it needs to be solved on the server, I want to do one of two things. First, either send the data and a function and have AWS run the code on that data. The other option is to have preset code with is given data.
AWS then runs the code and returns a JSON of the solution.
For example, lets say that a user has a numeric matrix of 1,000 by 1,000 and they want to take the inverse or do Gaussian elimination. My code would look at the size of the matrix and decide that it needs to be run on the server. The code would then call my function on AWS to solve this, send it the data, and AWS returns the answer.
As I read, I don't understand exactly how to set up EC2 to call a function from a server or from an ajax call. Does AWS not do what I think it does? Do I need to host my site on AWS to do this?
If it matters, I am running a LAMP stack on Hostmonster.
You can use Amazon EC2 to create a server (eg a web server) that is accessible on the Internet. What you load on the server, and how you use the server, is up to you.
There is no functionality provided by Amazon EC2 that would help you for your specific stated use case. Anything you would run on a "normal" server can be run on Amazon EC2, since it is just a virtual machine running an operating system and whatever software you configure.
From your description, you will need to develop a web app that runs mostly in the browser (eg with JavaScript), but also makes calls to a back-end server. How you do that is totally in your control.
I have a web app running on php, mysql, apache on a virtual windows server. I want to redesign it so it is scalable (for fun so I can learn new things) on AWS.
I can see how to setup an EC2 and dump it all in there but I want to make it scalable and take advantage of all the cool features on AWS.
I've tried googling but just can't find a simple guide (note - I have no command line experience of Linux)
Can anyone direct me to detailed resources that can lead me through the steps and teach me? Or alternatively, summarise the steps in an answer so I can research based on what you say.
Thanks
AWS is growing and changing all the time, so there aren't a lot of books to help. Amazon offers training that's excellent. I took their three day class on Architecting with AWS that seems to be just what you're looking for.
Of course, not everyone can afford to spend the travel time and money to attend a class. The AWS re:Invent conference in November 2012 had a lot of sessions related to what you want, and most (maybe all) of the sessions have videos available online for free. Building Web Scale Applications With AWS is probably relevant (slides and video available), as is Dissecting an Internet-Scale Application (slides and video available).
A great way to understand these options better is by fiddling with your existing application on AWS. It will be easy to just move it to an EC2 instance in AWS, then start taking more advantage of what's available. The first thing I'd do is get rid of the MySql server on your own machine and use one offered with RDS. Once that's stable, create one or more read replicas in RDS, and change your application to read from them for most operations, reading from the main (writable) database only when you need completely current results.
Does your application keep any data on the web server, other than in the database? If so, get rid of all local storage by moving that data off the EC2 instance. Some of it might go to the database, some (like big files) might be suitable for S3. DynamoDB is a good place for things like session data.
All of the above reduces the load on the web server to just your application code, which helps with scalability. And now that you keep no state on the web server, you can use ELB and Auto-scaling to automatically run multiple web servers (and even automatically launch more as needed) to handle greater load.
Does the application have any long running, intensive operations that you now perform on demand from a web request? Consider not performing the operation when asked, but instead queueing the request using SQS, and just telling the user you'll get to it. Now have long running processes (or cron jobs or scheduled tasks) check the queue regularly, run the requested operation, and email the result (using SES) back to the user. To really scale up, you can move those jobs off your web server to dedicated machines, and again use auto-scaling if needed.
Do you need bigger machines, or perhaps can live with smaller ones? CloudWatch metrics can show you how much IO, memory, and CPU are used over time. You can use provisioned IOPS with EC2 or RDS instances to improve performance (at a cost) as needed, and use difference size instances for more memory or CPU.
All this AWS setup and configuration can be done with the AWS web console, or command-line tools, or SDKs available in many languages (Python's boto library is great). After learning the basics, look into CloudFormation to automate it better (I've written a couple of posts about that so far).
That's a bit of the 10,000 foot high view of one approach. You'll need to discover the details of each AWS service when you try to use them. AWS has good documentation about all of them.
Depending on how you look at it, this is more of a comment than it is an answer, but it was too long to write as a comment.
What you're asking for really can't be answered on SO--it's a huge, complex question. You're basically asking is "How to I design a highly-scalable, durable application that can be deployed on a cloud-based platform?" The answer depends largely on:
The specifics of your application--what does it do and how does it work?
Your tolerance for downtime balanced against your budget
Your present development and deployment workflow
The resources/skill sets you have on-staff to support the application
What your launch time frame looks like.
I run a software consulting company that specializes in consulting on Amazon Web Services architecture. About 80% of our business is investigating and answering these questions for our clients. It's a multi-week long project each time.
However, to get you pointed in the right direction, I'd recommend that you look at Elastic Beanstalk. It's a PaaS-like service that abstracts away the underlying AWS resources, making AWS easier to use for developers who don't have a lot of sysadmin experience. Think of it as "training wheels" for designing an autoscaling application on AWS.
I'm writing an app wherein the premise is that people who can't directly fund a charity out of their pocket could automatically work on Amazon AWS HITs in order to bring clean water to the 3rd world - I'd known there was an Amazon AWS API but is very unclear on how to retrieve a HIT to be worked on, rather than just consume some data about some tasks I'm trying to complete.
Is there any way to retrieve a HIT to be worked on through the AWS API or otherwise?
Thanks ahead of time.
As far as I know, the only way to work on a HIT is through the mTurk website. i.e. - not via API.
There is a site that is trying to do something very similar to what you have described. http://www.sparked.com/
At my day job we have load balanced web servers which talk to load balanced app servers via web services (and lately WCF). At any given time, we have 4-6 different teams that have the ability to add new web sites or services or consume existing services. We probably have about 20-30 different web applications and corresponding services.
Unfortunately, given that we have no centralized control over this due to competing priorities, org structures, project timelines, financial buckets, etc., it is quite a mess. We have a variety of services that are reused, but a bunch that are specific to a front-end.
Ideally we would have better control over this situation, and we are trying to get control over it, but that is taking a while. One thing we would like to do is find out more about what all of the inter-relationships between web sites and the app servers.
I have used Reflector to find dependencies among assemblies, but would like to be able to see the traffic patterns between services.
What are the options for trying to map out web service relationships? For the most part, we are mainly talking about internal services (web to app, app to app, batch to app, etc.). Off the top of my head, I can think of two ways to approach it:
Analyze assemblies for any web references. The drawback here is that not everything is a web reference and I'm not sure how WCF connections are listed. However, this would at least be a start for finding 80% of the connections. Does anyone know of any tools that can do that analysis? Like I said, I've used Reflector for assembly references but can't find anything for web references.
Possibly tap into IIS and passively monitor the traffic coming in and out and somehow figure out what is being called and where from. We are looking at enterprise tools that could help but it would be a while before they are implemented (and cost a lot). But is there anything out there that could help out quickly and cheaply? One tool in particular (AmberPoint) can tap into IIS on the servers and monitor inbound and outbound traffic, adds a little special sauce and begin to build a map of the traffic. Very nice, but costs a bundle.
I know, I know, how the heck did you get into this mess in the first place? Beats me, just trying to help us get control of it and get out of it.
Thanks,
Matt
The easiest way is to look through the logs, but if that doesn't include the referrer than you may also want to monitor what is going out from your web to the app server. You can use tools like Wireshark or Microsoft Network Monitor to see this traffic.
The other "solution" and I use this loosely is to bind a specific web server to app server and then run through a bundle and see what it is hitting on the app server. You could probably do this in a test environment to lesson the effects on the users of the site.
You need a service registry (UDDI??)... If you had a means to catalog these services and their consumers, it would make this job of dependency discovery a lot easier. That is not an easy solution, though. It takes time and documentation to get a catalog in place.
I think the quickest solution would be to query your IIS logs and find source URLs which originate from your own servers. You would at least be able to track down which servers your consumers are coming from.
Also, if you already have some kind of authentication mechanism in place, you could trace who is using a particular service based on login.
You are right about AmberPoint. There are other tools that catalog the service traffic and provide reports showing what is happening to your services. Systinet, SOA Software and Actional also has a products similar to Amberpoint but Amberpoint has a free-ware version, I believe.