I have a deep learning web application deployed in GCE. I created a template to build a VM instance group. Then added loading balancing to it.
I plan that when each user accesses the URL, the requests from the same user will always be assigned to a VM instance. I use gunicorn -b 0.0.0.0:5000 wsgi:app -t 600 as part of the startup script. (I also tried with workers, gevent. But requests from different users can be handled in the same instance, as a result of which, the results were affected by each other. So I want requests from different users to be handled in different instances.)
To do so, I tried different CPU utilization for autoscaling. It can autoscale with new instances. But from the results, sometimes the requests are still handled in the same instance.
I also tried Kurbenetes, app engine, and cloud run. Mistakes are similar. I feel I am working in the wrong direction.
Thanks in advance.
---UPDATE---
As mentioned by #John Hanley, assigning requests from a user always to the same instance is not the targeted feature of these products. If you are looking for the answer to this question, you may try the Cloud Tasks + App Engine.
Actually, I want requests from different users to be handled in different instances so that the back-end deep learning algorithm's results cannot affect each other.
So, instead of spinning up an instance, another way to solve this is to store necessary data from each user in a common database with a unique session ID.
A simple demo can be found in https://cloud.google.com/python/docs/getting-started/session-handling-with-firestore
Hope this can be helpful for anyone struggling with similar problems.
As mentioned in the comment section by #John Hanley, the best approach would be to use App Engine + Cloud Tasks, I suggest you to check the next tutorial, although it uses ngrok instead of unicorn the idea of the of the workflow should be similar for what you want to achieve.
Related
I am trying to find the best way to architect a low cost solution to provide an on-demand web server for a certain amount of time.
The context is as follows: I have some large amount of data sitting on S3. From time to time, users will want to consult that data. I've written a Flask app that can display the data in a nice way for them. Beign poorly written, it really only accepts a single user session at the time. Currently therefore they have to download the Flask app and run it on their own machine.
I would like to find a way for users to request a cloud-based web server that would run the Flask app (through a docker container for example) on-demand, and give them access to it quickly, without having to do much if anything on their own machine.
Every user wanting to view the data would have their own web server created on demand (to avoid multiple users sharing the same web server, which wouldn't work with my Flask app)
Critically, and in order to avoid cost, the web server would terminate itself automatically after some (configurable) idle time (possibly with the Flask app informing the user that it's about to shut down, so that they can "renew" the lease).
Initially I thought that maybe AWS Fargate would be good: it can run docker instances, is quite configurable in terms of CPU/disk it can get (my Flask app is resource-hungry), and at least on paper could be used in a way that there is zero cost when users are not consulting the data (bar S3 costs naturally). But it's when it comes to the detail that I'm not sure...
How to ensure that every new user gets their own Fargate instance?
How to shut-down the instance automatically after idle time?
Is Fargate quick enough in terms of boot time?
The closest I can think is AWS App Runner. It's built on top of Fargate and it provides an intelligent scale out mechanism (probably you are not interested in this) as well as a scale to (almost) 0 capability. The way it works is that when the endpoint is solicited and it's doing work you pay for the entire fargate task (cpu/memory) you have selected in the configuration. If the endpoint is doing nothing you only pay for the memory (note the memory cost is roughly 20% of the entire cost so it's not scale to 0 but "quasi"). Checkout the pricing examples at the bottom of this page.
Please note you can further optimize costs by pausing/starting the endpoint (when it's paused you pay nothing) but in that case you need to create the logic that pauses/restarts it.
Another option you may want to explore is using Lambda this way (which would allow you to use the same container image and benefit from the intrinsic scale to 0 of Lambda). But given your comment "Lambda doesn’t have enough power, and the timeout is not flexible" may be a show stopper.
I am trying to build a web application with Kubernetes and Amazon Web Service.
I believe there are many different approaches but I would like to ask for your opinions!
To make it simple, the app would be a simple web page that display information. User logs in, and based on filters, get specific information.
My reflection is on the k8s inside architecture:
Can I put my whole application as a Pod? That way, cluster and node scalability would make the app available for each user by allocating each of them 1 pod. Is that good practice?
By following that logic, every different elements of my app would be a container. For instance, in a simple way, 1 container that contains the front of the app, 1 container that has the data access/management, 1 container for backend/auth etc
So 1 user would "consume" 1 pod which containers' are discussing together to give the required data from the user. k8s would create a pod for every users, scaling up/down the nodes number etc...
But then, except for the data itself, everything would be dockerized and stored on ECR (Elastic Container Registry) right? And so no need for any S3/EBS/EFS in my opinion.
I am quite new at AWS and k8s, so please feel free to give honest opinions :) Feedback, good or bad, is always good to take.
Thanks in advance!
I'd recommend a layout where:
Any container can perform its unit of work for any user
One container per pod
Each component of your application has its own (stateless) deployment
It probably won't work well to try to put your entire application into a single multi-container pod. This means the entire application would need to fit on a single node; some applications are larger than that, and even if it fits, it can lead to trouble with scheduling. It also means that, if you update the image for any single container, you need to delete and recreate all of the containers, which could be more disruptive than you want.
Trying to create a pod per user also will present some practical problems. You need to figure out how to route inbound requests to a particular user's pod, and keep requests within that user's set of containers; Kubernetes doesn't have any sort of native support for this. A per-user pod will also keep using resources even if it's overnight or the weekend for that user and they're not using the application. You would also need something with access to the Kubernetes API to create and destroy resources as new users joined your platform.
In an AWS-specific context you might consider using RDS (hosted PostgreSQL/MySQL) or S3 (object storage) for data storage (and again one database or S3 bucket shared across all customers). ECR is useful, but as a place to store your Docker images; that is, your built code, but not any of the persisted data or running containers.
My company is currently evaluating hyperledger(fabric) and we're using it for our POC. It looks very promising and we're targeting rolling out to production in next few months.
We're targeting AWS as our production environment.
However, we're struggling to find good tutorial/practices/recommendations about operating hyperledger network in such environment.
I'm aware that Cello is aiming to solve/ease deploying/monitoring hyperledger network but i also read that its not production ready yet. Question is, should we even consider looking at Cello at this point?
If not, what are our alternatives? Docker swarm, kubernetes?
I also didn't find information about recommended instance types. I understand this is application and AWS specific but what are the minimal system requirements
(memory&CPU&network) for example for 'peer' node (our application is not network intensive, nor a lot of transactions will be submitted per hour/day, only few of them per day).
Another question is where to create those instances on AWS from geographical&decentralization point of view. Does it make sense all of them to be created in same region? Or, we must create instances running in different regions?
Tnx a lot.
Igor.
yes, look at Cello.. if nothing else it will help you see the aws deployment model.
really nothing special..
design the desired system, peers, orderer, gateways, etc..
then decide who many ec2 instance u need to support that.
as for WHERE (region).. depends on where the connecting application is and what kind of fault tolerance you need for your business model.
one of the businesses I am working with wants a minimum of 99.99999 % availability. so, multi-region is critical. its just another ec2 instance with sockets open from different hosts..
aws doesn't provide much in terms of support for hyperledger. they have some templates which allow you to setup the VMs initially, but that's stuff you can do yourself as well.
you are right, the documentation is very light and most of the time confusing. I got to the point where I can start from scratch with a brand new VM and got everything ready and deploy my own network definition and chaincode and have the scripts to do that.
IBM cloud has much better support for hyperledger however. you can design your network visually, you can download your connection profiles, deploy and instantiate chaincode, create and join channels, handle certificates, pretty much everything you need to run and support such a network. It's light years ahead of AWS. They even have a full CI / CD pipepline that you could replicate for your own project. if you look at their marbles demo, you'll see what i mean.
Cello is definitely worth looking at, with the caveat that it's incubation meaning, not real yet, not production ready and not really useful until it becomes a fully fledged product.
I've been tasked with setting up multiple isolated, client-specific instances of a web application on a single Amazon EC2 instance using Docker. The base application is fundamentally the same for each instance, but has been customized for each client.
The goal is as follows:
1) Each container would be secured and "sandboxed" such that no one container could affect the others or the host. It's my understanding Docker does this anyway, but I just want to be sure.
2) Each container would be "complete" and have its own users, database, access keys, etc.
3) A user associated with one container should have no way to access any aspect of any other container or the host.
I've searched the similar questions and some seem to touch on the idea but not completely answer my question.
I know this likely goes against the Docker philosophy, but that aside, is this feasible, and what would be the best approach to achieve this? We have used SSH tunnels to access relevant ports in the past when there was only one client per host, but is there a safe way to do this with multiple clients on the same host? Would this setup be better served by a reverse proxy like Nginx or Apache? I should specify here that we are currently looking at only having one domain to access this host.
I guess the question boils down to, how do I restrict remote access on a per-container level when running multiple client containers on a single host?
Any help is much appreciated.
It is feasible but the solution is too big to contain in a typical Stack Overflow answer. This is part of the value that PaaS providers like dotCloud, Heroku, and others provide.
You can try to roll your own multi-tenant solution, maybe by starting with something like Deis, but they warn against it.
Security is hard, especially when you are trying to make things easy for your customers.
You may find this series in the dotCloud blog helpful:
Episode 1: Kernel Namespaces (docker helps)
Episode 2: cgroups (docker helps)
Episode 3: AUFS (docker helps)
Episode 4: GRSEC (your kernel, up to you)
Episode 5: Distributing routing (your network, up to you, though Docker Swarm may help eventually)
Episode 6: Memory Optimization (up to your users)
notable bias: I used to work for dotCloud and now Docker.
I've been runnning a single django application on Amazon EC2 using gunicorn to serve the django portion and nginx for the static files.
I'm going to be starting new project soon, and wondering which of the following options would be better:
A larger amazon EC2 instance (Medium) runnning multiple django applications
Multiple smallers EC2 instances (Small/Micro) all running their own django applications?
Would anybody have any experience with this? What would the relevant performance metrics I could measure to get a good cost to performance ratio?
The answer to this question really depends on your app I'm afraid. You need to benchmark to be sure you are running on the right instance type. Some key metrics to watch are:
CPU
Memory usage
Requests per second, per instance size
App startup time
You will also need to tweak nginx/gunicorn settings to make sure you are running with a configuration that is optimised for your instance size.
If costs are a factor for you, one interesting metric is "cost per ten thousand requests", i.e. how much are you paying per 10000 requests for each instance type?
I agree with Mike Ryan's answer. I would add that you also have to evaluate whether your app needs a separate database. Sometimes it makes sense to isolate large/complex applications with their own database, which makes changes and maintenance easier. (Also reduces your risk in the case that something goes wrong). Not all of your user base would be affected in the case of an outage. You might want to create a separate instance for these applications. Note: Django supports multiple databases in one project but, again, that increases complexity for changes and maintenance.