I'm building a Sails app that is using socket.io and see that Sails offers a method for using multiple servers via redis:
http://sailsjs.org/documentation/concepts/realtime/multi-server-environments
Since I will be placing the app on AWS, preferably with ELB (elastic load balancer) and autoscale group with multiple EC2 instances was wondering how I can handle so it doesn't need a separate redis instance?
Maybe we can use AWS Elasticache? If so - how would this be done?
Now that AWS has released the new ALB application load balancer which has websockets, could this be used to help simplify things?
Thanks in advance
Updates for use-cases in application
Allow end-user to update data dynamically from their own dashboard
and display analytics/stats in real-time to an administrator
Application status' to change based on specific timings eg. at a
given start date/time the app allows users to update data.
Regarding your first question, you don't want to run Redis on the same servers that Sails is running on, especially if you are using AutoScaling. The Redis server needs to be a separate server that won't disappear if your environment experiences a "scale-in" event. So Redis is going to have to be on a separate "server" somewhere.
ElastiCache is just separate EC2 instances, running Redis, where AWS handles most of the management for you to the point that you can't even SSH into the instance. It's similar to how RDS works. ElastiCache will certainly work for your scenario. You might also want to look at the third-party service RedisLabs which also manages Redis instances on AWS for you.
Regarding your second question, an Application Load Balancer will have no bearing on your Redis usage. It will however bring actual support for WebSockets which it sounds like you are using. So yes, you should be using an ALB instead of an ELB.
Related
I'm building a chat app in React Native, and the backend is in Node.JS I'm using GKE to deploy the server code.
I'm using a cloud sql postgresql, connecting with internal IP. This works. I also use a memorystore (redis). Here is the problem.
For autoscaling, I'm planning to get multiple GKE clusters in different regions (for now, europe-west1 and us-central1). I have configured a load balancer with one backend containing all instance groups. I don't know of this is the correct/ideal solution, but it works. The problem lies in the fact that you can only connect to a redis database from an instance within the same region. If i use use-central1 as the region for my memorystore instance, I cannot connect to it through the vm's in the eu-cluster I created.
What is the best solution to overcome this problem? I've created an extra VM in the same region as the redis instance with haproxy configured to use as a reverse proxy to the memorystore, and this way, I can connect to the redis database through all instances, no matter what region they're in. But I don't know if this is the correct solution?
EDIT:
I'm using websockets (socket.io) for chat messages. Because I'm planning to use multiple servers, I need a centralized database to store (references to) the socket ID's, so users can send messages to users that are connected to other servers.
I'm thinking redis is the correct solution for a number of reasons:
I can use socket.io-redis to store the socket ID's on redis
fast response time
I don't know about the size of the data stored, but it's definitely not Mb's
I'm using a postgresql database to store other information (like username, passwords), but it seems to me that redis is a far better solution for real time applications.
AWS seems a little daunting with too many overlapping services so I'm looking for some advice and direction.
We have a mobile app for which we've developed a sync server (i.e. user will sign-up, sync data kept on AWS). Currently we've setup an EC2 instance with a web server, Django end-points and a postgres server. However we need the following:
Ensure the service is available from different regions of the
world for faster access
If that requires putting the postgres server outside of the EC2, what service do we need and how would replication work?
We will have larger file attachments stored on S3 separately, but need to do this securely and encrypt the files
Eventually we will host a web-app (i.e. an Angular 2 app) that would
connect to the same database.
We also would need to do all this in the most economical way and then scale up as the load increases.
Please any guidance would be appreciated. I'm struggling with terminologies at the moment. We also setup an Amazon SSL Certificate however that requires an Elastic Load Balancer but we only have one EC2 instance. What do we do to get this all working securely?
Based on the information provided, I would recommend you to start with AWS Elastic Beanstalk, where it will manage autoscaling and loadbalancing while providing you with a DNS URL for external domain mapping.
To ensure that the service is available from different regions for faster access, you can cache the static Angular App using Cloudfront. Then you will be able to add SSL Certificate to Cloudfront instead of ELB. If you plan to create multiple environments for different regions, you can use Route53 for geo based routing.
To take Postgres server outside EC2, you can use AWS RDS and it supports synchronous replication with fail-over for Multi-AZ deployments and also Postgres in RDS also supports Cross Region Replication if you plan to setup multiple deployment environments in different regions. Also you can create Read Replicas to improve reading speeds which will be asynchronously replicated.
You can encrypt the files in S3 using AES256 using Keys from KMS or from your client and I would recommend using Signed URLs with Cloudfront in front of S3 serving these files, so that clients can securely and directly access them improving the performance by getting advantage from distributed caching.
You can host the Angular App in AWS S3 and Cache using Cloudfront for faster access. Another option is to cache the static asset path in Cloudfront so that subsequent requests for static assets will be served from Cloudfront.
FAQs from Amazon
Who should use AWS Elastic Beanstalk?
Those who want to deploy and manage their applications within minutes
in the AWS Cloud. You don’t need experience with cloud computing to
get started. AWS Elastic Beanstalk supports Java, .NET, PHP, Node.js,
Python, Ruby, Go, and Docker web applications.
Your current environment isn't scalable (either load-responsive or to another region). If you need scalability then it should be re-arranged. It is difficult to provide you with details because the required environment depends on the applications architecture, however there are some suggestions:
DB: For better stability multi-AZ RDS setup for the DB is recommended. Benefit is RDS is fully managed service so you don't need to worry about replication, maintenance etc.
Web/app servers: you can deploy a copy in any region you want and connect to the same DB.
S3: you can enable crosss-region replication as well as encryption, but make sure it is used wisely (e.g. files are served to the client from bucket in closest region)
You can set up your own SSL on the server and it does not require ELB. However, you can use ELB with one webnode only.
I do NOT suggest to use Beanstalk because despite it really makes the first steps more easier you may have trouble trying to configure something non-standard in the future (unless you're very well familiar with EBT, of course).
To add efficiency you may want to add CDN (either AWS ot another vendor).
Make sure your environment configuration is really secure. You may need for your team someone who is familiar with AWS because every topic can be converted to a separate article.
I am new to Amazon Web Services. I was reading about Amazon ElastiCache and wanted to clarify if it is like (may be more than that) using RAM filesystem in Linux where we use a portion of system memory as a file system. As I referred AWS documentation it says ElastiCache is a web service. Is it like an EC2 instance with few memory modules attached? I really want to understand how it exactly works.
Our company has decided to migrate our physical servers into AWS cloud. We use Apache web server and MySQL Database running in Linux. We provide a SaaS platform for e-mail marketing and event scheduling for our customers. There is usually a high web traffic to our website during 9am-5pm on weekdays. I would like to understand if we want to use ElastiCache service, how it will be configured in AWS.? We have planned two EC2 instances for our web server and an RDS instance for the database.
Thanks.
Elastic cache is simply managed Redis or Memcached. Depending which one you choose, you would use the client for the cache with your application.
How you would implement it depends on what kind of caching you are trying to accomplish.
I have an application on an Windows server EC2 with an SQL server for our database.
What I would like to do is an load balancer so the application won't fail due to overload.
I have a couple of questions that Im not certain about it.
I believe that i need to create an image of my current instance and duplicate it. my problem is that my database is based on my current instance so it would duplicate my database as well.
Do I need another instance just for my database?
If yes, then it means that I need a total of 3 instances. 2 for the application and 1 for the database.
In this case I need to change my application to connect to the new instance database instead of the current database.
After all that happens I need to add a load balancer.
I hope I made myself clear.
I would recommend using RDS (http://aws.amazon.com/rds/) for this. This way you don't have to worry about the database server and just host your application server on EC2 instances. Your AMI will then contain just the application server and thus when you scale up you will be launching additional app servers only and not database servers.
Since you are deploying a .NET application, I would also recommend taking a look at Elastic Beanstalk (http://aws.amazon.com/elasticbeanstalk/) since it will really help to make auto scaling much easier and your solution will scale up/down as well as self-heal itself.
As far the load balancer is concerned, you can either manually update your load balancer will the new instances of your application server or you can let your auto scale script do it for you. If you go for ElasticBeanstalk, then Elastic Beanstalk will take care of adding/removing instances to/from your Elastic Load Balancer for you on its own.
As I understand I can use EC2 as web server for my application. But how load balancer is working?
For example I have one EC2 instance. In this way load balancer will not work. Am I right?
For example I have few EC2 instances. In this way I can configure load balancer to balance between all my EC2 servers.
Am I right?
Application files at all instances should be synced? Is there is any Amazon tool to sync? Or I should use something like rsync or post commit hooks to sync files between EC2 instances?
Is it possible to use one EC2 instance for web application (php + nginx) and for CDN (Cloud Front)? Or what is the better way to reach this: I need to store static files but I should access them from web application (php scripts) through file system. So I am going to use EC2 and Clod Front. But how can I get access?
Thanks for your time.
Technically, the load belancer will work, it's just that it'll only balance the traffic to one instance.
Correct. You register the instances with the elastic load balancer, and whilst those instances are healthy - it will respond to them.
There's many different ways to sync files - it all depends on what you want to sync. Cloud Architecture is a little different to traditional architecture. For example, rather than loading the images onto the EBS volume, you'd try and offload them (and serve them) from S3. Therefore the only things you'd need to "sync" would be the webserver files themselves. You could use CloudFormation to roll out updates, post commit hooks and rsync are also good options. The challenge is to remember that it can scale / fail almost at will - so you need to ensure that each instance knows how to get the information and keep itself updated in isolation.
Yes. It's called a custom origin. What you want to do though is put a url rewrite on the outbound server that rewrites the local urls to cloudfront domains.
Hope that helps