I am running a storm cluster on AWS. But I want the storm cluster to expand automatically when the need comes. I figured out mesos is something like that. But I do not have much knowledge about mesos and its deployment on AWS.
Can mesos on AWS automatically increase the parallelism of my topology tasks by launching new instances and shutting them down when not necessary? If it can, how do we configure mesos for the same.
Mesos does not directly handle autoscaling itself, but allows frameworks running on top of it to receive new resource offers and react to them by launching new task instances. I haven't used it personally, but you could try the Storm-Mesos framework for running Storm on Mesos: https://github.com/mesos/storm
Once you have Storm running on Mesos, ready to launch new instances as resources come available, you're ready to autoscale within the existing cluster's capacity. You'll probably want to take advantage of Amazon's Auto-Scaling Groups (ASGs) to scale up the number of Mesos nodes based on your need. As the ASG scales up more Mesos nodes, the resources from those nodes will be automatically offered to the Storm-Mesos framework, which can launch more Storm instances.
Yes, you're heading in the right direction. However I'd suggest to use Marathon rather than the low-level Mesos API.
See for example the GitHub repo obaidsalikeen/storm-marathon, which is particularly well done in terms of completeness and documentation richness.
Related
I have a service that needs to run on our own EC2 instances, since it requires some support from the kernel. My previous experience is all with containers in AWS. The application itself is distributed as a single JAR file and I'm looking for advice for how I should automate deployments. The architecture is:
An ALB in front of the ASG.
EC2 instance running a single Java application.
Any open sockets are open for an hour tops and to not cause any trouble, we have to drain the connections to the EC2 instances before performing an update, so a hard requirement is for the ALB to stop opening new connections for an hour before updating the software. The application is mission critical and ECS has had some issues last year, so I want to minimize the AWS services I depend on. While I could do what I want on my own ECS cluster with custom AMIs, I don't want to do it, since I will run a single instance of the app per host and don't need the extra layer.
My question: What is the simplest method to achieve this using CodePipeline? My understanding is that I need to use a CodeDeploy deployment step to push something to bare EC2 instances. How does draining with an ALB work in this case? We're using CloudFormation for the deployment.
You need to use codedeploy. You can find tutorial on AWS codedeploy documentation.
Codedeploy deployment lifecycle hooks for EC2.
https://docs.aws.amazon.com/codedeploy/latest/userguide/reference-appspec-file-structure-hooks.html#appspec-hooks-server
For multiple high-volume .NET web applications running on Windows and IIS, is there any advantage to changing to CloudFormation and ECS or even EKS instead of just using an AMI of an instance that already has auto-run scripts to update the codebase and configuration on launch, tied into an auto-scaling group behind an ELB?
Our usage is to scale web application servers based on load. Unless I have missed information or grossly misunderstood what has been read, it seems we might not gain anything by moving away from the pure AMI and auto-scaling group.
#sivart Are you using containers? ECS and EKS are container orchestrators. These will benefit you if you package your web applications as docker containers.
CloudFormation helps stand up/down your infrastructure stack using IaC (infrastructure as code) and will help you spin up new infrastructures within minutes without clicking through the AWS console. This will help, for example, when you want to produce production comparable stack for you performance/load/QA before going live. You can create a stack, perform tests and destroy it once happy. You can integrate CloudFormation with your CI/CD pipelines.
I am newbie in AWS and totally confused about the deploy. here i have
React for front-end , Nodejs for API, Mongodb for database and redis for session store.
Can i use 1 EC2 for every service ? or
Divide every service as different EC2
Can i use Elastic Beanstalk Environment?
Which is better option for scaling and update without downtime in future ?
Can I use 1 EC2 for every service?
Its depend on your case but the best approach is to utilize the underlying EC2 instance is to run multiple services on single EC2 for nodejs and front-end app, as nodejs container-based application take maximum advantage in this case. in this case, ECS blue-green deployment with the dynamic port of the container can help to scale with zero downtime.
Divide every service as different EC2
In nodejs based application this approach does not help you a lot where for Redis and mongo it make sense if you are planning for clustering and replica also these applications need persistent storage so will keep storage on each instance, so my suggestion is to keep redis and mongo DB in daemon mode and application in replica mode, as these are application that will do blue-green deployment not the redis or Db.
AWS provides two types of task to deal with such cases
REPLICA—
The replica scheduling strategy places and maintains the desired
number of tasks across your cluster. By default, the service scheduler
spreads tasks across Availability Zones. You can use task placement
strategies and constraints to customize task placement decisions. For
more information, see Replica.
DAEMON—
The daemon scheduling strategy deploys exactly one task on each active
container instance that meets all of the task placement constraints
that you specify in your cluster. When using this strategy, there is
no need to specify a desired number of tasks, a task placement
strategy, or use Service Auto Scaling policies. For more information
ecs_services
I have multiple microservices that I have to run independently. I am thinking of deploying them in docker containers in one ec2 instances. But then there is the question of scaling. I knowledge is ecs gives me the ability to scale. I haven't used ecs though. So my question is can I scale all my containers by creating one ec2 networks ? Or is there anything I haven't thought or know about ? Also, what is the performance issues with this deployment ?
Thanks
Amit
For the microservices deployment, one ec2 instance will never suffice for production workload considering HA, scaling, performance etc.
You should think of cluster. A compute cluster is a multi-tenant computing environment consisting of servers (called “nodes”) whose resources have been pooled together and are used to execute processes. To enable this behavior, the nodes in a cluster must be managed by some sort of cluster management framework.
So you have to choose between multiple options from Kubernets, Mesos & Marathon and AWS ECS.
The EC2 Container Service is: A cluster management framework that uses optimistic, shared state scheduling to execute processes on EC2 instances using Docker containers.
The above option provides all the functionality you looking for. So you need to analyze more of them and select the most suitable option as per your need.
This might sound like a newbie question but I have a neo4j instance running on Amazon cloud. The instance is set to Autoscale at 80% usage. That means Amazon one the usage is reaches 80%, Amazon will create another instance on Neo4j with the same configuration and will keep adding more once this one reaches 80%..
My questions are -
1) Does this setup on Amazon means we have a cluster of neo4j in place?
2) Do I need to do anything else in order to have neo4j cluster, what I have read is that you need some tool like zookeeper to maintain the cluster..
3) Does this current setup on Amazon will have both instances as Master or will it be more like master/slave setup..
Any help, feedback, suggestions would be helpful.
Thanks in advance,
Ravi
Yes, if you are using Auto Scaling group for Neo4j you need to set a cluster. As #stefan-armbruster mentioned, you need to Neo4j Enterprise edition for that. In that case it's master/slave setup.
Neo4j has its own solution for Cluster management, instead of Zookeeper.
But with AWS and EC2 there are few open question, how to properly deploy Neo4j with Auto Scaling group.
From configuration file perspective
* You need to maintain unique clusterId for each machine in cluster.
* You need to know ip addresses/hostnames of other machines in cluster.
Neo4j Enterprise edition features clustering, see the docs on this. With some well written scripts around that to configure the new instances properly I don't see a reason why AWS autoscale should not work.