We need to add a coordinating node on AWS Elasticsearch but having trouble figuring out how. Is it possible at all? How to do that?
Unfortunately, AWS Elasticsearch does not support "coordinating only" nodes as of now.
By default, every data node that you add in the cluster acts as a coordinating node. All the incoming requests are first received by the load balancer and then served by one of the "acting" coordinating node (changes for every request). This coordinating node is selected from the available data nodes in a round robin fashion so that the load distributed equally amount the configured number of data nodes.
I'm trying to change my AWS ElasticSearch nodes to lower configuration. But when I do this, it adds more nodes to the original cluster. I can't delete them. And I search on Google, everybody says I have to delete the domain. But this may let me lose all indexes I built on it, right?
Is there any way that I can delete some nodes but remain the others?
You should be able to simply change the number of nodes in your ElasticSearch domain (Note: your data must fit into smaller amount of nodes)
The number of nodes visible in CloudWatch metrics can be misleading and AWS points this fact in ElasticSearch documentation:
Note
The Nodes metric is not accurate during changes to your cluster
configuration and during routine maintenance for the service. This is
expected behavior. The metric will report the correct number of
cluster nodes soon.
I actually tried to simulate what you want to do.
I created ES domain with 4 nodes
Once the domain was created, I changed the number of nodes to 2
When looking at nodes count metric, I could see that temporarily the amount of nodes in the ES domain was 6, but eventually went down to 2.
In case when number of nodes does not stabilize for a long time, I recommend contacting AWS.
I am pretty new to ElasticSearch and I am trying to use it to index document content for our users. The document content will be extracted using Apache Tika along with the file meta data and relative information (size, date, extension, etc.) and will all be stored and indexed in ElasticSearch.
The problem is that I cannot tell before hand how much data I will be indexing as this will be an added service for new clients.
What I am thinking of is to start with a single node in my cluster which will have 1000 shards and 1 replica (2000 shards in total). Each client will have his own index, this means that this node will only support 1000 clients which should be sufficient since we do not have many clients to begin with. Once the node is full we will expand the cluster and add a new node and this will extend the cluster to support 2000 clients and so on.
My concern is with the storage. Since I'll be indexing large data sets the stored data will expand rapidly in size.
For the sake of argument lets assume that I will attach a 1GB volume to my node and lets assume that I cannot expand it beyond that. Now, If I add a new node to the cluster, how will elasticsearch behave assuming the first node has already reached its storage limit (say it is now using 999MB).
If lets say client A has a new document to index and the document size is say 5MB. how will Elasticsearch behave to this? Will it move the index to the new node? or does it keep the index in the original and marks the new index request as a failure?
The reason I am asking this is because I will be hosting my ElasticSearch cluster on Amazon EC2 with an EBS volume attached to it, and since Amazon charge per provisioned GB for EBS it would be wise to start small and expand the volume when needed, so we do not have to incur massive cost at the start of the project.
For your purposes, you need to worry about shards; 1000 shards per index is probably insane overkill. Each index (not each node!) consists of some number of shards and their replicas. Elasticsearch will allocate shards on nodes in your cluster automatically in an attempt to balance their distribution throughout the cluster. When disk usage reaches a configured high watermark on a node, Elasticsearch will stop allocating shards to that node. Shards can be transparently migrated to different nodes.
You can upgrade the total storage capacity of your cluster by attaching a new node with more storage. You should take care that your shard size remains small enough that it can live on a single node, but if it's too small then the overhead of managing so many shards may become prohibitive. Keep in mind that ES can query across multiple indices; a common pattern for high-volume chronological writes is to create a new index every N days, and when data is required, you run a query across all of the indexes that hold your dataset. This lets you control the size of individual indexes while accommodating significant scaling needs.
I have a cluster of 3 ElasticSearch nodes running on AWS EC2. These nodes are setup using OpsWorks/Chef. My intent is to design this cluster to be very resilient and elastic (nodes can come in and out when needed).
From everything I've read about ElasticSearch, it seems like no one recommends putting a load balancer in front of the cluster; instead, it seems like the recommendation is to do one of two things:
Point your client at the URL/IP of one node, let ES do the load balancing for you and hope that node never goes down.
Hard-code the URLs/IPs of ALL your nodes into your client app and have the app handle the failover logic.
My background is mostly in web farms where it's just common sense to create a huge pool of autonomous web servers, throw an ELB in front of them and let the load balancer decide what nodes are alive or dead. Why does ES not seem to support this same architecture?
I believe load balancing an Elasticsearch cluster is a good idea (designing a fault tolerant system, resilient to single node failure.)
To architect your cluster you'll need background on the two primary functions of Elasticsearch: 1. Writing and updating documents and 2. Querying Documents.
Writing / indexing documents in elasticsearch:
When a new document comes into Elasticsearch to be indexed, Elasticsearch determines the "primary shard" the document should be assigned to using the "Shard Routing Algorithm"
The Lucene process associated with the shard "maps" the fields in the document;
The Lucene process adds the document to the shard's Lucene "inverted index"
Any "replica shard(s)" then receive the document; the replica shard "maps" the document and adds the document to the replica shard's Lucene "inverted index"
Querying documents in Elasticsearch:
By default, when a query is sent to Elasticsearch, the query hits a node -- this becomes the "query node" or the "gateway query node" for that query
The node broadcasts the query to every shard in the index (primary & replica)
each shard performs query on the shard's local Lucene inverted index.
each shard returns the top 10 - 20 results to the "gateway query node"
the "gateway query node" then performs a merge-sort on the combined results returned from the other shards,
once the merge-sort is finished, the "gateway query node" and returns results to the client
the merge-sort is CPU and Memory resource heavy
Architect a Load Balancer for Writes / Indexing / Updates
Elasticsearch self manages the location of shards on nodes. The "master node" keeps and updates the "shard routing table". The "master node" provides a copy of the shard routing table to other nodes in the cluster.
Generally, you don't want your master node doing much more than health checks for the cluster and updating routing tables, and managing shards.
It's probably best to point the load balancer for writes to the "data nodes" (Data nodes are nodes that contain data = shards) and let the data nodes use their shard routing tables to get the writes to the correct shards.
Architecting for Queries
Elasticsearch has created a special node type: "client node", which contains "no data", and cannot become a "master node". The client node's function is to perform the final resource heavy merge-sort at the end of the query.
For AWS you'd probably use a c3 or c4 instance type as a "client node"
Best practice is to point the load balancer for queries to client nodes.
Cheers!
References:
Elasticsearch Node Types
Elasticsearch: Shard Routing Algorithm
Elasticsearch: Replica Shards
Elasticsearch: Cluster State i.e. the Shard Routing Table
ElasticHQ - Introduction to Elasticsearch Video
Elasticsearch: Shard numbers and Cluster Scaling
You don't need a load balancer — ES is already providing that functionality. You'd just another component, which could misbehave and which would add an unnecessary network hop.
ES will shard your data (by default into 5 shards), which it will try to evenly distribute among your instances. In your case 2 instances should have 2 shards and 1 just one, but you might want to change the shards to 6 for an equal distribution.
By default replication is set to "number_of_replicas":1, so one replica of each shard. Assuming you are using 6 shards, it could look something like this (R is a replicated shard):
node0: 1, 4, R3, R6
node1: 2, 6, R1, R5
node2: 3, 5, R2, R4
Assuming node1 dies, the cluster would change to the following setup:
node0: 1, 4, 6, R3 + new replicas R5, R2
node2: 3, 5, 2, R4 + new replicas R1, R6
Depending on your connection setting, you can either connect to one instance (transport client) or you could join the cluster (node client). With the node client you'll avoid double hops, since you'll always connect to the correct shard / index. With the transport client, your requests will be routed to the correct instance.
So there's nothing to load balance for yourself, you'd just add overhead. The auto-clustering is probably ES's greatest strength.
You're quite right to want to design for 'failover', and in AWS, here's how I recommend you do it.
1) Limit the nodes in your cluster that can be elected master. For the rest, set node.client: true. Base your choice of how many master electable nodes you have on how many you want available for failover.
2) Create an ELB that includes only the master electable nodes.
3) In Route 53, create a CNAME for your cluster, with the value set to the DNS name of your ELB.
It is a little unclear what to do when the master node fails. After digging, here are a couple of options:
If you re using a client to connect to elasticsearch cluster, you can specify an array of DNS names in the connection configuration. The client will try to resolve DNS in sequence. If the first one fails, the second one will be attempted.
Use round-robin DNS. Similar to the option number one, but without the client.
Both of these options might cause a delay because DNS needs to fail first to be routed to the next available alias. DNS or client have no way of knowing that a particular host is down.
Use some sort of load balancer, that way LB will know about misbehaving hosts ahead of time and not cause any delays.
I have a RedShift cluster of 4 nodes.
When one of the nodes goes down, will the entire cluster become unavailable?
If yes - for how long?
When the cluster gets back - is it returned to exactly the same point it was before the failure, or the data may be rolled back a to S3 snapshot from a few hours ago?
How can I simulate this situation to check this scenario by myself?
Thanks a lot!
If it's a single node failure - amazon will start a new node and stream data from other nodes (each block is written to two different nodes if any).
In such case, we can expect:
Downtime of the entire cluster till a new node starts up + filled with the DB information. Should be about 3-4 minutes.
After these 3-4 minutes that cluster will return to exactly the same point it was before it went down. The cluster will be available to both reads and writes.
Some slowdown will be experienced due to data redistribution in the cluster.
In case more than one nodes fails, redshift will restore itself from the latest S3 backup.
S3 backups are done on the following occasions:
If it's been 8 hours since the last backup
If RedShift was filled with more then 5GB of data since the last backup
Manually
You have the option of a final snapshot when you chose to terminate your cluster
It just happened to my cluster - one of nodes failed. It took almost 20 minutes to get noticed in the dashboard (unhealthy was shown in 'Performance' tab, but healthy in 'Status' tab).
After 1h from initial failure, cluster changed its state to 'modifying' and after another 1h a new node was in place.
There is a message in 'Recent Events':
A node on Amazon Redshift cluster 'xxx' was automatically replaced at 2013-12-18 11:42 UTC. The cluster is now operating normally.
For the whole time cluster was unavailable - no queries were run, no imports were possible.
Data is exactly the same as in the moment of a failure.