Elasticsearch on AWS: How to fix unassigned shards? - amazon-web-services

Question

I have an index on AWS Elasticsearch which were unassighed due to NODE_LEFT. Here's an output of _cat/shards
rawindex-2017.07.04 1 p STARTED
rawindex-2017.07.04 3 p UNASSIGNED NODE_LEFT
rawindex-2017.07.04 2 p STARTED
rawindex-2017.07.04 4 p STARTED
rawindex-2017.07.04 0 p STARTED
under normal circumstances, it would be easy to reassign these shards by using the _cluster or _settings. However, these are the exact APIs that are not allowed by AWS. I get the following message:
{
Message: "Your request: '/_settings' is not allowed."
}
According to an answer to a very similar question, I can change the setting of an index using _index API, which is allowed by AWS. However, it seems like index.routing.allocation.disable_allocation is not valid for Elasticsearch 5.x, which I am running. I get the following error:
{
"error": {
"root_cause": [
{
"type": "remote_transport_exception",
"reason": "[enweggf][x.x.x.x:9300][indices:admin/settings/update]"
}
],
"type": "illegal_argument_exception",
"reason": "unknown setting [index.routing.allocation.disable_allocation] please check that any required plugins are installed, or check the breaking changes documentation for removed settings"
},
"status": 400
}
I tried prioritizing index recovery with high index.priority as well as setting index.unassigned.node_left.delayed_timeout to 1 minute, but I am just not being able to reassign them.
Is there any way (dirty or elegant) to achieve this on AWS managed ES?
Thanks!

Answer 1

I had a similar issue with AWS Elasticsearch version 6.3, namely 2 shards failed to be assigned, and the cluster had status RED. Running GET _cluster/allocation/explain showed that the reason was that they had exceeded the default maximum allocation retries of 5.
Running the query GET <my-index-name>/_settings revealed the few settings that can be changed per index. Note that all queries are in Kibana format which you have out of the box if you are using AWS Elasticsearch service. The following solved my problem:
PUT <my-index-name>/_settings
{
"index.allocation.max_retries": 6
}
Running GET _cluster/allocation/explain immediately afterwards returned an error with the following: "reason": "unable to find any unassigned shards to explain...", and after some time the problem was resolved.

Answer 2

There might be an alternative solution when the other solutions fail. If you have a managed Elasticsearch Instance on AWS the chances are high that you can "just" restore a snapshot.
Check for failed indexes.
You can use for e.g.:
curl -X GET "https://<es-endpoint>/_cat/shards"
or
curl -X GET "https://<es-endpoint>/_cluster/allocation/explain"
Check for snapshots.
To find snapshot repositories execute the following query:
curl -X GET "https://<es-endpoint>/_snapshot?pretty"
Next let's have a look at all the snapshots in the cs-automated repository:
curl -X GET "https://<es-endpoint>/_snapshot/cs-automated/_all?pretty"
Find a snapshot where failures: [ ] is empty or the index you want to restore is NOT in a failed state. Then delete the index you want to restore:
curl -XDELETE 'https://<es-endpoint>/<index-name>'
... and restore the deleted index like this:
curl -XPOST 'https://<es-endpoint>/_snapshot/cs-automated/<snapshot-name>/_restore' -d '{"indices": "<index-name>"}' -H 'Content-Type: application/json'
There is also some good documentation here:
https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-managedomains-snapshots.html#es-managedomains-snapshot-restore
https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/aes-handling-errors.html#aes-handling-errors-red-cluster-status

Answer 3

I also faced a similar problem. The solution is pretty simple. You can solve it in 2 different ways.
First solution is to edit all indexes collectively:
PUT _all/_settings
{
"index.allocation.max_retries": 3
}
Second solution is to edit specific indexes:
PUT <myIndex>/_settings
{
"index.allocation.max_retries": 3
}