janusgraph encounters `PermanentBackendException` while interacting with AWS opensearch version 7 and above - amazon-web-services

Question

We are trying to run Janusgraph version 0.6.2 using AWS opensearch(elasticsearch) version 7.10 as the indexing backend. Things work fine with version 6.x but when we try to connect to version 7.x we encounter the following exception.
org.janusgraph.diskstorage.PermanentBackendException: method [PUT], host [https://vpc-xxxxxx.us-east-2.es.amazonaws.com:443], URI [/_cluster/settings], status line [HTTP/1.1 401 Unauthorized]
{"Message":"Your request: '/_cluster/settings' payload is not allowed."}
Janusgraph version info:
86 [main] INFO org.janusgraph.graphdb.server.JanusGraphServer - JanusGraph Version: 0.6.2
86 [main] INFO org.janusgraph.graphdb.server.JanusGraphServer - TinkerPop Version: 3.5.3
More detailed stack trace is below:
3115 [main] INFO org.janusgraph.diskstorage.Backend - Configuring index [search]
3387 [main] INFO com.newforma.janusgraph.es.awsauth.AWSV4AuthHttpClientConfigCallback - Initialized AWSV4AuthHttpClientConfigCallback for region us-east-2
3782 [main] WARN org.apache.tinkerpop.gremlin.server.util.DefaultGraphManager - Graph [graph] configured at [/etc/opt/janusgraph/janusgraph.properties] could not be instantiated and will not be available in Gremlin Server. GraphFactory message: GraphFactory could not instantiate this Graph implementation [class org.janusgraph.core.JanusGraphFactory]
java.lang.RuntimeException: GraphFactory could not instantiate this Graph implementation [class org.janusgraph.core.JanusGraphFactory]
at org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:84)
at org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:80)
... 14 more
Caused by: java.lang.IllegalArgumentException: Could not instantiate implementation: org.janusgraph.diskstorage.es.ElasticSearchIndex
at org.janusgraph.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:79)
at org.janusgraph.diskstorage.Backend.getImplementationClass(Backend.java:527)
at org.janusgraph.diskstorage.Backend.getIndexes(Backend.java:511)
at org.janusgraph.diskstorage.Backend.<init>(Backend.java:239)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:127)
... 19 more
Caused by: org.janusgraph.diskstorage.PermanentBackendException: method [PUT], host [https://vpc-xxxxxx.us-east-2.es.amazonaws.com:443], URI [/_cluster/settings], status line [HTTP/1.1 401 Unauthorized]
{"Message":"Your request: '/_cluster/settings' payload is not allowed."}
at org.janusgraph.diskstorage.es.ElasticSearchIndex.setupMaxOpenScrollContextsIfNeeded(ElasticSearchIndex.java:445)
at org.janusgraph.diskstorage.es.ElasticSearchIndex.<init>(ElasticSearchIndex.java:388)
... 32 more
Caused by: org.elasticsearch.client.ResponseException: method [PUT], host [https://vpc-xxxxxx.us-east-2.es.amazonaws.com:443], URI [/_cluster/settings], status line [HTTP/1.1 401 Unauthorized]
{"Message":"Your request: '/_cluster/settings' payload is not allowed."}
at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:326)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:296)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:270)
at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.performRequest(RestElasticSearchClient.java:482)
at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.performRequest(RestElasticSearchClient.java:473)
at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.updateClusterSettings(RestElasticSearchClient.java:269)
at org.janusgraph.diskstorage.es.ElasticSearchIndex.setupMaxOpenScrollContextsIfNeeded(ElasticSearchIndex.java:443)

Answer 1

From the stack trace it appears that janusgraph was trying to set a high value for the elasticsearch property max_open_scroll_context. It is 500 by default.
AWS opensearch(elasticsearch) 7.x onwards doesn't let us set cluster properties.
Tried the following from kibana and I was able to get a similar response. This operation was supported in AWS managed elasticsearch 6.x version.
PUT _cluster/settings
{
"persistent" : {
"search.max_open_scroll_context": 1024
},
"transient": {
"search.max_open_scroll_context": 1024
}
}
401 - Unauthorized
{"Message":"Your request: '/_cluster/settings' payload is not allowed."}
We can disable setting max_open_scroll_context property while janugraph starts by setting the property index.[x].elasticsearch.setup-max-open-scroll-contexts to false.
You can read more on this in configuration reference section on elasticsearch https://docs.janusgraph.org/configs/configuration-reference/#indexxelasticsearch