How to manually setup AWS ElasticSearch and use kibana through browser? - amazon-web-services

Question

I've been struggling on setting up AWS ES and use kibana. I was following Amazon Elasticsearch Service docs on AWS.
When I get to Step 2: Upload Data to an Amazon ES Domain for Indexing, I was failed on running curl -XPUT elasticsearch_domain_endpoint/movies/_doc/1 -d '{"director": "Burton, Tim", "genre": ["Comedy","Sci-Fi"], "year": 1996, "actor": ["Jack Nicholson","Pierce Brosnan","Sarah Jessica Parker"], "title": "Mars Attacks!"}' -H 'Content-Type: application/json' as the docs indicated, getting error of {"Message":"User: anonymous is not authorized to perform: es:ESHttpPut"}.
I've set the policy on ES as:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::my_id:user/my_iam_user"
},
"Action": "es:*",
"Resource": "arn:aws:es:us-west-2:my_id:domain/my-domain/*"
},
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"AWS": "*"
},
"Action": "es:*",
"Resource": "arn:aws:es:us-west-2:my_id:domain/my-domain/*",
"Condition": {
"IpAddress": {
"aws:SourceIp": [my_ips]
}
}
}
]
}
I got the IPs above by calling ifconfig | grep "inet " | grep -v 127.0.0.1 from terminal, hitting checkip.amazonaws.com, and checking Developer Tools -> Network on chrome (those are 3 different IPs and I added them all).
I've also added following role on my IAM user:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"es:DescribeReservedElasticsearchInstanceOfferings",
"es:DescribeReservedElasticsearchInstances",
"es:ListDomainNames",
"es:PurchaseReservedElasticsearchInstance",
"es:DeleteElasticsearchServiceRole",
"es:ListElasticsearchInstanceTypes",
"es:DescribeElasticsearchInstanceTypeLimits",
"es:ListElasticsearchVersions"
],
"Resource": "*"
},
{
"Sid": "VisualEditor1",
"Effect": "Allow",
"Action": "es:*",
"Resource": "arn:aws:es:us-west-2:my_id:domain/my-domain"
}
]
}
I've setup AWS CLI on my machine and I'm able to get the correct result through aws es describe-elasticsearch-domain --domain my-domain.
Still, I failed to call curl XPUT above and failed to access kibana for the same reason {"Message":"User: anonymous is not authorized to perform: es:ESHttpPut"}
Here's couples of article I read before I raised the question here:
https://aws.amazon.com/premiumsupport/knowledge-center/anonymous-not-authorized-elasticsearch/?nc1=h_ls
https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-ac.html
https://aws.amazon.com/cn/blogs/database/set-access-control-for-amazon-elasticsearch-service/
Proper access policy for Amazon Elastic Search Cluster
https://aws.amazon.com/cn/blogs/security/how-to-control-access-to-your-amazon-elasticsearch-service-domain/
And still couldn't get it work.
Can anyone kindly guide me through the whole process of setting up the AWS ES manually, and being able to manipulate it through AWS CLI as well as kibana on browser? I would be really appreciated if it could be a detailed step-by-step guide instead of throwing aws docs. Thank you so much.

Answer 1

It turns out that the IP I used was incorrect. I should have called checkip.amazonaws.com while not under VPN, and the IP may change down the line. The policy on ES should looks like: Anther Answer
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::xxxxxxxxxxxx:root"
},
"Action": "es:*",
"Resource": "arn:aws:es:us-west-2:xxxxxxxxxxxx:domain/my-elasticsearch-domain/*"
},
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"AWS": "*"
},
"Action": "es:*",
"Resource": "arn:aws:es:us-west-2:xxxxxxxxxxxx:domain/my-elasticsearch-domain/*",
"Condition": {
"IpAddress": {
"aws:SourceIp": [
"192.168.1.0",
"192.168.1.1"
]
}
}
}
]
}
*PS. Make sure you are calling checkip.amazonaws.com on the same browser (same user as well if you are using chrome)