I am running into following issue while running elastic search exporter(prometheus) for VPC based AWS elasticsearch instance.
Appreciate if anyone knows what could could be the issue?
[elasticsearch-exporter-5558555bbf-blzhn elasticsearch-exporter] level=error ts=2020-09-02T15:56:20.134343455Z caller=clusterinfo.go:174 msg="failed to retrieve cluster info from ES" err="HTTP Request failed with code 403"
[elasticsearch-exporter-5558555bbf-blzhn elasticsearch-exporter] level=debug ts=2020-09-02T15:56:20.13437932Z caller=clusterinfo.go:120 msg="updating cluster info metrics"
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": [
"arn:aws:iam::awsaccount:role/role1”,
"arn:aws:iam::awsaccount:role/role2"
]
},
"Action": [
"es:ESHttpHead",
"es:ListDomainNames",
"es:DescribeElasticsearchDomain",
"es:ESHttpPost",
"es:ESHttpGet",
"es:ESHttpPatch",
"es:DescribeElasticsearchDomains",
"es:ESHttpDelete",
"es:ESHttpPut"
],
"Resource": "arn:aws:es:us-east-1:awsaccount:domain/domain/*”
}
]
}
Related
I have .ova VM's stored on my S3 bucket, I am trying to create AMI from these OVA.
I was going through this video to Import a VM as an Image Using VM Import/Export to Amazon EC2.
I have created an EC2 Instance which I will use to trigger the necessary CLI commands for Importing.
I have created an IAM Role and attached it to the EC2 Instance.
Please refer to the details of the Role:
Trust Policy
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "vmie.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
Inline Policy for Access to S3 and EC2
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"ec2:CopySnapshot",
"s3:ListAccessPointsForObjectLambda",
"s3:GetAccessPoint",
"s3:PutAccountPublicAccessBlock",
"s3:ListAccessPoints",
"ec2:RegisterImage",
"s3:ListJobs",
"s3:PutStorageLensConfiguration",
"s3:ListMultiRegionAccessPoints",
"s3:ListStorageLensConfigurations",
"ec2:Describe*",
"s3:GetAccountPublicAccessBlock",
"ec2:ModifySnapshotAttribute",
"s3:ListAllMyBuckets",
"s3:PutAccessPointPublicAccessBlock",
"s3:CreateJob",
"ec2:ImportImage"
],
"Resource": "*"
},
{
"Sid": "VisualEditor1",
"Effect": "Allow",
"Action": "s3:*",
"Resource": "arn:aws:s3:::vms"
},
{
"Sid": "AllowStsDecode",
"Effect": "Allow",
"Action": "sts:DecodeAuthorizationMessage",
"Resource": "*"
}
]
}
Inline Policy for KMS Decrypt
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": "kms:Decrypt",
"Resource": "*"
}
]
}
Also, I have attached the AWSImportExportFullAccess managed policy to the Role.
I am using the following command to Import the VM to AMI:
aws ec2 import-image --description "MY_VM_Image" --disk-containers "file://configuration.json"
Here are the contents of configuration.json
[{
"Description": "Image",
"Format": "ova",
"UserBucket": {
"S3Bucket": "vm",
"S3Key": "xzt.ova"
}
}
]
But I am facing the following error:
An error occurred (InvalidParameter) when calling the ImportImage operation: The service role vmimport provided does not exist or does not have sufficient permissions
I tried to have a look at the Troubleshooting document. It states the following
This error can also occur if the user calling ImportImage has Decrypt permission but the vmimport role does not.
So, I have also disabled the default encryption at S3.
Still no luck.
What else permissions are needed to run the command successfully.
I was facing the same issue and it turned out to be an issue with the clock not being in sync with the NTP servers (it was around 6 minutes off). As soon as the time was synced, the aws ec2 import-image worked as expected.
Here is a link for the importance of Time Synchronization in Kerberos:
https://tldp.org/HOWTO/Kerberos-Infrastructure-HOWTO/time-sync.html#:~:text=If%20you%20allow%20your%20clocks,errors%20and%20refuse%20to%20function.
I'm trying to setup AWS Elasticsearch but I can't get past the policy settings.
This is the policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": [
"arn:aws:iam::999999999999:user/xxxxx"
]
},
"Action": [
"es:*"
],
"Resource": "arn:aws:es:eu-central-1:99999999999:domain/xyzdomain/*"
}
]
}
this is the error:
CreateElasticsearchDomain: {"message":"Error setting policy: [{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Principal\":{\"AWS\":[\"arn:aws:iam::9999999999:user/xxxxx\"]},\"Action\":[\"es:*\"],\"Resource\":\"arn:aws:es:eu-central-1:9999999999:domain/xyzdomain/*\"}]}]“}
What's wrong with that?
When I try to query AWS Keyspaces (managed Cassandra) from an AWS Lambda, I get this error:
{
"errorType": "AggregateException",
"errorMessage": "One or more errors occurred. (All hosts tried for query failed (tried 11.11.111.11:9142: UnauthorizedException 'User arn:aws:iam::111111111111:user/user-for-keyspaces has no permissions.'; 11.11.111.11:9142: UnauthorizedException 'User arn:aws:iam::111111111111:user/user-for-keyspaces has no permissions.'))",
"stackTrace": [
"at lambda_method(Closure , Stream , Stream , LambdaContextInternal )"
],
"cause": {
"errorType": "NoHostAvailableException",
...
But in the AWS console for Keyspaces, I don't see anywhere to adder permissions.
The user policy for user-for-keyspaces already has this attached:
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"cassandra:*"
],
"Effect": "Allow",
"Resource": "*"
}
]
}
How do I add permissions in AWS Keyspaces?
You should only require cassandra
{
"Statement": [
{
"Sid": "keyspaces-full-access",
"Principal": "*",
"Action": [
"cassandra:*"
],
"Effect": "Allow",
"Resource": "*"
}
]
}
Additionally, Amazon Keyspaces populates the system.peers table in your account with an entry for each availability zone where a VPC endpoint is available. To look up and store available interface VPC endpoints in the system.peers table, Amazon Keyspaces requires that you grant the IAM entity used to connect to Amazon Keyspaces access permissions to query your VPC for the endpoint and network interface information.
{
"Version":"2012-10-17",
"Statement":[
{
"Sid":"ListVPCEndpoints",
"Effect":"Allow",
"Action":[
"ec2:DescribeNetworkInterfaces",
"ec2:DescribeVpcEndpoints"
],
"Resource":"*"
}
]
}
Learn more about VPC endpoints here
The problem was actually nothing to do with the user in the error message, but the VPC endpoint I had created for Keyspaces.
The endpoint requires cassandra:* permissions to perform queries, e.g.
{
"Statement": [
{
"Sid": "keyspaces-full-access",
"Principal": "*",
"Action": [
"cassandra:*",
"keyspaces:*"
],
"Effect": "Allow",
"Resource": "*"
}
]
}
I've been struggling on setting up AWS ES and use kibana. I was following Amazon Elasticsearch Service docs on AWS.
When I get to Step 2: Upload Data to an Amazon ES Domain for Indexing, I was failed on running curl -XPUT elasticsearch_domain_endpoint/movies/_doc/1 -d '{"director": "Burton, Tim", "genre": ["Comedy","Sci-Fi"], "year": 1996, "actor": ["Jack Nicholson","Pierce Brosnan","Sarah Jessica Parker"], "title": "Mars Attacks!"}' -H 'Content-Type: application/json' as the docs indicated, getting error of {"Message":"User: anonymous is not authorized to perform: es:ESHttpPut"}.
I've set the policy on ES as:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::my_id:user/my_iam_user"
},
"Action": "es:*",
"Resource": "arn:aws:es:us-west-2:my_id:domain/my-domain/*"
},
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"AWS": "*"
},
"Action": "es:*",
"Resource": "arn:aws:es:us-west-2:my_id:domain/my-domain/*",
"Condition": {
"IpAddress": {
"aws:SourceIp": [my_ips]
}
}
}
]
}
I got the IPs above by calling ifconfig | grep "inet " | grep -v 127.0.0.1 from terminal, hitting checkip.amazonaws.com, and checking Developer Tools -> Network on chrome (those are 3 different IPs and I added them all).
I've also added following role on my IAM user:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"es:DescribeReservedElasticsearchInstanceOfferings",
"es:DescribeReservedElasticsearchInstances",
"es:ListDomainNames",
"es:PurchaseReservedElasticsearchInstance",
"es:DeleteElasticsearchServiceRole",
"es:ListElasticsearchInstanceTypes",
"es:DescribeElasticsearchInstanceTypeLimits",
"es:ListElasticsearchVersions"
],
"Resource": "*"
},
{
"Sid": "VisualEditor1",
"Effect": "Allow",
"Action": "es:*",
"Resource": "arn:aws:es:us-west-2:my_id:domain/my-domain"
}
]
}
I've setup AWS CLI on my machine and I'm able to get the correct result through aws es describe-elasticsearch-domain --domain my-domain.
Still, I failed to call curl XPUT above and failed to access kibana for the same reason {"Message":"User: anonymous is not authorized to perform: es:ESHttpPut"}
Here's couples of article I read before I raised the question here:
https://aws.amazon.com/premiumsupport/knowledge-center/anonymous-not-authorized-elasticsearch/?nc1=h_ls
https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-ac.html
https://aws.amazon.com/cn/blogs/database/set-access-control-for-amazon-elasticsearch-service/
Proper access policy for Amazon Elastic Search Cluster
https://aws.amazon.com/cn/blogs/security/how-to-control-access-to-your-amazon-elasticsearch-service-domain/
And still couldn't get it work.
Can anyone kindly guide me through the whole process of setting up the AWS ES manually, and being able to manipulate it through AWS CLI as well as kibana on browser? I would be really appreciated if it could be a detailed step-by-step guide instead of throwing aws docs. Thank you so much.
It turns out that the IP I used was incorrect. I should have called checkip.amazonaws.com while not under VPN, and the IP may change down the line. The policy on ES should looks like: Anther Answer
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::xxxxxxxxxxxx:root"
},
"Action": "es:*",
"Resource": "arn:aws:es:us-west-2:xxxxxxxxxxxx:domain/my-elasticsearch-domain/*"
},
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"AWS": "*"
},
"Action": "es:*",
"Resource": "arn:aws:es:us-west-2:xxxxxxxxxxxx:domain/my-elasticsearch-domain/*",
"Condition": {
"IpAddress": {
"aws:SourceIp": [
"192.168.1.0",
"192.168.1.1"
]
}
}
}
]
}
*PS. Make sure you are calling checkip.amazonaws.com on the same browser (same user as well if you are using chrome)
I currently working on a web project. We are trying to use elastic search. Our all web projects and other tools currently hosting on Amazon.
We just create an EC2 instance for Elastic search, Log Stash, and kibana. After that, we configured Elastic search. We're successfully using elastic search for our purposes.
But there is a problem way the communicating with elastic search and our Web APIs.
Our servers auto-scaling so we configured elastic search for IP security like below.
So we are currently suffocating because of auto-scaling. IPs changing time to time. How can we configure our Elastic Search for using from our Web API servers? What are the best practices for this situation?
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"AWS": "*"
},
"Action": "es:*",
"Resource": "arn:aws:es:eu-west:1111111111:domain/xx-log/*",
"Condition": {
"IpAddress": {
"aws:SourceIp": [
"1.1.1.1",
"1.1.1.2",
"1.1.1.3",
...
]
}
}
}
]
}
Your best bet is to create an iam role, and then in your launch configuration set that role as the "IAM role". http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html
And then after use a policy that allows the AWS role and some specific IPs access to your elasticsearch domain.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::ACCOUNTNUMBER:role/ROLENAME"
},
"Action": [
"es:ESHttpGet",
"es:ESHttpHead",
"es:ESHttpPost",
"es:ESHttpPut"
],
"Resource": "arn:aws:es:ca-central-1:ACCOUNTNUMBER:domain/ELASTICSEARCHDOMAINNAME/*"
},
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"AWS": "*"
},
"Action": [
"es:ESHttpGet",
"es:ESHttpHead",
"es:ESHttpPost",
"es:ESHttpPut"
],
"Resource": "arn:aws:es:ca-central-1:ACCOUNTNUMBER:domain/ELASTICSEARCHDOMAINNAME/*",
"Condition": {
"IpAddress": {
"aws:SourceIp": "IPADDRESSHERE"
}
}
}
]
}