what is the best approach in using aws elastic search with nodejs? I am using aws ecs ec2 instance for running my docker containers and is using the IAM role to accessing the other aws resource like S3 bucket and dynamodb from nodejs.
Can we use the same procedure for accessing the aws elastic search endpoint too?
I added an inline policy with the existing role and added the elastic search end point arn. but the nodejs sdk is not able to connect to the ES. when the aws key and id is added as environment variable in task definition it starts working. But I dont need to use that method as it will conflict with the other aws resource. (looks like the dev team is configured the program such that it looks for env)
It for sure is not the best method but you can also use a ip based restriction. We currently use this and it works fine. Just set an elastic ip on your ec2 instance (if you haven't already) and set the ip address in the access policy like this:
"Condition": {
"IpAddress": {
"aws:SourceIp": [
"XXX.XXX.XXX.XXX",
]
}
}
For anybody else stumbling across this, here's a few things I learnt whilst I was stuck on something similar:
EC2's role ARN can be added in the access policy for your Elasticsearch domain along with the permissions you want the role to have. For eg. for an EC2 running with role "aws-ec2" needing permissions to make HTTP GET requests to ES, you could have the following in your ES domain access policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": [
"arn:aws:iam::<ACCOUNT_ID>:role/aws-ec2",
]
},
"Action": "es:ESHttpGet",
"Resource": "arn:aws:es:<REGION>:<ACCOUNT_ID>:domain/<DOMAIN_NAME>/*"
}
]
}
Any requests made by an EC2 instance running with role "aws-ec2" in your account will have access to elasticsearch.
Note that if you have trouble getting credentials, try the following:
AWS.config.getCredentials(function(err) {
if (err) console.log(err.stack);
// credentials not loaded
else {
// credentials are loaded and can be accessed using
AWS.config.credentials.accessKeyId, AWS.config.credentials.secretAccessKeyId etc.
}
});
This will usually pull the credentials in like magic, I have a theory about how it works (tl:dr; I think it pulls them from the EC2 instance metadata by making a request to a fixed IP) but it's unproven so I won't embarrass myself until I know more. Note that this should work even if you don't have credentials stored in your environment or in the shared credentials file.
Related
I've currently writing a Terraform module for EC2 ASGs with ECS. Everything about starting the instances works, including IAM-dependent actions such as KMS-encrypted volumes. However, the ECS agent fails with:
Error getting ECS instance credentials from default chain: NoCredentialProviders: no valid providers in chain.
Unfortunately, most posts I find about this are about the CLI being run without credentials configured, however this should of course use the instance profile.
The posts I do find regarding that are about missing IAM policies for the instance role. However, I do have this trust relationship
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": [
"ec2.amazonaws.com",
"ecs.amazonaws.com"
]
},
"Action": "sts:AssumeRole"
}
]
}
(Added ECS because someone on SO had it in there, I don't think that's right. Also removed some conditions.)
It has these policies attached:
AmazonSSMManagedInstanceCore
AmazonEC2ContainerServiceforEC2Role
When I connect via SSH and try to run awscli commands, I get the error
Unable to locate credentials. You can configure credentials by running "aws configure".
running any command.
But with
curl http://169.254.169.254/latest/meta-data/iam/info
I see the correct instance profile ARN and with
curl http://169.254.169.254/latest/meta-data/identity-credentials/ec2/security-credentials/ec2-instance
I see temporary credentials. With these in the awscli configuration,
aws sts get-caller-identity
returns the correct results. There are no iptables rules or routes blocking access to the metadata service and I've deactivated IMDSv2 token requirement for the time being.
I'm using the latest stable version of ECS-optimized Amazon Linux 2.
What might be the issue here?
Is there a way where to attach an AWS IAM profile to an Azure VM.
I'm trying to develop a common infrastructure for Azure and AWS and i want to use resources which are in AWS from an Azure VM.
I know this can do this by exporting AWS creds to Azure VM but is there a way where I can attach an already existing AWS IAM profile to the Azure VM (if not directly may be through an interface or a service?) and access the resources (which how is I'm doing from an ec2 instance currently) ?
Sadly you can't do this. IAM instance profiles are only valid and usable from ec2 instances. You can't use instance profiles from outside of aws.
As you mentioned, you have to explicitly provide aws credentials to your azure vm. For example by creating .aws/ folder with aws profile.
You should be able to achieve what you are looking for by using the same IAM role for both the EC2 instance profile and a managed identity assigned to your Azure VM.
From my limited understanding of AWS, the instance profile identifies your EC2 instance, so you cant use it directly.
To use the same role for both EC2 and Azure VM, here is how I would try this out:
First, familiarize yourself with how you can assign an Azure managed identity a role in AWS. I wrote this blog post recently to show how Managed identities can be granted access to AWS resources: https://blog.identitydigest.com/azuread-access-aws/
Now rather than create a new role as mentioned in the blog, you can reuse the role in your EC2 instance, by adding a trust relationship for the managed identity to AssumeRoleWithWebIdentity.
So the trust relationship for your existing role used in your EC2 profile will look something as follows: (please note, I have not tried multiple statements in a role but expect this to work based on the AWS documentation)
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "ec2.amazonaws.com"
},
"Action": "sts:AssumeRole"
},
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::<your account>:oidc-provider/sts.windows.net/<your azure ad tenant>"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"sts.windows.net/<your tenant>:aud": "app audience or managed identity client_id",
"sts.windows.net/<your tenant>:sub": "in case you want to also include sub"
}
}
}
]
}
Now if you assign the managed identity to the VM, it should be able to access the same resources as your EC2 instance.
I have an ES domain and I want to access Kibana locally from within the same browser. Reading the documentation, it said that i could use Amazon Cognito to do the same with authentication for the users. I set the whole thing up as per the following AWS documentation Link
The problem is, whenever i try to access kibana from the browser using the link which looks like this :
https://vpc-something1-something2.us-east-1.es.amazonaws.com/_plugin/kibana/
the request times out. I'll post my access policy for the ES cluster here :
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::Account_ID:role/Cognito_Something_Auth_Role"
},
"Action": "es:*",
"Resource": "arn:aws:es:us-east-1:Account_ID:domain/domain_name/*"
}
]
}
I followed the procedure in the above link exactly and created a user group, identity group etc. But the link does not seem to load. Any help would be much appreciated.
PS : I'm new to AWS.
All this is assuming that i can directly access Kibana through my browser if i have cognito set up correctly.
If your ES cluster is created in a VPC, then you need networking access to it. I would recommend creating a cluster with 'Public access' instead, which is still subject to your access policy.
If you want a VPC cluster, and you want to access it (either ES directly, or Kibana) from outside that VPC, then you will need to VPN into the VPC, or do some routing that enables it to be exposed. The latter might be a bit tricky when the instances running your cluster aren't directly available to you, but you should be able to do it with some combination of Internet gateways, NAT gateways, security groups, routing tables, etc.
This might help: Connecting to a VPC
We have a requirement where we need to validate the AWS accountID from our code running on EC2 instance. One way I found is to get this information from AWS metadata IP at this URL:
http://169.254.169.254/latest/dynamic/instance-identity/document
but what if I dont have access to internet. Is it saved and retrievable from Instance without pinging any outside URL.
You will be able to access that URL even if your instance does not have internet access. Another way you can get the id is by using the aws cli. The get-caller-identity command returns the account, userid and the ARN. You will want to make sure you EC2 instance has permissions to call this.
aws sts get-caller-identity
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "sts:GetCallerIdentity",
"Resource": "*"
}
]
}
but what if I dont have access to internet. Is it saved and retrievable from Instance without pinging any outside URL.
This is not an outside IP, this IP is the local metadata service for your ec2 instance. It's not going through the internet. It's perfectly acceptable to use this to retrieve the account id, in fact amazon provides you with everything you need to retrieve this kind of information from an instance using their various SDK's.
The alternative solution, as quasar pointed out is to use aws sts get-caller-identity, however this will require permissions on the instance role to work.
I have been struggling to figure out how to communicate with the Amazon ES service from my EC2 instances.
The documentation clearly states that the Amazon ES service supports IAM User & Role based access policies. http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-createupdatedomains.html#es-createdomain-configure-access-policies
However, when I have this access policy for my ES domain:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::123456789:role/my-ec2-role"
},
"Action": "es:*",
"Resource": "arn:aws:es:us-west-2:123456789:domain/myDomain/*"
}
]
}
I can't log into an ec2 instance and run a curl to hit my elasticsearch cluster.
Trying to do a simple curl of the _search API:
curl "http://search-myDomain.es.amazonaws.com/_search"
Produces an authentication error response:
{"Message":"User: anonymous is not authorized to perform: es:ESHttpGet on resource: arn:aws:es:us-west-2:123456789:domain/myDomain/_search"}
Just to be extra safe I put the AmazonESFullAccess Policy on my IAM Role, still doesn't work.
I must be missing something, because being able to programmatically interact with Elasticsearch from ec2 instances that use an IAM Role is essential to getting anything accomplished with the Amazon ES Service.
I also see this contradictory statement in the docs.
IAM-based Policy Example You create IAM-based access policies by
using the AWS IAM console rather than the Amazon ES console. For
information about creating IAM-based access policies, see the IAM
documentation.
That link to IAM documentation, is to the home page of IAM and contains exactly zero information about how to do it. Anyone got a solution for me?
When using IAM service with AWS, you must sign your requests. curl doesn't support signed requests (which consists of hashing the request and adding a parameter to the header of the request). You can use one of their SDK's that has the signing algorithm built in, and then submit that request.
See:
http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/what-is-amazon-elasticsearch-service.html#signing-requests
You can find the SDKs for popular languages here:
http://aws.amazon.com/tools/
First, you said you can't login to an EC2 instance to curl the ES instance? You can't login? Or you can't curl it from EC2?
I have my Elasticsearch (Service) instance open to the world (with nothing on it) and am able to curl it just fine, without signing. I changed the access policy to test, but unfortunately it takes forever to come back up after changing it...
My policy looks like this:
{ "Version": "2012-10-17", "Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": "*",
"Action": "es:*",
"Resource": "arn:aws:es:us-east-1:843348267853:domain/myDomain/*"
},
{
"Sid": "",
"Effect": "Allow",
"Principal": "*",
"Action": "es:*",
"Resource": "arn:aws:es:us-east-1:843348267853:domain/myDomain"
}
]
}
I realize this isn't exactly what you want, but start off with this (open to the world), curl from outside AWS and test it. Then restrict it, that way you're able to isolate the issues.
Also, I think you have an issue with the "Principal" in your access policy. You have your EC2 Role. I understand why you're doing that, but I think the Principal requires a USER, not a role.
See below:
http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-createupdatedomains.html#es-createdomain-configure-access-policies
Principal
Specifies the AWS account or IAM user that is allowed or denied access
to a resource. Specifying a wildcard (*) enables anonymous access to
the domain, which is not recommended. If you do enable anonymous
access, we strongly recommend that you add an IP-based condition to
restrict which IP addresses can submit requests to the Amazon ES
domain.
EDIT 1
To be clear, you added the AmazonESFullAccess policy to the my-ec2-role? If you're going to use IAM access policies, I don't think you can have a resource based policy attached to it (which is what you're doing).
http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_compare-resource-policies.html
For some AWS services, you can grant cross-account access to your
resources. To do this, you attach a policy directly to the resource
that you want to share, instead of using a role as a proxy. The
resource that you want to share must support resource-based policies.
Unlike a user-based policy, a resource-based policy specifies who (in
the form of a list of AWS account ID numbers) can access that
resource.
Possibly try removing the access policy altogether?
Why you don't create a proxy with elastic ip and allow your proxy to access your ES?
Basically exists three forms that you can limit access in your ES:
Allow everyone
White IP list
Signing the access key and secret key provided by AWS.
I'm using two forms, in my php apps I prefer to use proxy behind the connection to ES and in my nodejs app I prefer to sign my requests using the http-aws-es node module.
It's useful to create a proxy environment because my users needs to access the kibana interface to see some reports and it's possible because they have configured the proxy in their browsers =)
I must recommend to you close the access to your ES indexes, because it's pretty easy to delete them, curl -XDELETE https://your_es_address/index anyone can do it but you can say: "how the others users will get my ES address?" and I will answer you: "Security based in dimness isn't a real security"
My security access policy is basically something like it:
http://pastebin.com/EUKT1ekX
I encountered this issue recently and the root problem is that none of the Amazon SDKs yet support calling Elasticsearch operations like search, put, etc.
The only workaround at the moment is to execute requests directly against the endpoint using signed requests:
http://docs.aws.amazon.com/general/latest/gr/sigv4-signed-request-examples.html
The example here is for calling EC2, but it can be modified to instead call against Elasticsearch. Just modify the "service" value to "es". From there, you have to fill in values for
the endpoint (which is the full URL of your cluster including operation without request parameters)
the host (the part between https:// and your canonical URI like /_status
the canonical uri which is the URI after the first / inclusive (like /_status) but without the query string
the request parameters (everything after ? inclusive)
Note that I've only managed to get this working so far using AWS credentials as the assumption is that you pass in an access key and secret key to the various signing calls (access_key and secret_key in the example). It should be doable using IAM roles but you'll have to call into the security token service first to get temporary credentials that can be used to sign the request. Until you do that, be sure to edit your access policy on the Elasticsearch cluster to allow user creds (user/
you need to sign your request and unfortunately, it is no longer supported by the official elasticsearch library. Check this Github issue (https://github.com/elastic/elasticsearch-js/issues/1182#issuecomment-630641702)
They want to enforce their own cloud solution