We have a requirement where we need to validate the AWS accountID from our code running on EC2 instance. One way I found is to get this information from AWS metadata IP at this URL:
http://169.254.169.254/latest/dynamic/instance-identity/document
but what if I dont have access to internet. Is it saved and retrievable from Instance without pinging any outside URL.
You will be able to access that URL even if your instance does not have internet access. Another way you can get the id is by using the aws cli. The get-caller-identity command returns the account, userid and the ARN. You will want to make sure you EC2 instance has permissions to call this.
aws sts get-caller-identity
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "sts:GetCallerIdentity",
"Resource": "*"
}
]
}
but what if I dont have access to internet. Is it saved and retrievable from Instance without pinging any outside URL.
This is not an outside IP, this IP is the local metadata service for your ec2 instance. It's not going through the internet. It's perfectly acceptable to use this to retrieve the account id, in fact amazon provides you with everything you need to retrieve this kind of information from an instance using their various SDK's.
The alternative solution, as quasar pointed out is to use aws sts get-caller-identity, however this will require permissions on the instance role to work.
Related
I'm trying to use aws ec2 authorize-security-group-egress to have a script on the ec2 instance temporarily (I'll be pairing with revoke) open a port out to a particular IP. However, when I try to run the command it tells me the user isn't authorized to perform that command. The rest of the error message is encrypted, and unfortunately aws sts decode-authorization-message is also not authorized so I can't get any more information. I gather I need to go into the web console and give the user a particular IAM role but I haven't been able to find what role that is.
The error message is: An error occurred (UnauthorizedOperation) when calling the AuthorizeSecurityGroupIngress operation: You are not authorized to perform this operation. followed by five lines of junk that is the encrypted remainder of the message.
Generally, what is a good practice, when a script or program executes on an EC2 instance you give permissions to it through instance roles.
In your case, since you want to use AuthorizeSecurityGroupEgress, the instance role would need to have such permission.
An example of an inline policy in an instance role that you could use is:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": "ec2:AuthorizeSecurityGroupEgress",
"Resource": "*"
}
]
}
AWS SDK and CLI automatically will use the instance role, thus there are no actions required from you to make use of them.
Also check this,
--group-id (string)
The ID of the security group. You must specify either the security group ID or the security group name in the request. For security groups in a nondefault VPC, you must specify the security group ID.
This is what blocked my user using the --group-name flag because the security group was not in the default VPC.
https://awscli.amazonaws.com/v2/documentation/api/latest/reference/ec2/authorize-security-group-ingress.html#options
I have an ES domain and I want to access Kibana locally from within the same browser. Reading the documentation, it said that i could use Amazon Cognito to do the same with authentication for the users. I set the whole thing up as per the following AWS documentation Link
The problem is, whenever i try to access kibana from the browser using the link which looks like this :
https://vpc-something1-something2.us-east-1.es.amazonaws.com/_plugin/kibana/
the request times out. I'll post my access policy for the ES cluster here :
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::Account_ID:role/Cognito_Something_Auth_Role"
},
"Action": "es:*",
"Resource": "arn:aws:es:us-east-1:Account_ID:domain/domain_name/*"
}
]
}
I followed the procedure in the above link exactly and created a user group, identity group etc. But the link does not seem to load. Any help would be much appreciated.
PS : I'm new to AWS.
All this is assuming that i can directly access Kibana through my browser if i have cognito set up correctly.
If your ES cluster is created in a VPC, then you need networking access to it. I would recommend creating a cluster with 'Public access' instead, which is still subject to your access policy.
If you want a VPC cluster, and you want to access it (either ES directly, or Kibana) from outside that VPC, then you will need to VPN into the VPC, or do some routing that enables it to be exposed. The latter might be a bit tricky when the instances running your cluster aren't directly available to you, but you should be able to do it with some combination of Internet gateways, NAT gateways, security groups, routing tables, etc.
This might help: Connecting to a VPC
what is the best approach in using aws elastic search with nodejs? I am using aws ecs ec2 instance for running my docker containers and is using the IAM role to accessing the other aws resource like S3 bucket and dynamodb from nodejs.
Can we use the same procedure for accessing the aws elastic search endpoint too?
I added an inline policy with the existing role and added the elastic search end point arn. but the nodejs sdk is not able to connect to the ES. when the aws key and id is added as environment variable in task definition it starts working. But I dont need to use that method as it will conflict with the other aws resource. (looks like the dev team is configured the program such that it looks for env)
It for sure is not the best method but you can also use a ip based restriction. We currently use this and it works fine. Just set an elastic ip on your ec2 instance (if you haven't already) and set the ip address in the access policy like this:
"Condition": {
"IpAddress": {
"aws:SourceIp": [
"XXX.XXX.XXX.XXX",
]
}
}
For anybody else stumbling across this, here's a few things I learnt whilst I was stuck on something similar:
EC2's role ARN can be added in the access policy for your Elasticsearch domain along with the permissions you want the role to have. For eg. for an EC2 running with role "aws-ec2" needing permissions to make HTTP GET requests to ES, you could have the following in your ES domain access policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": [
"arn:aws:iam::<ACCOUNT_ID>:role/aws-ec2",
]
},
"Action": "es:ESHttpGet",
"Resource": "arn:aws:es:<REGION>:<ACCOUNT_ID>:domain/<DOMAIN_NAME>/*"
}
]
}
Any requests made by an EC2 instance running with role "aws-ec2" in your account will have access to elasticsearch.
Note that if you have trouble getting credentials, try the following:
AWS.config.getCredentials(function(err) {
if (err) console.log(err.stack);
// credentials not loaded
else {
// credentials are loaded and can be accessed using
AWS.config.credentials.accessKeyId, AWS.config.credentials.secretAccessKeyId etc.
}
});
This will usually pull the credentials in like magic, I have a theory about how it works (tl:dr; I think it pulls them from the EC2 instance metadata by making a request to a fixed IP) but it's unproven so I won't embarrass myself until I know more. Note that this should work even if you don't have credentials stored in your environment or in the shared credentials file.
I need to fire up an S3 bucket so my EC2 instances have access to store image files to it. The EC2 instances need read/write permissions. I do not want to make the S3 bucket publicly available, I only want the EC2 instances to have access to it.
The other gotcha is my EC2 instances are being managed by OpsWorks and I can have may different instances being fired up depending on load/usage. If I were to restrict it by IP, I may not always know the IP the EC2 instances have. Can I restrict by VPC?
Do I have to make my S3 bucket enabled for static website hosting?
Do I need to make all files in the bucket public as well for this to work?
You do not need to make the bucket public readable, nor the files public readable. The bucket and it's contents can be kept private.
Don't restrict access to the bucket based on IP address, instead restrict it based on the IAM role the EC2 instance is using.
Create an IAM EC2 Instance role for your EC2 instances.
Run your EC2 instances using that role.
Give this IAM role a policy to access the S3 bucket.
For example:
{
"Version": "2012-10-17",
"Statement":[{
"Effect": "Allow",
"Action": "s3:*",
"Resource": ["arn:aws:s3:::my_bucket",
"arn:aws:s3:::my_bucket/*"]
}
]
}
If you want to restrict access to the bucket itself, try an S3 bucket policy.
For example:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": ["arn:aws:iam::111122223333:role/my-ec2-role"]
},
"Action": "s3:*",
"Resource": ["arn:aws:s3:::my_bucket",
"arn:aws:s3:::my_bucket/*"]
}
]
}
Additional information: http://blogs.aws.amazon.com/security/post/TxPOJBY6FE360K/IAM-policies-and-Bucket-Policies-and-ACLs-Oh-My-Controlling-Access-to-S3-Resourc
This can be done very simply.
Follow the following steps:
Open the AWS EC2 on console.
Select the instance and navigate to actions.
Select instances settings and select Attach/Replace IAM Role
Create a new role and attach S3FullAccess policy to that role.
When this is done, connect to the AWS instance and the rest will be done via the following CLI commands:
aws s3 cp filelocation/filename s3://bucketname
Please note... the file location refers to the local address. And the bucketname is the name of your bucket.
Also note: This is possible if your instance and S3 bucket are in the same account.
Cheers.
IAM role is the solution for you.
You need create role with s3 access permission, if the ec2 instance was started without any role, you have to re-build it with that role assigned.
Refer: Allowing AWS OpsWorks to Act on Your Behalf
I have been struggling to figure out how to communicate with the Amazon ES service from my EC2 instances.
The documentation clearly states that the Amazon ES service supports IAM User & Role based access policies. http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-createupdatedomains.html#es-createdomain-configure-access-policies
However, when I have this access policy for my ES domain:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::123456789:role/my-ec2-role"
},
"Action": "es:*",
"Resource": "arn:aws:es:us-west-2:123456789:domain/myDomain/*"
}
]
}
I can't log into an ec2 instance and run a curl to hit my elasticsearch cluster.
Trying to do a simple curl of the _search API:
curl "http://search-myDomain.es.amazonaws.com/_search"
Produces an authentication error response:
{"Message":"User: anonymous is not authorized to perform: es:ESHttpGet on resource: arn:aws:es:us-west-2:123456789:domain/myDomain/_search"}
Just to be extra safe I put the AmazonESFullAccess Policy on my IAM Role, still doesn't work.
I must be missing something, because being able to programmatically interact with Elasticsearch from ec2 instances that use an IAM Role is essential to getting anything accomplished with the Amazon ES Service.
I also see this contradictory statement in the docs.
IAM-based Policy Example You create IAM-based access policies by
using the AWS IAM console rather than the Amazon ES console. For
information about creating IAM-based access policies, see the IAM
documentation.
That link to IAM documentation, is to the home page of IAM and contains exactly zero information about how to do it. Anyone got a solution for me?
When using IAM service with AWS, you must sign your requests. curl doesn't support signed requests (which consists of hashing the request and adding a parameter to the header of the request). You can use one of their SDK's that has the signing algorithm built in, and then submit that request.
See:
http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/what-is-amazon-elasticsearch-service.html#signing-requests
You can find the SDKs for popular languages here:
http://aws.amazon.com/tools/
First, you said you can't login to an EC2 instance to curl the ES instance? You can't login? Or you can't curl it from EC2?
I have my Elasticsearch (Service) instance open to the world (with nothing on it) and am able to curl it just fine, without signing. I changed the access policy to test, but unfortunately it takes forever to come back up after changing it...
My policy looks like this:
{ "Version": "2012-10-17", "Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": "*",
"Action": "es:*",
"Resource": "arn:aws:es:us-east-1:843348267853:domain/myDomain/*"
},
{
"Sid": "",
"Effect": "Allow",
"Principal": "*",
"Action": "es:*",
"Resource": "arn:aws:es:us-east-1:843348267853:domain/myDomain"
}
]
}
I realize this isn't exactly what you want, but start off with this (open to the world), curl from outside AWS and test it. Then restrict it, that way you're able to isolate the issues.
Also, I think you have an issue with the "Principal" in your access policy. You have your EC2 Role. I understand why you're doing that, but I think the Principal requires a USER, not a role.
See below:
http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-createupdatedomains.html#es-createdomain-configure-access-policies
Principal
Specifies the AWS account or IAM user that is allowed or denied access
to a resource. Specifying a wildcard (*) enables anonymous access to
the domain, which is not recommended. If you do enable anonymous
access, we strongly recommend that you add an IP-based condition to
restrict which IP addresses can submit requests to the Amazon ES
domain.
EDIT 1
To be clear, you added the AmazonESFullAccess policy to the my-ec2-role? If you're going to use IAM access policies, I don't think you can have a resource based policy attached to it (which is what you're doing).
http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_compare-resource-policies.html
For some AWS services, you can grant cross-account access to your
resources. To do this, you attach a policy directly to the resource
that you want to share, instead of using a role as a proxy. The
resource that you want to share must support resource-based policies.
Unlike a user-based policy, a resource-based policy specifies who (in
the form of a list of AWS account ID numbers) can access that
resource.
Possibly try removing the access policy altogether?
Why you don't create a proxy with elastic ip and allow your proxy to access your ES?
Basically exists three forms that you can limit access in your ES:
Allow everyone
White IP list
Signing the access key and secret key provided by AWS.
I'm using two forms, in my php apps I prefer to use proxy behind the connection to ES and in my nodejs app I prefer to sign my requests using the http-aws-es node module.
It's useful to create a proxy environment because my users needs to access the kibana interface to see some reports and it's possible because they have configured the proxy in their browsers =)
I must recommend to you close the access to your ES indexes, because it's pretty easy to delete them, curl -XDELETE https://your_es_address/index anyone can do it but you can say: "how the others users will get my ES address?" and I will answer you: "Security based in dimness isn't a real security"
My security access policy is basically something like it:
http://pastebin.com/EUKT1ekX
I encountered this issue recently and the root problem is that none of the Amazon SDKs yet support calling Elasticsearch operations like search, put, etc.
The only workaround at the moment is to execute requests directly against the endpoint using signed requests:
http://docs.aws.amazon.com/general/latest/gr/sigv4-signed-request-examples.html
The example here is for calling EC2, but it can be modified to instead call against Elasticsearch. Just modify the "service" value to "es". From there, you have to fill in values for
the endpoint (which is the full URL of your cluster including operation without request parameters)
the host (the part between https:// and your canonical URI like /_status
the canonical uri which is the URI after the first / inclusive (like /_status) but without the query string
the request parameters (everything after ? inclusive)
Note that I've only managed to get this working so far using AWS credentials as the assumption is that you pass in an access key and secret key to the various signing calls (access_key and secret_key in the example). It should be doable using IAM roles but you'll have to call into the security token service first to get temporary credentials that can be used to sign the request. Until you do that, be sure to edit your access policy on the Elasticsearch cluster to allow user creds (user/
you need to sign your request and unfortunately, it is no longer supported by the official elasticsearch library. Check this Github issue (https://github.com/elastic/elasticsearch-js/issues/1182#issuecomment-630641702)
They want to enforce their own cloud solution