I am trying to use AWS Elasticsearch services.On localhost, I installed elastic search locally which is working perfectly. But, now I want to use AWS elastic search but for that I need credentials like
'aws' => env('AWS_ELASTICSEARCH_ENABLED', false),
'aws_region' => env('AWS_REGION', ''),
'aws_key' => env('AWS_ACCESS_KEY_ID', ''),
'aws_secret' => env('AWS_SECRET_ACCESS_KEY', '')
'aws_credentials' => $credentials
I don't know from where I can get that information. Please help me to know
Bilal, on creating index in AES, you get an endpoint. Simply use that endpoint in your application and send cURL/ CLI request to your index.
Note, in your Access Policy of AES, allow your IP to access AES endpoint.
How to get your AWS credentials is explained in:
Understanding and getting your AWS credentials
Further, you can also setup a local AWS profile on your computer as explained here.
Related
I am trying to work with the new Instance Metadata Service Version 2 (IMDSv2) API.
It works as expected when I try to query the metadata manually as described on Retrieve instance metadata - Amazon Elastic Compute Cloud.
However, if I try to query for the instance tags it fails with error message:
Couldn't find AWS credentials in environment, credentials file, or IAM role
The tags query is done by the Rusoto SDK that I am using, that works when I set --http-tokens optional as described on Configure the instance metadata options - Amazon Elastic Compute Cloud.
I don't fully understand why setting the machine to work with IMDSv2 would effect the DescribeTags request, as I believe it's not using the same API - so I am guessing that's a side effect.
If I try and do a manual query using curl (instead of using the SDK):
https://ec2.amazonaws.com/?Action=DescribeTags&Filter.1.Name=resource-id&Filter.1.Value.1=ami-1a2b3c4d
I get:
The action DescribeTags is not valid for this web service
Thanks :)
The library that I was using (Rusoto SDK 0.47.0) doesn't support fetching the credentials needed when the host is set to work with the IMDSv2.
The workaround was to manually query for the IAM role credentials.
First, you get the token:
GET /latest/api/token
Next, use the token header "X-aws-ec2-metadata-token" with the value from the previous:
GET /meta-data/iam/security-credentials
Afterwards, use the result from the previous query (and don't forget to set the token header), and query:
GET /meta-data/iam/security-credentials/<query 2 result>
This will provide with the following data:
struct SecurityCredentials {
#[serde(rename = "AccessKeyId")]
access_key_id: String,
#[serde(rename = "SecretAccessKey")]
secret_access_key: String,
#[serde(rename = "Token")]
token: String,
}
Then what I needed to do was to build a custom credentials provider using that data (but this part is already lib specific).
I enjoy using the AWS SDK without having to specify where to find the credentials, it makes it easier to configure on multiple environment where different types of credentials are available.
The AWS SDK for Ruby searches for credentials [...]
Is there some way I can retrieve the code that does this to configure Faraday with AWS ? In order to configure Faraday I need something like
faraday.request(:aws_sigv4,
service: 'es',
credentials: credentials,
region: ENV['AWS_REGION'],
)
Now I would love that this credentials be picked "automatically" as in the aws sdk v3. How Can I do that ?
(ie where is the code in the AWS SDK v3 that does something like
credentials = Aws::Credentials.new(ENV['AWS_ACCESS_KEY_ID'], ENV['AWS_SECRET_ACCESS_KEY'])
unless credentials.set?
credentials = Aws::InstanceProfileCredentials.new
end
...
The class Aws::CredentialProviderChain is the one in charge of resolving the credentials, but it is tagged as #api private so there's currently no guarantee it will still be there after updates (I've opened a discussion to make it public).
If you're okay to use it, you can resolve credentials like this. I'm going to test it in CI (ENV creds), development (Aws config creds), and staging/production environments (Instance profile creds).
Aws::CredentialProviderChain.new.resolve
You can use it in middlewares like this (for example configuring Elasticsearch/Faraday)
faraday.request(:aws_sigv4,
service: 'es',
credentials: Aws::CredentialProviderChain.new.resolve,
region: ENV['AWS_REGION'],
)
end
I have a PHP app running on a AWS EC2 instance. I want to upload a file to unit and then save that file to a S3 bucket. I can get the file up to the Ec2 instance with now problem but for the life of me I cannot figure out how to use the puObject call to move it to the S3 instance... I am very new to AWS so if anyone can give me a pointer, that'd help out a lot!
You have three options for how to transfer data from the EC2 instance to S3 using PHP:
1. Use the AWS PHP SDK
This is the preferred option. The SDK is released by Amazon and includes an easy to use API interface for all of their services. It can also allow you to use the EC2 instance's IAM role for credentials, meaning you don't necessarily need to store your API keys etc in the code. Example use:
// Use an Aws\Sdk class to create the S3Client object.
$s3Client = $sdk->createS3();
// Send a PutObject request and get the result object.
$result = $s3Client->putObject([
'Bucket' => 'my-bucket',
'Key' => 'my-key',
'Body' => file_get_contents($pathToYourFile)
]);
// Download the contents of the object.
$result = $s3Client->getObject([
'Bucket' => 'my-bucket',
'Key' => 'my-key'
]);
// Print the body of the result by indexing into the result object.
echo $result['Body'];
2. Use the EC2 instance's AWS CLI
This is less ideal, but still doable. I wouldn't recommend doing this as it would mean you use PHP to access the shell, and assume that the shell has the AWS CLI configured (most if not all EC2 instances would do by default). Example:
shell_exec('aws s3 cp /path/to/your/file.txt s3://bucket-name/path/your/file.txt');
3. Bake your own S3 API call using REST
This is probably the next best option after using the SDK, although it requires that you store the credentials in code, have to formulate your HMAC signatures and ensure that your API structure matches Amazon's guidelines. Note that the PHP SDK does this all for you, but this way you don't need the whole SDK installed.
If you go this way, you'll need to read up about how to sign your requests.
I have setup a new ElasticSearch cluster on AWS which is only allowing access to a specific IAM user.
However, I'm trying to connect to this from Ruby and looked at using the AWS SDK but that has no methods for actually making HTTP operations against your ES cluster, only accessing the configuration APIs.
As usual, this requires all the AWS request signing stuff that they require for API access, but I can't find anything that indicates how to do this stuff. I'm using Ruby.
Essentially, what I'm after is being able to make GET and PUT requests to this cluster using the IAM user creds. IP restriction isn't an option for me.
You can make signed, secure requests to Amazon Elasticsearch from Ruby. I did the following with an app on Heroku.
Ensure you have elasticsearch gem >= v1.0.15 as support for this was only implemented there Dec 4th, 2015.
You also need this gem:
gem 'faraday_middleware-aws-signers-v4'
Example from the elasticsearch-ruby/elasticsearch-transport
documentation:
You can use any standard Faraday middleware and plugins in the configuration block, for example sign the requests for the AWS Elasticsearch service:
With the following code:
require 'faraday_middleware/aws_signers_v4'
client = Elasticsearch::Client.new(url: ENV['AWS_ENDPOINT_URL']) do |f|
f.request :aws_signers_v4,
credentials: Aws::Credentials.new(ENV['AWS_ACCESS_KEY_ID'], ENV['AWS_SECRET_ACCESS_KEY']),
service_name: 'es',
region: 'us-east-1'
f.adapter Faraday.default_adapter
end
This also works with the searchkick gem with Rails. Set Searchkick.client using the above example, in an initializer:
# config/initializers/elasticsearch.rb
require 'faraday_middleware/aws_signers_v4'
Searchkick.client = Elasticsearch::Client.new(url: ENV['AWS_ENDPOINT_URL']) do |f|
f.request :aws_signers_v4,
credentials: Aws::Credentials.new(ENV['AWS_ACCESS_KEY_ID'], ENV['AWS_SECRET_ACCESS_KEY']),
service_name: 'es',
region: 'us-east-1'
f.adapter Faraday.default_adapter
end
I'm having a problem with my AWS credentials. I used the credentials file that I created on ~/.aws/credentials just as it is written on the AWS doc. However, apache just can't read it.
First, I was getting this error:
Error retrieving credentials from the instance profile metadata server. When you are not running inside of Amazon EC2, you must provide your AWS access key ID and secret access key in the "key" and "secret" options when creating a client or provide an instantiated Aws\Common\Credentials CredentialsInterface object.
Then I tried some solutions that I found on internet. For example, I tried to check my HOME variable. It was /home/ubuntu. I tried also to move my credentials file to the /var/www directory even if it is not my web server directory. Nothing worked. I was still getting the same error.
As a second solution, I saw that we could call directly the CredentialsProvider and indicate the directory on the client.
https://forums.aws.amazon.com/thread.jspa?messageID=583216򎘰
The error changed but I couldn't make it work:
Cannot read credentials from /.aws/credentials
I saw also that we could use the default provider of the CredentialsProvider instead of indicating a path.
http://docs.aws.amazon.com/aws-sdk-php/v3/guide/guide/credentials.html#using-credentials-from-environment-variables
I tried and I keep getting the same error:
Cannot read credentials from /.aws/credentials
Just in case you need this information, I'm using aws/aws-sdk-php (3.2.5). The service I'm trying to use is the AWS Elastic Transcoder. My EC2 instance is an Ubuntu 14.04. It runs a Symfony application deployed using Capifony.
Before I try on this production server, I tried it in a development server where it works perfectly only with the ~/.aws/credentials file. This development server is exactly a copy of the production server. However, it doesn't use Capifony for the deployment. It is just a normal git clone of the project. And it has only one EBS volume while the production server has one for the OS and one for the application.
Ah! And I also checked if the permissions/owners of the credentials file were the same on both servers and they are the same. I tried a 777 to see if it could change something but nothing.
Does anybody have an idea?
It sounds like you're doing it wrong. You do not need to deploy credentials to an EC2 instance in order to have that instance interact with other AWS services, and if fact should not ever deploy credentials to an EC2 instance.
Instead, when you create your instance, you associate an IAM role with it. That role has policies that control access to the other AWS services.
You can create an empty role, launch the instance, and then modify the role later. You can't assign a role after the instance has been launched.
You can now add roles to an instance after it has been assigned.
It is still considered a best practice to not deploy actual credentials to an EC2 instance.
If this can help someone, I managed to make my .ini file work, doing this way:
$profile = 'default';
$path = '/mnt/app/www/.aws/credentials/default.ini';
$provider = CredentialProvider::ini($profile, $path);
$provider = CredentialProvider::memoize($provider);
$client = ElasticTranscoderClient::factory(array(
'region' => 'eu-west-1',
'version' => '2012-09-25',
'credentials' => $provider
));
The CredentialProvider is explained on this doc:
http://docs.aws.amazon.com/aws-sdk-php/v3/guide/guide/credentials.html#ini-provider
I still don't understand why my application can't read the file on the home directory (~/.aws/credentials/default.ini) on one server but in the other it does.
If someone knows something about it, please let me know.
The SDK reads from a file located at ~/.aws/credentials, but it looks like you're saving a file at ~/.aws/credentials/default.ini. If you move the file, the error you were experiencing should be cleared up.
2 Ways of solving this problem to me Node.js
Its going to get my credentials from /home/{USER}/.aws/credentials usin' the default profile
const aws = require('aws-sdk');
aws.config.credentials = aws.SharedIniFileCredentials({profile: 'default'})
...
The hardcoded way
var lambda = new aws.Lambda({
region: 'us-east-1',
accessKeyId: <KEY>
secretAccessKey: <KEY>
});