i'm trying to connect a custom kibana with the AWS elasticsearch service, AWS ES cluster uses api keys based authentication and requires a signed request, viewing the code i can see there is a module aws4 installed but i don't see how is it used to sign requests, i can't see no configuration related in kibana.
Thanks.
Related
I am trying to add an authentication method to AWS OpenSearch.
By default it comes with basic auth with internal db,
I would like to configure Security Plugin with a second authentication mechanism OpenId Connect
Documentation says to use Update Security Configuration , which doesn't seem to be allowed by AWS.
tried PUT _plugins/_security/api/securityconfig and PUT _plugins/_security/api/securityconfig/authc both seems to be failing with {"Message":"Your request: '/_plugins/_security/api/securityconfig/authc' is not allowed."}
Is there an alternative, all I want to do is , use JWT Token(from OpenID Connect Cognito) to run OpenSearch _search API, rather than using internal database basic auth or IAM Role from Federated Pool.
AWS Opensearch has blocked apis for updating security configuration which is why you see the error. The documentation you have linked is applicable when using opensearch security plugin for your self hosted opensearch service and not AWS hosted solution.
I am not sure if this will be helpful to you but this blog does talk about using OpenId Connect along with AWS Cognito for opensearch auth: https://aws.amazon.com/blogs/apn/use-amazon-opensearch-service-with-kibana-for-identity-federation-auth0/
AWS recommends using its SDKs (such as boto3) or command-line tools to configure an ElasticSearch cluster.
However, some ElasticSearch API endpoints are not exposed in AWS APIs (e.g. _cat/shards).
Even some AWS support documents (such as this one on cluster rebalancing) seem to make direct request to the cluster API.
The trouble is: such requests need to be authenticated using AWS4Auth (only certain IAM roles have permissions to write to ElasticSearch, in my setup) – and even AWS recommends against making manually creating signed HTTP requests because it's such a pain.
My question is: do I need to manually create signed HTTP requests against my ES cluster in order to manage it, or is there an easier way that I've missed?
Based on the comments.
The proposed solution is to use third party aws-requests-auth:
This package allows you to authenticate to AWS with Amazon's signature version 4 signing process with the python requests library.
The example of its use for ElasticSearch is:
from aws_requests_auth.aws_auth import AWSRequestsAuth
from elasticsearch import Elasticsearch, RequestsHttpConnection
es_host = 'search-service-foobar.us-east-1.es.amazonaws.com'
auth = AWSRequestsAuth(aws_access_key='YOURKEY',
aws_secret_access_key='YOURSECRET',
aws_host=es_host,
aws_region='us-east-1',
aws_service='es')
# use the requests connection_class and pass in our custom auth class
es_client = Elasticsearch(host=es_host,
port=80,
connection_class=RequestsHttpConnection,
http_auth=auth)
print es_client.info()
I ended up finding this AWS doc on request signing for AWS ElasticSearch. It clearly shows that the intended approach is to use scripts, using the HTTP client of choice for the language.
As Marcin mentioned in hist answer, aws-requests-auth is one choice to simplifiy this in Python
I need to get elements from AWS dynamoDB and thrid party https service and merge those results in AWS appSyn and send back the result as graphQL response
Third party service which I am using, expects client side certificate. I am not finding proper AWS documents on how to pass agent using AWS appSync resolver.
I am also not finding documents to store certificate as secret in AWS secret manager.
Is there anyone faced similar problem? Or do you guys have any solution to it?
It depends on the size of your certificate - Secrets for AWS Secrets Manager have various limits such as length in bytes (7168 bytes) or characters (4096) see more here:
https://docs.aws.amazon.com/secretsmanager/latest/userguide/reference_limits.html
But otherwise you should be able to store your certificate using AWS Secrets Manager.
See number item 3 in the following link: https://aws.amazon.com/blogs/compute/maintaining-transport-layer-security-all-the-way-to-your-container-part-2-using-aws-certificate-manager-private-certificate-authority/
Using HTTP Resolvers (or even Lambda Resolvers) you will be able to make http calls to AWS Secrets Manager to obtain the secret.
See links:
https://docs.aws.amazon.com/appsync/latest/devguide/tutorial-lambda-resolvers.html
https://docs.aws.amazon.com/appsync/latest/devguide/tutorial-http-resolvers.html
Adding to Ashwin's answer, According to documentation, HTTP resolver supports only public endpoint at this point and does not seem to have ability to pass a certificate for app to app call
Is there any AWS ElasticSearch Client SDK for Java, which signs the requests with AWS credentials? I saw this, but I guess its for managing AWS Elasticsearch Service.
You can create AWS client for Elasticsearch in the following way:
AwsClientBuilder.withCredentials(AWSCredentialsProvider) for example: AWSElasticsearchClientBuilder.standard().withCredentials(new AWSStaticCredentialsProvider(awsCredentials)).build();
How I do this is by first creating AWSBasicSessionCredential instance by providing awsAccessKey, awsSecretKey, and sessionToken information and then passing this instance in the above code to build the client instance. However, I do this for test programming the clients. Its not advised to create a client this way though. For example, one secure way would be using federated identification to generate a temporary security token and then use that to assume a role through AWS' AssumeRoleRequest, receive its response in the form of AssumeRoleResult and then retrieve credential information from the assumeRoleResult response received above. Use this credential information in generating AWSCredential.
Source of generating AWS Elasticsearch client.
I am new to AWS and working on an application in which I have to display different Metrics about my AWS EC2 instance Health Like CPU usage and Memory and Running Processes info. I am trying to perform the Get Request from a REST client (Chrome Extention) for clarity of the function of this API exposed by AWS.
https://dns.regionxx.ec2.amazonaws.com/?
Action=DescribeInstanceStatus
&AWSAccessKeyId=0GS7553JW74RRM612K02EXAMPLE
&Timestamp=1397109362
&Signature=lBP67vCvGlDMBQ1dofZxg8E8SUEXAMPLE
&SignatureVersion=2
&SignatureMethod=HMAC-SHA1
REST Client Support OAuth 1.0 is this an issue because AWS is supporting Signature Version 2 and 4.
Any Help?
You can do it the straight way with Signatures but why not use one of the existing SDKs like boto (Python), Ruby AWS SDK or the Java AWS SDK. These SDKs take care all of the authentication/authorization mechanism and you just need provide your aws_access_key_id and your aws_secret_access_key
For example with boto you can do something like this:
import boto.ec2
conn = boto.ec2.connect_to_region("us-east-1",
aws_access_key_id='<key>',
aws_secret_access_key='<secret>')
reservations = conn.get_all_instances
for reservation in reservations:
for instance in reservation:
print instance.status
Or you can use the EC2 command line API http://aws.amazon.com/developertools/351 :
ec2-describe-instance-status <instances->
Or you can use the AWS CLI:
aws ec2 describe-instance-status <instance id>