I'm trying to make a simple Python Lambda that makes snapshots of our Elasticsearch database. This is done through Elasticsearch's REST API using simple HTTP requests.
However, for AWS, I have to sign these requests. I have a feeling it can be achieved through boto3's low-level clients probably with generate_presigned_url, but I cannot for the life of me figure out how to invoke this function correctly. For example, what are the valid ClientMethods? I've tried ESHttpGet but to no avail.
Can anyone point me in the right direction?
Edit: Apparently this workaround has been broken by Elastic.
I struggled for a while to do a similar thing. Currently the boto3 library doesn't support making signed es requests, though since I raised an issue with them it's become a feature request.
Here's what I've done in the meantime using DavidMuller's library mentioned above and boto3 to get my STS session credentials:
import boto3
from aws_requests_auth.aws_auth import AWSRequestsAuth
from elasticsearch import Elasticsearch, RequestsHttpConnection
session = boto3.session.Session()
credentials = session.get_credentials().get_frozen_credentials()
es_host = 'search-my-es-domain.eu-west-1.es.amazonaws.com'
awsauth = AWSRequestsAuth(
aws_access_key=credentials.access_key,
aws_secret_access_key=credentials.secret_key,
aws_token=credentials.token,
aws_host=es_host,
aws_region=session.region_name,
aws_service='es'
)
# use the requests connection_class and pass in our custom auth class
es = Elasticsearch(
hosts=[{'host': es_host, 'port': 443}],
http_auth=awsauth,
use_ssl=True,
verify_certs=True,
connection_class=RequestsHttpConnection
)
print(es.info())
Hope this saves somebody some time.
There are several Python extensions to the requests library that will perform the SigV4 signing for you. I have used this one and it works well.
While other answers are perfectly fine, I wanted to eliminate the use of external packages. Obviously, botocore itself has all the required functionality to sign requests it was just a matter of looking at the source code. This is what I ended up with for sending AWS API requests directly (things are hardcoded for the demonstration purposes):
import boto3
import botocore.credentials
from botocore.awsrequest import AWSRequest
from botocore.endpoint import URLLib3Session
from botocore.auth import SigV4Auth
params = '{"name": "hello"}'
headers = {
'Host': 'ram.ap-southeast-2.amazonaws.com',
}
request = AWSRequest(method="POST", url="https://ram.ap-southeast-2.amazonaws.com/createresourceshare", data=params, headers=headers)
SigV4Auth(boto3.Session().get_credentials(), "ram", "ap-southeast-2").add_auth(request)
session = URLLib3Session()
r = session.send(request.prepare())
I recently published requests-aws-sign, which provides AWS V4 request signing for the Python requests library.
If you look at this code you will see how you can use Botocore to generate the V4 request signing.
why not just use requests?
import requests
headers = {'Content-Type': 'application/json',}
data = '{"director": "Burton, Tim", "genre": ["Comedy","Sci-Fi","R-rated"],"profit" : 98 , "year": 1996, "actor": ["Jack Nicholson","PierceBrosnan","Sarah Jessica Parker"], "title": "Mars Attacks!"}'
response = requests.post('https://search-your-awsendpoint.us-west-2.es.amazonaws.com/yourindex/_yourdoc/', headers=headers, data=data)
this worked for me
Related
I have deployed a simple fast api to aws API gateway .All the end points working fine however i am unable to load the swagger docs page I see below error
Api Code:
from fastapi import FastAPI
from mangum import Mangum
import os
from fastapi.middleware.cors import CORSMiddleware
stage = os.environ.get('STAGE', None)
openapi_prefix = f"/{stage}" if stage else "/"
app = FastAPI(title="MyAwesomeApp",root_path="stage")
#app.get("/")
def get_root():
return {"message": "FastAPI running in a Lambda function"}
#app.get("/info")
def get_root():
return {"message": "TestInfo"}
app.add_middleware(
CORSMiddleware,
allow_origins=["*"],
allow_credentials=True,
allow_methods=["*"],
allow_headers=["*"],
)
handler = Mangum(app)
I tried adding the root as mentioned below
https://fastapi.tiangolo.com/advanced/behind-a-proxy/
Any help on this will be appreciated .
I found the solution we need to add root_path="/dev" like app = FastAPI(title=settings.NAME,root_path="/dev")
while creating fast api app .This root path should be same as the api gatway stage name .
However this approach does not work if you are using fastapi versioning
I have created a Google Cloud function that can be invoked through HTTP. The access to the function is limited to the Service account only.
If I had a Django View which should invoke this function and expect a response?
Here is what I have tried
1) Before starting Django I set the environment variable
export GOOGLE_APPLICATION_CREDENTIALS
2) I tried invoking the function using a standalone code, but soon realised this was going nowhere, because I could not figure out the next step after this.
from google.oauth2 import service_account
from apiclient.http import call
SCOPES = ['https://www.googleapis.com/auth/cloud-platform']
SERVICE_ACCOUNT_FILE = 'credentials/credentials.json'
credentials = service_account.Credentials.from_service_account_file(
SERVICE_ACCOUNT_FILE, scopes=SCOPES)
Google's documentation does give you documentation around the API, but there is no sample code on how to invoke the methods or what to import within your Python code and what are the ways to invoke those methods.
How do you send a POST request with JSON data to an Cloud Function, with authorization through a service account?
**Edit
A couple hours later I did some more digging and figured this out partially
from google.oauth2 import service_account
import googleapiclient.discovery
import json
SCOPES = ['https://www.googleapis.com/auth/cloud-platform']
SERVICE_ACCOUNT_FILE = 'credentials/credentials.json'
credentials = service_account.Credentials.from_service_account_file(
SERVICE_ACCOUNT_FILE, scopes=SCOPES)
cloudfunction = googleapiclient.discovery.build('cloudfunctions', 'v1', credentials=credentials)
#projects/{project_id}/locations/{location_id}/functions/{function_id}.
path='some project path'
data='some data in json that works when invoked through the console'
data=json.dumps(data)
a=cloudfunction.projects().locations().functions().call(name=path, body=data).execute()
I get another error now.
Details: "[{'#type': 'type.googleapis.com/google.rpc.BadRequest', 'fieldViolations': [{'description': 'Invalid JSON payload received. Unknown name "": Root element must be a message.'}]}]">
I cant find any documentation on this. How should the JSON be formed?
making the json like {"message":{my actual payload}} doesn't work.
The requested documentation can be found here.
The request body argument should be an object with the following form:
{ # Request for the `CallFunction` method.
"data": "A String", # Input to be passed to the function.
}
The following modification on your code should work correctly:
from google.oauth2 import service_account
import googleapiclient.discovery
SCOPES = ['https://www.googleapis.com/auth/cloud-platform']
SERVICE_ACCOUNT_FILE = 'credentials/credentials.json'
credentials = service_account.Credentials.from_service_account_file(
SERVICE_ACCOUNT_FILE, scopes=SCOPES)
cloudfunction = googleapiclient.discovery.build('cloudfunctions', 'v1', credentials=credentials)
path ="projects/your-project-name/locations/cloud-function-location/functions/name-of-cloud-function"
data = {"data": "A String"}
a=cloudfunction.projects().locations().functions().call(name=path, body=data).execute()
Notice that very limited traffic is allowed since there are limits to the API calls.
There are a few examples for the way to pre-sign the URL of an S3 request, but I couldn't find any working example to pre-sign other services in AWS.
I'm trying to write an item to DynamoDB using the Python SDK botos. The SDK included the option to generate the pre-signed URL here. I'm trying to make it work and I'm getting a URL, but the URL is responding with 404 and the Item is not appearing in the DynamoDB table.
import json
ddb_client = boto3.client('dynamodb')
response = ddb_client.put_item(
TableName='mutes',
Item={
'email': {'S':'g#g.c'},
'until': {'N': '123'}
}
)
print("PutItem succeeded:")
print(json.dumps(response, indent=4))
This code is working directly. But when I try to presign it:
ddb_client = boto3.client('dynamodb')
params = {
'TableName':'mutes',
'Item':
{
'email': {'S':'g#g.c'},
'until' : {'N': '1234'}
}
}
response = ddb_client.generate_presigned_url('put_item', Params = params)
and check the URL:
import requests
r = requests.post(response)
r
I'm getting: Response [404]
Any hint on how to get it working? I checked the IAM permissions, and they are giving full access to DynamoDB.
Please note that you can sign a request to DynamoDB using python, as you can see here: https://docs.aws.amazon.com/general/latest/gr/sigv4-signed-request-examples.html#sig-v4-examples-post . But for some reasons, the implementation in the boto3 library doesn't do that. Using the boto3 library is much easier than the code above, as I don't need to provide the credentials for the function.
You send an empty post request. You should add the data to the request:
import requests
r = requests.post(response, data = params)
I think you are having this issue, that's why you are recieving a 404.
They recommend using Cognito for authentication instead of IAM for this cases.
I am making an app in flutter which uses Google sign-in. I also have a Django backend linked to the app and I want to verify the user in the Django backend. I found many solutions on the internet but none is working. Probably I am messing up somewhere.
I tried using python-jose for verification and here is the code:
from jose import jwt
import urllib.request, json
token = '<token recieved using await user.getIdToken in flutter>'
target_audience = "<tried projectid/appid>"
certificate_url = 'https://www.googleapis.com/robot/v1/metadata/x509/securetoken#system.gserviceaccount.com'
response = urllib.request.urlopen(certificate_url)
certs = response.read()
certs = json.loads(certs)
print(certs)
user = jwt.decode(token, certs, algorithms='RS256',
audience=target_audience)
I also tried oauth2client, the code is here:
from oauth2client import crypt
import urllib.request, json
certificate_url = 'https://www.googleapis.com/robot/v1/metadata/x509/securetoken#system.gserviceaccount.com'
target_audience = 'tried projectid/appid'
response = urllib.request.urlopen(certificate_url)
certs = response.read()
certs = json.loads(certs)
print(certs)
crypt.MAX_TOKEN_LIFETIME_SECS = 30 * 86400
idtoken = 'token received from await user.getIdToken()'
crypt.verify_signed_jwt_with_certs(idtoken, certs, target_audience)
I also tried firebase_admin for python:
import firebase_admin
from firebase_admin import credentials
from firebase_admin import auth
cred = credentials.Certificate('<firebase service accounts private key>')
default_app = firebase_admin.initialize_app(cred)
token = 'token from flutter'
verifyied =auth.verify_id_token(id_token=token)
Just to check whether the firebase_admin library itself is working or not, I passed the userid to server from the app and tried deleting the user using firebase_admin and I could do that. But for some reason I am unable to verify the token.
Thanks for the help.
I have also faced the same issue.
Case:
Initially: I was printing auth token in vscode console and was verifying in terminal.
It gave me the error: token length cannot be 1 more than % 4.
I tried verifying the token from jwt.io and it was seemingly correct.
Actual Reason for the issue:
The console output of vscode (in my case windows 7 and 64 bit). Is limited to 1064 characters for a line.
Although the actual length of token is supposed to be 1170 characters.
Workaround Solution:
Print the substring in the vscode console and the join them in python shell to verify.
Answering my own question. The problem was that my server was not actually deployed, so, I was copying the token printed in vscode console when the user logs in and pasting it into the python code and trying to verify the token. Turns out it doesn't work that way.
I hosted my Django app and passed the token in a post request and then tried to verify the token and it worked.
You can refer the solutions here if you are stuck :
https://coders-blogs.blogspot.com/2018/11/authenticating-user-on-backend-of-your.html
I want to have a lambda calling a Sagemaker instance in another region. If both are in the same region, everything works fine. If they are not, I get the following error:
The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.
The Canonical String for this request should have been
'POST
/endpoints/foo-endpoint/invocations
host:runtime.sagemaker.us-east-1.amazonaws.com
x-amz-date:20180406T082536Z
host;x-amz-date
1234567890foobarfoobarfoobarboofoobarfoobarfoobarfoobarfoobarfoo'
The String-to-Sign should have been
'AWS4-HMAC-SHA256
20180406T082536Z
20180406/us-east-1/sagemaker/aws4_request
987654321abcdeffoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarf'
I use aws-requests-auth (0.4.1) with boto3 (1.5.15 - updating to 1.7.1 didn't change anything, changelog) like this:
import requests
from aws_requests_auth.aws_auth import AWSRequestsAuth
auth = AWSRequestsAuth(aws_access_key=config['AWS']['ACCESS_KEY'],
aws_secret_access_key=(
config['AWS']['SECRET_ACCESS_KEY']),
aws_host=config['AWS']['HOST'],
aws_region=config['AWS']['REGION'],
aws_service=config['AWS']['SERVICE'])
payload = {'foo': 'bar'}
response = requests.post(post_url,
data=json.dumps(payload),
headers={'content-type': 'application/json'},
auth=auth)
printing auth only gives <aws_requests_auth.aws_auth.AWSRequestsAuth object at 0x7f9d00c98390>.
Is there a way to print the "Canonical String" mentioned in the error message?
(Any other ideas how to fix this are appreciated as well)
A work-around for the asked question:
req = requests.request('POST', 'http://httpbin.org/get')
req.body = b''
req.method = ''
print(auth.get_aws_request_headers(req,
aws_access_key=auth.aws_access_key,
aws_secret_access_key=auth.aws_secret_access_key,
aws_token=auth.aws_token))
The problem is not solved, though. And now I wonder what the first argument of auth.get_aws_request_headers is.