I made a REST API with AWS Lambda+ API Gateway.
my API Gateway's Integration Request is LAMBDA_PROXY Type,
and I use params in Lambda like this. ( myparam is list type)
def lambda_handler(event, context):
# TODO implement
try:
myparam = event['multiValueQueryStringParameters']['param1']
#...
I tested my REST API in python like this.
url = 'https://***.amazonaws.com/default/myAPI'
param = {'param1':['1','2']}
res = requests.get(url=url,params=param).json()
print(res)
It works. but when I tried with another way like this,
url = 'https://***.amazonaws.com/default/myAPI?param1=1,2'
res = requests.get(url=url).json()
print(res)
It didn't work with this way.
How to query parameters in case if I want to insert parameter into url directly?
Those tow requests are not equivalent. In order to prove it, we can print the formatted URL for the first request:
url = 'https://***.amazonaws.com/default/myAPI'
param = {'param1':['1','2']}
res = requests.get(url=url,params=param).json()
# Print the request URL
print(res.request.url)
This will print something like:
https://***.amazonaws.com/myAPI?param1=1¶m1=2
So, in your second snippet, you probably would want to create your URL as follows:
url = 'https://***.amazonaws.com/myAPI?param1=1¶m1=2'
res = requests.get(url=url).json()
print(res)
If you want to separate your parameters with commas, the value for param1 will be a string ('1,2'), not an list.
Related
I am trying to figure out how to parse the URL parameters in GCP cloud functions. I am trying to do pretty much what this answer does. Since the entry point to a HTTP GCP cloud function takes a request AFAIK. I can's figure out a way to have Flask parse the URL parameters like in the example I mentioned.
The URL parameters isn't very clear. If you talk about query parameter (the params after the ? in the url), you can simply take the getting started example
def hello_world(request):
request_json = request.get_json()
# Here the query parameters
if request.args and 'message' in request.args:
return request.args.get('message')
# Here it's in the post body in JSON
elif request_json and 'message' in request_json:
return request_json['message']
else:
return f'Hello World!'
If you talk about path parameter, you can rely on the view_args value
# Function code
def entrypoint(request):
return request.view_args, 200
# Test like that
> curl https://us-central1-<PROJECT ID>.cloudfunctions.net/<FUNCTION NAME>/test/to/path
> {"path":"test/to/path"}
However, the path parameter isn't split for you, and you need to handle it manually.
EDIT 1
I found how to hack Flask and use its URL processor. Here my working code sample
# Function code
from flask import Flask
def entrypoint(request):
path = request.view_args['path']
f = Flask("internal")
f.add_url_rule("/test/<string:id>", "entrypoint_internal")
r = f.test_request_context(path=path)
r.push()
path_value = r.request.view_args
r.pop()
return path_value, 200
# Test like that
> curl https://us-central1-<PROJECT ID>.cloudfunctions.net/<FUNCTION NAME>/test/path
> {"id":"path"}
Concerning the following draft script I would like to know: How can I execute the named query I created?
I can access the query via the browser interface, but would like to execute it via the Session.
Here the answer is to use the client.start_query_execution(...) command. But whats the point when it is not the named query I created but instead a non-named query with the same query_string. Or am I missing something essential in how to use this?
import boto3
sess = boto3.session.Session(
region_name=region,
aws_access_key_id=ACCESS_KEY,
aws_secret_access_key=SECRET_KEY
)
athenaclient = sess.client('athena')
query_string = '''
SELECT *
FROM "ccindex"."ccindex"
WHERE crawl = 'CC-MAIN-2020-34'
AND subset = 'warc'
AND url_host_tld = 'de'
AND url_query IS NULL
AND url_path like '%impressum%'
LIMIT 20000
'''
resp = athenaclient.create_named_query(
Name='filter-ccindex-de',
Description='Filter *.de/impressum websites of Common Crawl index',
Database='ccindex',
QueryString=query_string
)
I don't think there is a direct option to pass named query to your start_query_execution method.But this can be achieved by using get_named_query which accepts Name of the named query and returns QueryString in response.
Then you can parse this response and pass QueryString to start_query_execution method.
I am using postman to hit my APIs. I have a question regarding sending query params through postman params. In my API i am getting params using services = request.GET.get('services') and then returning response for the services.
My Question is if have more than one service like 'A', 'B', 'C', then how can we send these services in params using postman?
views.py
class SomeAPIView(ModelViewSet):
def get_queryset(self):
services = self.request.GET.get('services')
print(services) # getting services
print(type(services)) #type is string
response_list = []
for service in services:
result = API(service=service)
response_list.append(result)
return response_list
I want get list of services and then iterate over that list to return response for that service.
It depending on how you would use this API entry in production.
At first, the good way in django is to use request.query_params to obtain query parameters. Also you must provide default value to get() method to avoid exceptions if there's no 'services' parameter passed.
Then, if your services parameter contain names or ids of some objects, you may just pass it with parameters in GET request as http://someurl?services=A,B,C, or within tab, named 'Params' in postman. So request.query_params.get('sevices', '') will return string, contains 'A,B,C'. Now you can split it by ',' like services_names = str.split(',').
Anyway, parameters of GET requests may return only str values.
According to your example, it may looks like:
class SomeAPIView(ModelViewSet):
def get_queryset(self):
services = self.request.query_params.get('sevices', '').split(',')
print(services) # getting services
print(type(services)) # now it will be List[str]
response_list = []
for service in services:
result = API(service=service)
response_list.append(result)
return response_list
I create a pre-signed URL and get back something like
https://s3.amazonaws.com/MyBucket/MyItem/
?X-Amz-Security-Token=TOKEN
&X-Amz-Algorithm=AWS4-HMAC-SHA256
&X-Amz-Date=20171206T014837Z
&X-Amz-SignedHeaders=host
&X-Amz-Expires=3600
&X-Amz-Credential=CREDENTIAL
&X-Amz-Signature=SIGNATURE
I can now curl this no problem. However, if I now add another query parameter, I will get back a 403, i.e.
https://s3.amazonaws.com/MyBucket/MyItem/
?X-Amz-Security-Token=TOKEN
&X-Amz-Algorithm=AWS4-HMAC-SHA256
&X-Amz-Date=20171206T014837Z
&X-Amz-SignedHeaders=host
&X-Amz-Expires=3600
&X-Amz-Credential=CREDENTIAL
&X-Amz-Signature=SIGNATURE
&Foo=123
How come? Is it possible to generate a pre-signed url that supports custom queries?
It seems to be technically feasible to insert custom query parameters into a v4 pre-signed URL, before it is signed, but not all of the AWS SDKs expose a way to do this.
Here's an example of a roundabout way to do this with the AWS JavaScript SDK:
const AWS = require('aws-sdk');
var s3 = new AWS.S3({region: 'us-east-1', signatureVersion: 'v4'});
var req = s3.getObject({Bucket: 'mybucket', Key: 'mykey'});
req.on('build', () => { req.httpRequest.path += '?session=ABC123'; });
console.log(req.presign());
I've tried this with custom query parameters that begin with X- and without it. Both appeared to work fine. I've tried with multiple query parameters (?a=1&b=2) and that worked too.
The customized pre-signed URLs work correctly (I can use them to get S3 objects) and the query parameters make it into CloudWatch Logs so can be used for correlation purposes.
Note that if you want to supply a custom expiration time, then do it as follows:
const Expires = 120;
const url = req.presign(Expires);
I'm not aware of other (non-JavaScript) SDKs that allow you to insert query parameters into the URL construction process like this so it may be a challenge to do this in other languages. I'd recommend using a small JavaScript Lambda function (or API Gateway plus Lambda function) that would simply create and return the customized pre-signed URL.
The custom query parameters are also tamper-proof. They are included in the signing of the URL so, if you tamper with them, the URL becomes invalid, yielding 403 Forbidden.
I used this code to generate your pre-signed URL. The result was:
https://s3.amazonaws.com/MyBucket/MyItem
?Foo=123
&X-Amz-Algorithm=AWS4-HMAC-SHA256
&X-Amz-Credential=AKIA...27%2Fus-east-1%2Fs3%2Faws4_request
&X-Amz-Date=20180427T0012345Z
&X-Amz-Expires=3600
&X-Amz-Signature=e3...7b
&X-Amz-SignedHeaders=host
None of this is a guarantee that this technique will continue to work, of course, if AWS changes things under the covers but for right now it seems to work and is certainly useful.
Attribution: the source of this discovery was aws-sdk-js/issues/502.
If you change one of the headers or add / subtract, then you have to resign the URL.
This is part of the AWS signing design and this process is designed for higher levels of security. One of the AWS reasons for changing to signing version 4 from signing version 2.
The signing design does not know which headers are important and which are not. That would create a nightmare trying to track all of the AWS services.
I created this solution for Ruby SDK. It is sort of a hack, but it works as expected:
require 'aws-sdk-s3'
require 'active_support/core_ext/object/to_query.rb'
# Modified S3 pre signer class that can inject query params to the URL
#
# Usage example:
#
# bucket_name = "bucket_name"
# key = "path/to/file.json"
# filename = "download_file_name.json"
# duration = 3600
#
# params = {
# bucket: bucket_name,
# key: key,
# response_content_disposition: "attachment; filename=#{filename}",
# expires_in: duration
# }
#
# signer = S3PreSignerWithQueryParams.new({'x-your-custom-field': "banana", 'x-some-other-field': 1234})
# url = signer.presigned_url(:get_object, params)
#
# puts "url = #{url}"
#
class S3PreSignerWithQueryParams < Aws::S3::Presigner
def initialize(query_params = {}, options = {})
#query_params = query_params
super(options)
end
def build_signer(cfg)
signer = super(cfg)
my_params = #query_params.to_h.to_query()
signer.define_singleton_method(:presign_url,
lambda do |options|
options[:url].query += "&" + my_params
super(options)
end)
signer
end
end
While not documented, you can add parameters as arguments to the call to presigned_url.
obj.presigned_url(:get,
expires_in: expires_in_sec,
response_content_disposition: "attachment"
)
https://bucket.s3.us-east-2.amazonaws.com/file.txt?response-content-disposition=attachment&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=PUBLICKEY%2F20220309%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20220309T031958Z&X-Amz-Expires=43200&X-Amz-SignedHeaders=host&X-Amz-Signature=SIGNATUREVALUE
If you are looking on for JavaScript SDK V3:
import { HttpRequest } from "#aws-sdk/protocol-http";
import { S3RequestPresigner } from "#aws-sdk/s3-request-presigner";
import { parseUrl } from "#aws-sdk/url-parser";
import { Sha256 } from "#aws-crypto/sha256-browser";
import { Hash } from "#aws-sdk/hash-node";
import { formatUrl } from "#aws-sdk/util-format-url";
// Make custom query in Record<string, string | Array<string> | null> format
const customQuery = {
hello: "world",
};
const s3ObjectUrl = parseUrl(
`https://${bucketName}.s3.${region}.amazonaws.com/${key}`
);
s3ObjectUrl.query = customQuery; //Insert custom query here
const presigner = new S3RequestPresigner({
credentials,
region,
sha256: Hash.bind(null, "sha256"), // In Node.js
//sha256: Sha256 // In browsers
});
// Create a GET request from S3 url.
const url = await presigner.presign(new HttpRequest(s3ObjectUrl));
console.log("PRESIGNED URL: ", formatUrl(url));
Code template taken from: https://aws.amazon.com/blogs/developer/generate-presigned-url-modular-aws-sdk-javascript/
When trying to pull data from the youtube gdata api using urllib2.urlopen, I receive a HTTP 403 error. I've turned off the CSRF middleware for debugging purposes, and the view I'm using looks like this:
def videos (request):
params = {}
youtube_search_url = 'http://gdata.youtube.com/feeds/api/videos'
params['order_by'] = 'relevance'
params['max_results'] = 10
params['safeSearch'] = 'strict'
params['v'] = 2
params['key'] = '<developer key>'
f = urllib2.urlopen(youtube_search_url, encoded_params)
...
Any ideas?
When you make an API request, use the X-GData-Key request header to specify your developer key as shown in the following example:
X-GData-Key: key=<developer_key>
Include the key query parameter in the request URL.
http://gdata.youtube.com/feeds/api/videos?q=SEARCH_TERM&key=DEVELOPER_KEY
^^ Straight from the horse's mouth. You are missing the X-GData-Key request header.
The key seems to be required in both url and the header, so given your previous code try this:
req = urllib2.Request(youtube_search_url, encoded_params, { "X-GData-Key": '<developer key>' })
f = urllib2.urlopen(req)