Cloud Functions parsing request path parameters - google-cloud-platform

I am trying to figure out how to parse the URL parameters in GCP cloud functions. I am trying to do pretty much what this answer does. Since the entry point to a HTTP GCP cloud function takes a request AFAIK. I can's figure out a way to have Flask parse the URL parameters like in the example I mentioned.

The URL parameters isn't very clear. If you talk about query parameter (the params after the ? in the url), you can simply take the getting started example
def hello_world(request):
request_json = request.get_json()
# Here the query parameters
if request.args and 'message' in request.args:
return request.args.get('message')
# Here it's in the post body in JSON
elif request_json and 'message' in request_json:
return request_json['message']
else:
return f'Hello World!'
If you talk about path parameter, you can rely on the view_args value
# Function code
def entrypoint(request):
return request.view_args, 200
# Test like that
> curl https://us-central1-<PROJECT ID>.cloudfunctions.net/<FUNCTION NAME>/test/to/path
> {"path":"test/to/path"}
However, the path parameter isn't split for you, and you need to handle it manually.
EDIT 1
I found how to hack Flask and use its URL processor. Here my working code sample
# Function code
from flask import Flask
def entrypoint(request):
path = request.view_args['path']
f = Flask("internal")
f.add_url_rule("/test/<string:id>", "entrypoint_internal")
r = f.test_request_context(path=path)
r.push()
path_value = r.request.view_args
r.pop()
return path_value, 200
# Test like that
> curl https://us-central1-<PROJECT ID>.cloudfunctions.net/<FUNCTION NAME>/test/path
> {"id":"path"}

Related

How to query parameters with API Gateway to lambda

I made a REST API with AWS Lambda+ API Gateway.
my API Gateway's Integration Request is LAMBDA_PROXY Type,
and I use params in Lambda like this. ( myparam is list type)
def lambda_handler(event, context):
# TODO implement
try:
myparam = event['multiValueQueryStringParameters']['param1']
#...
I tested my REST API in python like this.
url = 'https://***.amazonaws.com/default/myAPI'
param = {'param1':['1','2']}
res = requests.get(url=url,params=param).json()
print(res)
It works. but when I tried with another way like this,
url = 'https://***.amazonaws.com/default/myAPI?param1=1,2'
res = requests.get(url=url).json()
print(res)
It didn't work with this way.
How to query parameters in case if I want to insert parameter into url directly?
Those tow requests are not equivalent. In order to prove it, we can print the formatted URL for the first request:
url = 'https://***.amazonaws.com/default/myAPI'
param = {'param1':['1','2']}
res = requests.get(url=url,params=param).json()
# Print the request URL
print(res.request.url)
This will print something like:
https://***.amazonaws.com/myAPI?param1=1&param1=2
So, in your second snippet, you probably would want to create your URL as follows:
url = 'https://***.amazonaws.com/myAPI?param1=1&param1=2'
res = requests.get(url=url).json()
print(res)
If you want to separate your parameters with commas, the value for param1 will be a string ('1,2'), not an list.

How to solve the URL decoding problem in Django and nginx

My web app is deployed using nginx. I have view like below for the url /incoming/`.
def incoming_view(request):
incoming = request.GET["incoming"]
user = request.GET["user"]
...
When I hit my url /incoming/?incoming=hello&user=nkishore I am getting the response I need. but when I call this url using requests module with below code I am getting an error.
r = requests.get('http://localhost/incoming/?incoming=%s&user=%s'%("hello", "nkishore"))
print r.json()
I have checked the nginx logs and the request I got was /incoming/?incoming=hiu0026user=nkishore so in my view request.GET["user"] is failing to get user.
I am not getting what I am missing here, is it a problem with nginx or any other way to call in requests.
See Requests Docs for how to pass parameters, e.g.
>>> payload = {'key1': 'value1', 'key2': 'value2'}
>>> r = requests.get('https://httpbin.org/get', params=payload)
>>> print(r.url)
https://httpbin.org/get?key2=value2&key1=value1
Internally, Requests will likely escape the & ampersand with &. If you really want to do the URL manually, try as your URL string:
'http://localhost/incoming/?incoming=%s&user=%s'

Wierd Behavior When Using Python requests.put() with Flask

Background
I have a service A accessible with HTTP requests. And I have other services that want to invoke these APIs.
Problem
When I test service A's APIs with POSTMAN, every request works fine. But when I user python's requests library to make these request, there is one PUT method that just won't work. For some reason, the PUT method being called cannot receive the data (HTTP body) at all, though it can receive headers. On the other side, the POST method called in the same manner receives the data perfectly.
I managed to achieve my goal simply by using httplib library instead, but I am still quite baffled by what exactly happened here.
The Crime Scene
Route 1:
#app.route("/private/serviceA", methods = ['POST'])
#app.route("/private/serviceA/", methods = ['POST'])
def A_create():
# request.data contains correct data that can be read with request.get_json()
Route 2:
#app.route("/private/serviceA/<id>", methods = ['PUT'])
#app.route("/private/serviceA/<id>/", methods = ['PUT'])
def A_update(id):
# request.data is empty, though request.headers contains headers I passed in
# This happens when sending the request with Python requests library, but not when sending with httplib library or with POSTMAN
# Also, data comes in fine when all other routes are commented out
# Unless all other routes are commented out, this happens even when the function body has only one line printing request.data
Route 3:
#app.route("/private/serviceA/schema", methods = ['PUT'])
def schema_update_column():
# This one again works perfectly fine
Using POSTMAN:
Using requests library from another service:
#app.route("/public/serviceA/<id>", methods = ['PUT'])
def A_update(id):
content = request.get_json()
headers = {'content-type': 'application/json'}
response = requests.put('%s:%s' % (router_config.HOST, serviceA_instance_id) + '/private/serviceA/' + str(id), data=json.dumps(content), headers = headers)
return Response(response.content, mimetype='application/json', status=response.status_code)
Using httplib library from another service:
#app.route('/public/serviceA/<id>', methods=['PUT'])
def update_course(id):
content= request.get_json()
headers = {'content-type': 'application/json'}
conn = httplib.HTTPConnection('%s:%s' % (router_config.HOST, serviceA_instance_id))
conn.request("PUT", "/private/serviceA/%s/" % id, json.dumps(content), headers)
return str(conn.getresponse().read())
Questions
1. What am I doing wrong for the route 2?
2. For route 2, the handler doesn't seem to be executed when either handler is commented out, which also confuses me. Is there something important about Flask that I'm not aware of?
Code Repo
Just in case some nice ppl are interested enough to look at the messy undocumented code...
https://github.com/fantastic4ever/project1
The serviceA corresponds to course service (course_flask.py), and the service calling it corresponds to router service (router.py).
The version that was still using requests library is 747e69a11ed746c9e8400a8c1e86048322f4ec39.
In your use of the requests library, you are using requests.post, which is sending a POST request. If you use requests.put then you would send a PUT request. That could be the issue.
Request documentation

is there any equivalent code that get buckets from google storage faster

this the code I'm using, is there anyway to make it run faster:
src_uri = boto.storage_uri(bucket, google_storage)
for obj in src_uri.get_bucket():
f.write('%s\n' % (obj.name))
This is an example where it pays to use the underlying Google Cloud Storage API more directly, using the Google API Client Library for Python to consume the RESTful HTTP API. With this approach, it is possible to use request batching to retrieve the names of all objects in a single HTTP request (thereby reducing the extra HTTP request overhead) as well as to use field projection with the objects.get operation (by setting &fields=name) to obtain a partial response so that you aren't sending all the other fields and data over the network (or waiting for retrieval of unnecessary data on the backend).
Code for this would look like:
def get_credentials():
# Your code goes here... checkout the oauth2client documentation:
# http://google-api-python-client.googlecode.com/hg/docs/epy/oauth2client-module.html
# Or look at some of the existing samples for how to do this
def get_cloud_storage_service(credentials):
return discovery.build('storage', 'v1', credentials=credentials)
def get_objects(cloud_storage, bucket_name, autopaginate=False):
result = []
# Actually, it turns out that request batching isn't needed in this
# example, because the objects.list() operation returns not just
# the URL for the object, but also its name, as well. If it had returned
# just the URL, then that would be a case where we'd need such batching.
projection = 'nextPageToken,items(name,selfLink)'
request = cloud_storage.objects().list(bucket=bucket_name, fields=projection)
while request is not None:
response = request.execute()
result.extend(response.items)
if autopaginate:
request = cloud_storage.objects().list_next(request, response)
else:
request = None
return result
def main():
credentials = get_credentials()
cloud_storage = get_cloud_storage_service(credentials)
bucket = # ... your bucket name ...
for obj in get_objects(cloud_storage, bucket, autopaginate=True):
print 'name=%s, selfLink=%s' % (obj.name, obj.selfLink)
You may find the Google Cloud Storage Python Example and other API Client Library Examples helpful in figuring out how to do this. There are also a number of YouTube videos on the Google Developers channel such as Accessing Google APIs: Common code walkthrough that provide walkthroughs.

Setting HTTP_REFERER header in Django test

I'm working on a Django web application which (amongst other things) needs to handle transaction status info sent using a POST request.
In addition to the HTTP security supported by the payment gateway, my view checks request.META['HTTP_REFERER'] against an entry in settings.py to try to prevent funny business:
if request.META.get('HTTP_REFERER', '') != settings.PAYMENT_URL and not settings.DEBUG:
return HttpResponseForbidden('Incorrect source URL for updating payment status')
Now I'd like to work out how to test this behaviour.
I can generate a failure easily enough; HTTP_REFERER is (predictably) None with a normal page load:
def test_transaction_status_succeeds(self):
response = self.client.post(reverse('transaction_status'), { ... })
self.assertEqual(response.status_code, 403)
How, though, can I fake a successful submission? I've tried setting HTTP_REFERER in extra, e.g. self.client.post(..., extra={'HTTP_REFERER': 'http://foo/bar'}), but this isn't working; the view is apparently still seeing a blank header.
Does the test client even support custom headers? Is there a work-around if not? I'm using Django 1.1, and would prefer not to upgrade just yet if at all possible.
Almost right. It's actually:
def transaction_status_suceeds(self):
response = self.client.post(reverse('transaction_status'), {}, HTTP_REFERER='http://foo/bar')
I'd missed a ** (scatter operator / keyword argument unpacking operator / whatever) when reading the source of test/client.py; extra ends up being a dictionary of extra keyword arguments to the function itself.
You can pass HTTP headers to the constructor of Client:
from django.test import Client
from django.urls import reverse
client = Client(
HTTP_USER_AGENT='Mozilla/5.0',
HTTP_REFERER='http://www.google.com',
)
response1 = client.get(reverse('foo'))
response2 = client.get(reverse('bar'))
This way you don't need to pass headers every time you make a request.