Python Requests - URL changes between params as text vs params as dictionary - python-2.7

I'm using Python Requests to call an API using GET
I generate a signature using hmac and hashlib which results in a signature that looks like:
4epwTDKhWcIJL6bMM5f2hmBrOoXXIGD9UwX8ErfYzqU%3D
When I use Requests and specify the params as a string, the API call is successful because the signature is used as-is and doesn't change.
However when I use Requests and specify the params as a dictionary, the signature above is somehow URL encoded again, resulting in the % being encoded to %25, causing the signature to display as follows (note the %25):
4epwTDKhWcIJL6bMM5f2hmBrOoXXIGD9UwX8ErfYzqU%253D
For more context, here is the string params:
url = 'https://example.com/API'
payload = '&parameterA=valueA&parameterB=valueB&parameterC=valueC& apikey='+apikey+'&salt='+salt+'&signature='+sig
#payload = {'parameterA': 'valueA', 'parameterB': 'valueB', 'apikey': apikey, 'salt': salt, 'signature': sig}
r = requests.get(url, params=payload)
print r.url
results in a URL of:
https://example.com/API&
&parameterA=valueA
&parameterB=valueB
&apikey=0e1026af-40ce-e354-f1f2-72d280ca122
&salt=12345
&signature=4epwTDKhWcIJL6bMM5f2hmBrOoXXIGD9UwX8ErfYzqU%3D
Now, the dictionary params:
url = 'https://example.com/API'
payload = {'parameterA': 'valueA', 'parameterB': 'valueB', 'apikey': apikey, 'salt': salt, 'signature': sig}
r = requests.get(url, params=payload)
print r.url
results in a URL of:
https://example.com/API
&signature=4epwTDKhWcIJL6bMM5f2hmBrOoXXIGD9UwX8ErfYzqU%253D
&salt=12345
&apikey=0e1026af-40ce-e354-f1f2-72d280ca122
&parameterA=valueA
&parameterB=valueB
Note again that the % in the signature has changed to %25
Thanks!

When passing a dictionary to params it gets url encoded. Your string is url encoded already so it gets double encoded, resulting in a malformed 'signature'.
You can use urllib.unquote to decode sig (or you can just replace '%3D' with '=')
payload = {
'parameterA': 'valueA', 'parameterB': 'valueB',
'apikey': apikey, 'salt': salt, 'signature': urllib.unquote(sig)
}
Note that in python3 unquote is located in urllib.parse.

Related

How to query parameters with API Gateway to lambda

I made a REST API with AWS Lambda+ API Gateway.
my API Gateway's Integration Request is LAMBDA_PROXY Type,
and I use params in Lambda like this. ( myparam is list type)
def lambda_handler(event, context):
# TODO implement
try:
myparam = event['multiValueQueryStringParameters']['param1']
#...
I tested my REST API in python like this.
url = 'https://***.amazonaws.com/default/myAPI'
param = {'param1':['1','2']}
res = requests.get(url=url,params=param).json()
print(res)
It works. but when I tried with another way like this,
url = 'https://***.amazonaws.com/default/myAPI?param1=1,2'
res = requests.get(url=url).json()
print(res)
It didn't work with this way.
How to query parameters in case if I want to insert parameter into url directly?
Those tow requests are not equivalent. In order to prove it, we can print the formatted URL for the first request:
url = 'https://***.amazonaws.com/default/myAPI'
param = {'param1':['1','2']}
res = requests.get(url=url,params=param).json()
# Print the request URL
print(res.request.url)
This will print something like:
https://***.amazonaws.com/myAPI?param1=1&param1=2
So, in your second snippet, you probably would want to create your URL as follows:
url = 'https://***.amazonaws.com/myAPI?param1=1&param1=2'
res = requests.get(url=url).json()
print(res)
If you want to separate your parameters with commas, the value for param1 will be a string ('1,2'), not an list.

Using requests to POST to pastebin API

I simply cannot get this to work.
header = {"Content-Type": "application/json; charset=utf8"}
params = {"api_dev_key": dev_key, "api_user_name": username, "api_user_password": password}
req = requests.post("http://pastebin.com/api/api_login.php", params = json.dumps(params), headers = header)
print(req.status_code, req.reason, req.text)
The variables (my credentials) are just strings.
The response I get:
(200, 'OK', u'Bad API request, invalid api_dev_key')
There's nothing wrong with the key, this POST works fine when I use https://www.hurl.it
You need to just use data=params:
req = requests.post("http://pastebin.com/api/api_login.php", data=params)

AWSAccessKeyId not authorized

I am attempting to use a GET request to use the Amazon Mechanical Turk GetFileUploadURL function. However, I get this error code when attempting it.
AWS.NotAuthorized The identity contained in the request is not authorized to use this AWSAccessKeyId
This is the code I'm using to create the request.
now = DateTime.now
#For creating the signature hash
data = "AWSMechanicalTurkRequesterGetFileUploadURL" + now.to_s
sha256 = OpenSSL::Digest::SHA256.new
sig = OpenSSL::HMAC.digest(sha256, Rails.configuration.secret_key, data)
signature = Base64.encode64(sig)
puts "https://mechanicalturk.amazonaws.com", "/?Service=AWSMechanicalTurkRequester&AWSAccessKeyId=#{Rails.configuration.aws_key}&Version=2014-08-15&Operation=GetFileUploadURL&Signature=#{sig}&Timestamp=#{now}&AssignmentId=#{mturk_results[0][:AssignmentId]}&QuestionIdentifier=file1"
puts "\n\n"
uri = URI('https://mechanicalturk.amazonaws.com')
params = {:Service=>"AWSMechanicalTurkRequester", :AWSAccessKeyId=>Rails.configuration.aws_key, :Version=>"2014-08-15", :Operation=>"GetFileUPloadURL", :Signature=>sig, :Timestamp=>now, :AssignmentId=>mturk_results[0][:AssignmentId], :QuestionIdentifier=>"file1"}
uri.query = URI.encode_www_form(params)
res = 0
Net::HTTP.start(uri.host, uri.port,
:use_ssl => uri.scheme == 'https') do |http|
request = Net::HTTP::Get.new uri
res = http.request request # Net::HTTPResponse object
end
Any ideas as to what I'm doing wrong?
You'll need to format your TimeStamp correctly. The time stamp must be in UTC and in the following ISO 8601 format:
YYYYMMDD'T'HHMMSS'Z'. For example, 20150830T123600Z is a valid time stamp. Do not include milliseconds in the time stamp.
It needs to be of the form:
Timestamp=2016-04-23T08:00:05Z
While Ruby's DateTime.now method returns them of the form:
2016-08-05T10:43:27-07:00
You can read more about Timestamp and AWS signatures here:
http://docs.aws.amazon.com/general/latest/gr/sigv4-date-handling.html

OAuth1Session Encoding of URL Django

Alright, I need some assistance with what I think is an easy question, but after digging for quite a while I'll leave it to you brilliant people to show me why I'm not!
I'm attempting to access a provider api, with the URL I'm attempting to access being: .../proposals/AnotherTestProposal/city_furniture/items?filter=description$CONT"Ag" :: this is the way they want it passed, and it should return all items who's descriptions contain the string "Ag". I know I have two of those currently.
I'm using Django 1.9.4, and requests_oauthlib to create an OAuth1Session, I'm doing this successfully also, because I can access resources without URL params. The trouble is I can't get the "?filter=description..." part to encode properly and it's giving me a 401.
When I render the .contents to HTML I get:
{"status": 404, "message": "", "data": [], "arguments": {"id": "items?filter=description$CONT\"Ag\"", "collection": "city_furniture"}}
which is telling telling me that the "AG" part is being escaped, which I don't want. As in: I don't want \"Ag\", I want "Ag".
So, my question: how can I pass the URL, with params, so they are not containing the slashes as these are causing the URL to be invalid and therefore keeping me from accessing the data correctly?
Other, possibly irrelevant info:
The params part of the URL string I'm passing to the OAuth1Session object now is: '/city_furniture/items%3Ffilter%3Ddescription%24CONT%22Ag%22'
An example of filtering from the API's website: proposals/master/roads/items?filter=description$CONT"highway"
I tried passing an 'encoding' arg to .get (in order to change the encoding used by to_native_string) but requests rejected it saying it was an invalid arg
Per comments, some additional code.
Using a function name get_protected_code() to get the OAuth info passed correctly, then in views.py:
api_filter_url = settings.IW_API_MODEL_COLLECTION + '/' + model_id + '/' + settings.IW_API_PROPOSAL_COLLECTION + '/' + proposal_name + '/city_furniture/items%3Ffilter%3Ddescription%24CONT%22Ag%22'
json_model_info_pull = get_protected_data(api_filter_url)
find_vendor = json_model_info_pull.content
def get_protected_data(url_str):
## ...stuffs to create OAuth1Session...
adsk_pull = OAuth1Session(key,
client_secret=secret,
resource_owner_key=oauth_token_use,
resource_owner_secret=oauth_token_secret_use,
)
headers = {'accept': 'application/vnd.autodesk.infraworks-v1+json'}
api_url = settings.ADSK_IW_INFO_BASE_URL + url_str
json_model_info_pull = adsk_pull.get(api_url, headers=headers)
return json_model_info_pull
Looks like you're passing the parameters incorrectly by appending them to the end of the URL in URL-encoding, which requests is respecting as intentional, and the API endpoint is translating in an unintended manner.
From the documentation for requests, you should provide a params keyword argument to requests.get: a dict containing the key-value pairs that should be encoded and sent as the query string for the request. For example, to run a query against the GitHub API, we can pass an API token as a query parameter:
requests.get('http://api.github.com/user',
params={ 'access_token' : MY_OAUTH_TOKEN })
The resultant request will contain a query string with an access_token parameter set to the value stored in MY_OAUTH_TOKEN and escaped properly, as needed. (Such tokens typically contain = characters, for example, that are invalid inside query string values.)

POST SOAP request in python getting error Response: 403 Forbidden

Please can anyone help out? i am trying to POST a SOAP request in python but i am getting the error Response 403: Forbidden. my code looks like below:
i am using the python imports:
import httplib
import base64
import string
#the message
message = """<soap:Envelope ...rest message </soap:Envelope>"""
host = "host.test.com"
url = 'https://server.etc.com' #End point url
i need to use the Basic Authentication too so i need the username and password in the http header
username = 'username'
password = 'password'
webSoapAction = 'urn:etc-com:document...'
#the Authentication in base64 encoding form for Basic Authentication
auth = 'Basic' + string.strip(base64.encodestring(username +'.'+ password))
webservice = httplib.HTTP(host) #connect to the server(host)
here i try to build the header:
webservice.putrequest("POST", url)
webservice.putheader("Host", host)
webservice.putheader("User-Agent", "Python http auth")
webservice.putheader("Content-Type:", "text/xml; charset=\"UTF-8\"")
webservice.putheader("Content-length", "%d" % len(message))
webservice.putheader("SOAPAction",webSoapAction)
webservice.putheader('Authorization', auth)
webservice.endheaders()
webservice.send(message)
i should get the response here
#get the response
statuscode, statusmessage, header = webservice.getreply()
print "Response: ", statuscode, statusmessage
print "Headers: ",header
res = webservice.getfile().read()
print 'Content: ', res
Two things regard to your basic auth header construction:
Put a single space between "Basic" and your secret
Use ':' instead of '.' in between username and password
So it should looks like:
#the Authentication in base64 encoding form for Basic Authentication
auth = 'Basic ' + string.strip(base64.encodestring(username +':'+ password))