How to get the file name from a simple curl post request in aws lambda throu aws API gateway - amazon-web-services

I have the following curl request,
curl -X POST "https://foo123.execute-api.us-east-1.amazonaws.com/default/test-2" \
-H "accept: application/json" \
-H "Content-Type: multipart/form-data" \
-F "file=#wiki.png;type=image/png"
Due to the existing pipeline structure, I am not able to change the present architecture of this curl request, I want to access the name of this file in the lambda function when I am using aws API gateway as the trigger.
I am aware I can simply send the file name in header but that doesn't work for the pipeline I am going to use it with, so my question is.
How can I access the name of this file inside a lambda function in aws, given that I can't change my curl request.
To further give clarity to my question in flask APIs we are able to get the name of this uploaded file by simply using something like.
args = upload_parser.parse_args()
uploaded_file = args['file']
required_file_name = uploaded_file.filename

Interesting problem, actually you get the values of the body as a base64 encoded image you will have to either use any of the existing libraries to parse it for the name and content from body or write your own parser, your can refer to the code snippet below for reference :
import cv2
import os
import base64
import numpy as np
import email
def http_api(event):
post_data = base64.b64decode(event['body'])
# fetching content-type
try:
content_type = event["headers"]['Content-Type']
except:
content_type = event["headers"]['content-type']
# concate Content-Type: with content_type from event
ct = "Content-Type: "+content_type+"\n"
# parsing message from bytes
msg = email.message_from_bytes(ct.encode()+post_data)
if msg.is_multipart():
multipart_content = {}
# retrieving form-data
for part in msg.get_payload():
# checking if filename exist as a part of content-disposition header
if part.get_filename():
# fetching the filename
file_name = part.get_filename()
multipart_content[part.get_param('name', header='content-disposition')] = part.get_payload(decode=True)
img_str = multipart_content["file"]
nparr = np.fromstring(img_str, np.uint8)
image = cv2.imdecode(nparr, cv2.IMREAD_COLOR)
return image

Related

How to publish to a GCP pub/sub topic using HTTP Bridge in Python3 & CURL?

I am trying to publish to a pub/sub topic via HTTP Bridge using python3 & CURL.
**Python3**
import json
import logging
import os
import socket
import sys
import time
import requests
URL = 'https://cloudiotdevice.googleapis.com/v1/projects/{}/locations/{}/registries/{}/devices/{}:publishEvent'
JWT = 'JWT'
def main():
if not URL or not JWT:
sys.exit("Are the Environment Variables set?")
get_sensor_data(socket.gethostname())
def get_sensor_data(device_id):
while True:
print("in get_sensor data")
payload = {'device': str('asd'),
'type': str('adssaff'),
'timestamp': str(time.time()),
'data': json.dumps({'temperature': str('23'),
'humidity': str('442')})}
post_data(payload)
print("data printed")
time.sleep(5)
def post_data(payload):
payload = json.dumps(payload)
headers = {
'Content-Type': 'application/json; charset=utf-8',
'Authorization': JWT
}
try:
req = requests.post(URL, json=str(payload), headers=headers)
print("request Successfull "+str(req))
except requests.exceptions.ConnectionError:
logging.error('Error posting data to Cloud Function!')
except requests.exceptions.MissingSchema:
logging.error('Error posting data to Cloud Function! Are Environment Variables set?')
if __name__ == '__main__':
This is giving an error 400 because i think i havent described the subfolder.
Now i am confuse that where can i define the subfolder(Topic name) in my code?
and is there only subfolder is missing? or i am doing something else wrong too?
CURL
i also tried using the CURL command described in
https://cloud.google.com/iot/docs/how-tos/http-bridge
The command is
curl -X POST -H 'authorization: Bearer JWT' -H 'content-type: application/json' --data '{"binary_data": "DATA", "sub_folder": "SUBFOLDER"}' -H 'cache-control: no-cache' 'https://cloudiotdevice.googleapis.com/v1/projects/{project-id}/locations/{cloud-region}/registries/{registry-id}/devices/{device-id}:publishEvent'
It triggers my cloud function which means the authorization works but i am not able to see "DATA" in my logs. which i assume i am not providing the right format for the binary_data. whyt would be the right format if i would like to publish 'payload' described above using curl too?
It looks like you are using a JSON payload with the data field set to an object, rather than binary string. Try to either json.dumps the object in the 'data' field or send the 'data' field as a string.
From this documentation.
https://cloud.google.com/iot/docs/reference/cloudiotdevice/rest/v1/projects.locations.registries.devices/publishEvent
I found out that my payload request body was not correct.
so payload should look like this below ..
s= json.jumps('json object')
payload = {"subFolder": 'Sub_FOLDER_NAME', "binaryData": base64.b64encode(s.encode('utf-8'))}

Flask passing uploaded file to another service using requests

I have Python flask webservice that takes in a file:
Headers:
Content-type: multipart/formdata
Content:
"fileTest": UPLOADED FILE
When I pass the file to another service using requests lib, I get issue where the uploaded file is not passed.
My Code:
files = {}
for form_file_param in request.files:
fs = request.files[form_file_param] # type: FileStorage
files[form_file_param] = (fs.filename, fs.read())
req_headers = {
"content-type": u "multipart/form-data; boundary=X-INSOMNIA-BOUNDARY",
}
r = requests.request(method='POST',
url=url,
headers=req_headers,
files=files)
I contact my other service directly through postman and it works successfully. I cannot seem to figure out what I am doing wrong in the above code.
You need to follow requests document.
http://docs.python-requests.org/en/master/user/quickstart/#post-a-multipart-encoded-file
url = 'https://httpbin.org/post'
files = {'file': ('report.xls', open('report.xls', 'rb'), 'application/vnd.ms-excel', {'Expires': '0'})}
r = requests.post(url, files=files)
r.text
Change . after watching OP response , the issue caused by header - Content-Type.
This is a special content type which can be visualized as multiple sub-requests in one big request. Each of those sub-requests (one form-data element) has their own set of headers. The content type of the actual data is in there.1
Note : there are no different between fs and fs.read()
#models.py line 149
if isinstance(fp, (str, bytes, bytearray)):
fdata = fp
else:
data = fp.read()

Convert curl get command to urllib get request

I have a curl request that I want to convert to urllib in python2.
curl which works and gives son response:
curl -i -X GET -H "X-AUTH-TOKEN: $AUTH_TOKEN" \
-H "Accept: application/json" \
"https://api.xyz.com/apiv1.2/reports/nodes?start_date=2014-04-01&end_date=2014-04-21"
I tried the following code and it keeps on redirecting me to login html page as response. How can I convert the above curl request to urllib?
headers = {"Content-Type": "application/json", "AUTH_TOKEN":'1234yyzxx'}
data = urllib.urlencode(values)
request = urllib2.Request(ENDPOINT + '?' + data, headers=headers)
response = urllib2.urlopen(request)
text = response.read()
print text
Found it using requests library.
import requests
import json
response = requests.get(ENDPOINT, headers=headers, params=values)
text = json.loads(response.text)
However, I wouldn't mind answers in all non-deprecated libraries (urllib, urllib2, urllib3 etc).
Which one is faster?

Python Requests: How can I properly submit a multipart/form POST using a file name

I have taken a look at other questions related to multipart/form POST requests in Python but unfortunately, they don't seem to address my exact question. Basically, I normally use CURL in order to hit an API service that allows me to upload zip files in order to create HTML5 assets. The CURL command I use looks like this:
curl -X POST -H "Authorization: api: 222111" --form "type=html" --form "file=Folder1/Folder2/example.zip" "https://example.api.com/upload?ins_id=123"
I am trying to use a python script to iterate through a folder of zip files in order to upload all of these files and receive a "media ID" back. This is what my script looks like:
import os
import requests
import json
ins_id = raw_input("Please enter your member ID: ")
auth = raw_input("Please enter your API authorization token: ")
for filename in os.listdir("zips"):
if filename.endswith(".zip"):
file_path = os.path.abspath(filename)
url = "https://example.api.com/upload?
ins_id="+str(ins_id)
header = {"Authorization": auth}
response = requests.post(url, headers=header, files={"form_type":
(None, "html"), "form_file_upload": (None, str(file_path))})
api_response = response.json()
print api_response
This API service requires the file path to be included when submitting the POST. However, when I use this script, the response indicates that "file not provided". Am I including this information correctly in my script?
Thanks.
Update:
I think I am heading in the right direction now (thanks to the answer provided) but now, I receive an error message stating that there is "no such file or directory". My thinking is that I am not using os.path correctly but even if I change my code to use "relpath" I still get the same message. My script is in a folder and I have a completely different folder called "zips" (in the same directory) which is where all of my zip files are stored.
To upload files with the request library, you can include the file handler directly in the JSON as described in the documentation. This is the corresponding example that I have taken from there:
url = 'http://httpbin.org/post'
files = {'file': open('path_to_your_file', 'rb')}
r = requests.post(url, files=files)
If we integrate this in your script, it would look as follows (I also made it slightly more pythonic):
import os
import requests
import json
folder = 'zips'
ins_id = raw_input("Please enter your member ID: ")
auth = raw_input("Please enter your API authorization token: ")
url = "https://example.api.com/upload?"
header = {"Authorization": auth}
for filename in os.listdir(folder):
if not filename.endswith(".zip"):
continue
file_path = os.path.abspath(os.path.join(folder, filename))
ins_id="+str(ins_id)"
response = requests.post(
url, headers=header,
files={"form_type": (None, "html"),
"form_file_upload": open(file_path, 'rb')}
)
api_response = response.json()
print api_response
As I don't have the API end point, I can't actually test this code block - but it should be something along these lines.

pycurl: How to authenticate making a POST request

Using the following command via cURL (Windows): curl -u username:password "https://website.com/update/" --data "simChangesList=%5B%7B%22simId%22%3A760590802%2C%22changeType%22%3A2%2C%22targetValue%22%3A%22000307%22%2C%22effectiveDate%22%3Anull%7D%5D" --compressed
I am able to successfully post my data authenticated as per the -u switch.
How can I successfully create the same POST request using pycurl?
import urllib
import pycurl
enc = urllib.quote('simChangesList:[{"simId":760590802,"changeType":2,"targetValue":000307,"effectiveDate":null}]')
c = pycurl.Curl()
c.setopt(c.URL, 'https://website.com/update/')
c.setopt(c.POSTFIELDS, enc)
c.setopt(c.VERBOSE, True)
c.perform()
c.close()
Executing the above code will not POST the data I am wanting it to do, and I suspect it's because I'm requiring authentication, as per the Windows cURL version.
How can I successfully modify my code so that I can POST my data authenticated?