422 Unprocessable Entity response when POSTing file upload in Clojure - clojure

I am trying to emulate this curl request
curl "https://{subdomain}.zendesk.com/api/v2/uploads.json?filename=myfile.dat&token={optional_token}" \
-v -u {email_address}:{password} \
-H "Content-Type: application/binary" \
--data-binary #file.dat -X POST
with the following code
(POST "/uploads" request
(let [filename (get-in request [:params "file" :filename])
file (get-in request [:params "file" :tempfile])
url (str "https://REDACTED.zendesk.com/api/v2/uploads.json?filename=" filename)]
(clj-http.client/post url {:headers {"Content-Type" “application/binary”}
:multipart-params [{:name "file"
:content file
:mime-type "application/binary”}]})
but I am getting a ‘422 Unprocessable Entity’ response from Zendesk. The file/tempfile is coming in as #object[java.io.File 0x3768306f "/var/folders/l3/7by17gp51sx2gb2ggykwl9zc0000gn/T/ring-multipart-6501654841068837352.tmp"] on the request.
I have played with clojure.java.io coercions (like clojure.java.io/output-stream) as mentioned at Saving an image form clj-http request to file, but that didn't help.
(PS. I’m fairly certain I don’t need to auth because I can get the direct upload to Zendesk to work through Postman.)

After revisiting this, the solution was simple. Zendesk expects the request body to be binary (as the curl request indicates). So, in this case, I passed the image to my server as base64 encoded data (just as JSON).
I then used this library to convert the base64 string to a byte array: https://github.com/xsc/base64-clj
(defn byte-array-from-base64
[base64-string]
(base64/decode-bytes (.getBytes base64-string)))
Finally, you can simple pass the byte array to Zendesk as the body of the clj-http library request.
(client/post
"https://REDACTED.zendesk.com/api/v2/uploads.jsonfilename=filename.jpg"
{:headers {"Authorization" "Basic AUTHORIZATION_TOKEN"
"Content-Type" "application/binary"}
:body (byte-array-from-base64 base64-string)})

Related

AWS Presigned URL works with Python's Requests library but fails with cURL

Recently I started using AWS pre-signed URLs to upload files to S3. The generated pre-signed URLs are working perfectly when using Python's Requests library as follows:
Generating the pre-signed url:
def create_presigned_post(bucket_name, object_name,
fields=None, conditions=None, expiration=3600):
"""Generate a presigned URL S3 POST request to upload a file
:param bucket_name: string
:param object_name: string
:param fields: Dictionary of prefilled form fields
:param conditions: List of conditions to include in the policy
:param expiration: Time in seconds for the presigned URL to remain valid
:return: Dictionary with the following keys:
url: URL to post to
fields: Dictionary of form fields and values to submit with the POST
:return: None if error.
"""
# Generate a presigned S3 POST URL
s3_client = boto3.client('s3')
try:
response = s3_client.generate_presigned_post(bucket_name,
object_name,
Fields=fields,
Conditions=conditions,
ExpiresIn=expiration)
except ClientError as e:
logging.error(e)
return None
# The response contains the presigned URL and required fields
return response
Running the request to get the presigned url
# Getting a presigned_url to upload the file into S3 Bucket.
headers = {'Content-type': 'application/json', 'request': 'upload_url', 'target': FILENAME, 'x-api-key': API_KEY}
r_upload = requests.post(url = API_ENDPOINT, headers = headers)
url = json.loads(json.loads(r_upload.text)['body'])['url']
fields_ = json.loads(json.loads(r_upload.text)['body'])['fields']
fields = {
"x-amz-algorithm": fields_["x-amz-algorithm"],
"key": fields_["key"],
"policy": fields_["policy"],
"x-amz-signature": fields_["x-amz-signature"],
"x-amz-date": fields_["x-amz-date"],
"x-amz-credential": fields_["x-amz-credential"],
"x-amz-security-token": fields_["x-amz-security-token"]
}
fileobj = open(FILENAME, 'rb')
http_response = requests.post(url, data=fields,files={'file': (FILENAME, fileobj)})
Valid Response
"{\"url\": \"https://****.s3.amazonaws.com/\",
\"fields\":
{\"key\": \"******\", \"x-amz-algorithm\": \"*******\", \"x-amz-credential\": \"*******\", \"x-amz-date\": \"*********\", \"x-amz-security-token\": \"********", \"policy\": \"**********\", \"x-amz-signature\": \"*******\"}}
And as you can see I'm providing no AWSAccessKey or any credentials when uploading the file using the generated pre-signed URL and this is so logical, as the pre-signed URL is created to be given for external users who have to provide no credentials when using such URL.
However and when trying to run the same call made by Python's Requests library, using cURL, the request is failing with the error:
< HTTP/1.1 403 Forbidden
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><Error>
To get the exact request call made by requests.post, I'm running:
req = http_response.request
command = "curl -X {method} -H {headers} -d '{data}' '{uri}'"
method = "PUT"
uri = req.url
data = req.body
headers = ['"{0}: {1}"'.format(k, v) for k, v in req.headers.items()]
headers = " -H ".join(headers)
print(command.format(method=method, headers=headers, data=data, uri=uri))
Which returns:
curl -v -X PUT -H "Connection: keep-alive" --upload-file xxxx.zip -H "Accept-Encoding: gzip, deflate" -H "Accept: */*" -H "User-Agent: python-requests/2.18.4" -H "Content-Length: xxxx" -H "Content-Type: multipart/form-data; boundary=8a9864bdxxxxx00100ba04cc055a" -d '--8a9864bd377041xxxxx04cc055a
Content-Disposition: form-data; name="x-amz-algorithm"
AWS4-HMAC-SHA256
--8a9864bd377041e0b00100ba04cc055a
Content-Disposition: form-data; name="key"
xxxxx.zip
--8a9864bd377041e0b00100ba04cc055a
Content-Disposition: form-data; name="x-amz-signature"
*****
--8a9864bd377041e0b00100ba04cc055a
Content-Disposition: form-data; name="x-amz-security-token"
*****
--8a9864bd377041e0b00100ba04cc055a
Content-Disposition: form-data; name="x-amz-date"
*****
--8a9864bd377041e0b00100ba04cc055a
Content-Disposition: form-data; name="policy"
*****
--8a9864bd377041e0b00100ba04cc055a
Content-Disposition: form-data; name="x-amz-credential"
xxxxx/xxxxx/xxxx/s3/aws4_request
' 'https://xxxxx.s3.amazonaws.com/'
Then reformulate it:
$ curl -v -T file "https://****.s3.amazonaws.com/?key=************&x-amz-algorithm=***************&x-amz-credential=*************&x-amz-security-token=************&policy=**********&x-amz-signature=****************
After researching, I found nothing similar to this issue, but:
https://aws.amazon.com/es/premiumsupport/knowledge-center/s3-access-denied-error/
This still seem not logical to me because I'm not supposed to enter any credentials when using a pre-signed URL.
I don't know if I'm missing something of the complete request made by Python's Requests library.
Any ideas, please!
Kind regards,
Rshad
This simple curl command should work:
With a usual presigned url, it would be as follows:
curl -v \
-F key=<filename> \
-F x-amz-algorithm=*** \
-F x-amz-credential=*** \
-F x-amz-date=*** \
-F x-amz-security-token=*** \
-F policy=*** \
-F x-amz-signature=*** \
-F file=#<filename> \
'https://<bucket>.s3.amazonaws.com/'
The -F field allows you to specify the additional POST data that should be uploaded to S3 (i.e. from the fields data returned w/ the pre-signed URLs.
Kind regards,

Connecting re-graph graphql client for clojurescript to lacinia-pedestal graphql server

I am using lacinia-pedestal for server and re-graph for client side clojuresript
My client code looks like
(re-frame/dispatch [::re-graph/init {:http-url
"http://localhost:8888/graphql"
:ws-url nil
:http-parameters {:headers {"Content-Type" "application/graphql"}
}}])
(re-frame/dispatch [::re-graph/query
"{current_user(token: ss) {id}}"
nil some-func])
However when I connect to my server, I see following error in console log
"message":"Failed to parse GraphQL query.","extensions":{"errors":[{"locations":[{"line":1,"column":null}],"message":"mismatched input '\"query\"' expecting {'query', 'mutation', 'subscription', '...', NameId}"}
Following curl request it works,
curl http://localhost:8888/graphql
-H 'Content-Type: application/graphql'
-d 'query {
current_user(token: "foo"){
id
}}'
Any help will be appreciated
I am attaching my network console log
re-graph uses application/json by default to send the query and variables as a JSON payload. You don't need to override the content-type in your init call.

Compojure-api: Request coercion for a map supplied via multipart-params

I am trying to integrate compojure-api (version 1.1.12) into an existing compojure-based application. While most things work, I am having an issue with request coercions on an existing REST call whose usage would be difficult to change at this point.
It's a POST
It expects parameters via multipart/form-data
Most parameters are optional.
Most parameters are simple: string or array of string.
One optional parameter is expected to be a JSON-encoded map.
I define the route like this:
(POST "/endpoint" request
:multipart-params
[required-strings :- (describe [s/Str] "Required, an array of strings"),
{optional-string :- (describe s/Str "An optional string") ""},
{others :- {s/Keyword s/Any} {}}]
...)
This works, unless I try to pass other-parameters in a request. For example, via curl:
curl -F "required-strings=[\"Hello\"]" -F "others={\"a\":1.0}" ...
This results in an invalid request (i.e. status 400) error with the content:
{"errors":{"others":"(not (map? a-clojure.lang.PersistentVector))"}}
I'm using ring-default's site-defaults, and I haven't modified the default coercions for the compojure api. I've traced the error to compojure.api.coerce/coerce. I can see the value that the coercer is working on, and it looks like:
{:required-strings "[\"Hello\"]"
:others "{\"a\":1.0}"}
On line 59 of coerce.clj, (coerce value) returns an error (per schema.utils/error?).
So, is it not possible to coerce a JSON-encoded multipart parameter to a Clojure map? I can define the parameter to be a string instead of a map, and do the parsing myself, but this defeats the purpose of using compojure-api and ring-swagger.
The coercer for others expects a clojure map not a string. To make it work you have to options.
First option: add wrap-json-params middleware and do application/json request instead of multipart/form-data:
curl ... -H 'Content-Type: application/json' \
-d $'{
"required-strings": ["Hello"],
"others": {"a": 1.0}
}'
Second option: add wrap-nested-params middleware and multipart/form-data request using nested param names:
curl ... -H 'Content-Type: multipart/form-data' \
-F "required-strings[]=Hello" \
-F "others[a]=1.0"

Which `format` would be negotiated for REST request?

There are three variants of format selection:
curl -uadmin:admin "http://localhost:8080/alfresco/service/hellouser.json"
curl -uadmin:admin "http://localhost:8080/alfresco/service/hellouser?format=json"
curl -uadmin:admin -H "Accept: text/html" "http://localhost:8080/alfresco/service/hellouser"
But this is unclear from the DOC what format would be selected for next query:
curl -uadmin:admin -H "Accept: text/html" "http://localhost:8080/alfresco/service/hellouser.xml?format=json"
I expect json here.
May someone provide links to relevant specifications or documentation which describes priority how {format} negotiated? like this is described for Rails:
Rails picks up the expected format from the query parameter format, or if not there from the URL path suffix, or it not there from the Accept header
UPD
The controller can handle all supplied formats: json, xml, html
UPD
Another corner case:
curl -uadmin:admin "http://localhost:8080/alfresco/service/hellouser.pl?format=json"
curl -uadmin:admin "http://localhost:8080/alfresco/service/hellouser.pl?format=xml"
I'd believe you wouldn't have a 200 response, only an error with content negotiation.
The code shows that:
?format=json(format_query_param) will be discarded by the .xml (format_suffix)
filter available renderers leaving only the XMLRenderer left
then it will loop on the accept header but none will match text/html
finally this will be down to the exception

writing a test to check the file returned by a ring response

I have a compojure route which returns a file. I want to test -
1) If a file is returned.
2) The specific file that was returned.
When I run (app (ring.mock/request :get "/myroute")) I get
{:body #<File resources/public/templates/index.html>, :headers {"Content-Length" "2349", "Last-Modified" "Sat, 16 Mar 2013 11:01:03 GMT"}, :status 200}
How do I check that the returned value in the body is of a type file ? And getting more ambitious can I check it is the file located at 'resources/public/templates/index.html' ?
Ring requests are just maps, so you can extract the body with the :body keyword and then check it's type with type
(type (:body (app (ring.mock/request :get "/myroute"))))
(perhaps I'm not understanding the question though?)