Does curl need some special parameters to enforce charset/encoding? - web-services

I'm using a curl command to send a large file (100+ MB) to a web service. I'm noticing that only when I send the file to the web service using curl the file gets mangled and data is lost.
Here is the command I'm using to send the file:
curl -v --raw -X POST -H "Transfer-Encoding: chunked" -H "Content-Type: text/xml; charset=UTF-8" -d #medline16n0736.xml "http://localhost:2323/TestWebService"
Am I missing something? I thought telling it to use text/xml and charset=UTF-8 would keep it UTF-8 once received by the web service.

You are asking curl to post the XML file using the -d option, which will post the file as if it were being submitted via an HTTP webform in application/x-www-webform-urlencoded format. To post the file by itself, use the -T option instead. Also, you are using the --raw option, which will disable handling of HTTP transfer encodings, even though you are sending a Transfer-Encoding: chunked header. Remove --raw and -T will detect the header to enable chunking.
You are also asking curl to send a Content-Type header to tell the WebService that the uploaded data is UTF-8 encoded XML. It is your responsibility to make sure the XML file is actually UTF-8 encoded. Curl won't check that for you.

Related

How to fix 'TypeError: failed to fetch' (Chrome) / 'NetworkError when attempting to fetch resource.' (FireFox) in WSO2-AM

I am setting up WSO2-AM and uploading an API's servicedefinition swagger.json but upon trying the service in WSO2-AM, I get the response in FireFox "TypeError: NetworkError when attempting to fetch resource.". In Chrome its response is "TypeError: Unable to fetch".
The server is informationproxy.com and has SSL configured. Looks fine having a green lock in the urlbar with trusted certificate-chain. It does not certify for using the ip-number "https://209.182.203.81:9443/store/"
Question is: what makes WSO2 respond (in FireFox) with "TypeError: NetworkError when attempting to fetch resource."
The service does not seem to be the problem, since it works fine in editor.swagger.io.
The service also works fine directly called from curl to the external serviceprovider, returning a valid JSON with the data I expect:
curl -k -X GET "https://tieka.nl/webservices/address/luxembourg/v1/?postcode=8080&number=1" -H "accept: application/json"
Also: all other api-services that I have tried to implement fail with the same error message.
There is a side-effect. The response to this curl is an encrypted payload:
curl -k -X GET "https://209.182.203.81:8243/webservices/address/luxembourg/v1/?postcode=8080&number=1" -H "accept: application/json" -H "Authorization: Bearer a1111111-1aa1-1111-aaaa-1111aaaa1111"
<ns:binary xmlns:ns="http://ws.apache.org/commons/ns/payload">PCFET0NUWVBFIEhUTUwgUFVCTElDICItLy9JRVRGLy9EVEQgSFRNTCAyLjAvL0VOIj4KPGh0bWw+PGhlYWQ+Cjx0aXRsZT4zMDEgTW92ZWQgUGVybWFuZW50bHk8L3RpdGxlPgo8L2hlYWQ+PGJvZHk+CjxoMT5Nb3ZlZCBQZXJtYW5lbnRseTwvaDE+CjxwPlRoZSBkb2N1bWVudCBoYXMgbW92ZWQgPGEgaHJlZj0iaHR0cHM6Ly93d3cudGlla2Eubmw6ODAvd2Vic2VydmljZXMvYWRkcmVzcy9sdXhlbWJvdXJnL3YxLz9wb3N0Y29kZT04MDgwJmFtcDtudW1iZXI9MSI+aGVyZTwvYT4uPC9wPgo8aHI+CjxhZGRyZXNzPkFwYWNoZSBTZXJ2ZXIgYXQgd3d3LnRpZWthLm5sIFBvcnQgODA8L2FkZHJlc3M+CjwvYm9keT48L2h0bWw+Cg==</ns:binary>

libcurl POST xml file to url

I have been working on uploading a simple XML file to a specialized Web Server. This webserver only supports the use of POST or GET, so it eliminates the use of CURLOPT_UPLOAD which only provides for PUT.
I can use the command line to send the file using the following command line
curl -H 'Content-Type: text/xml; charset=utf-8' -H 'SOAP Action:'
-T igs_meters2.xml -X POST //url//
I tried to use CURLOPT_POST and CURLOPT_READDATA, but received a 413 Request Entity Too Large.
I know this cannot be this hard, but I am out in areas I have never work in. Thanks in advance for an idea.
SOLVED:
the tip about CUSTOMREQUEST was what I needed to find. That helped, and I also needed to add in an HTTPHEADER for CONTENT-TYPE and SOAPAction:

Direct file upload to S3 using sigv4

I'm looking for a secure way to directly upload large files to S3 (for performance issue).
After few hours of research, I've come to the (maybe incorrect) conclusion that I should use "Browser-Based Uploads Using POST".
As referenced in this thread:
Amazon S3 direct file upload from client browser - private key disclosure
Before trying this directly, I thought about making a cURL POC with direct upload and signature computation.
I've failed to make it work, and haven't found a successful POC on the web using Authentification Version 4 manual settings.
My signature is OK.
The only issue is that Amazon is double SHA256ing my file content and is thus not validation my x-amz-content-sha-256 header.
lower(SHA256(e8379a31b13fb9423928fe28dd41a5e3204a52072634503c31e8b3ea42605b46))
= 4fa84cd7d18e0d33dbd62d0492eca4a159e122391ae0a3e636bd3cf527680c87
I'm not sure of understanding what should I put in my cURL and canonical request payload (and the linked content-length value) and the x-amz-content-sha-256 header.
Should it all be the same values ?
If yes, then the Amazon doc specifies it should all be encrypted to SHA256, so I've no idea why Amazon reSHA256 my already SHA256 payload...
Error:
<?xml version="1.0" encoding="UTF-8"?>
<Error>
<Code>XAmzContentSHA256Mismatch</Code>
<Message>The provided 'x-amz-content-sha256' header does not match what was computed.</Message><ClientComputedContentSHA256>e8379a31b13fb9423928fe28dd41a5e3204a52072634503c31e8b3ea42605b46</ClientComputedContentSHA256><S3ComputedContentSHA256>4fa84cd7d18e0d33dbd62d0492eca4a159e122391ae0a3e636bd3cf527680c87</S3ComputedContentSHA256>
<RequestId>419A185269B0F891</RequestId><HostId>QHWxK0Mzz6AfG44ypXBti3W0tYx1xkG9lZGqc2kUKyMF9STwP18M3racio0k06aH5+1ok/Irdn8=</HostId>
</Error>
cURL command:
curl
-v https://??.s3.amazonaws.com/recordtest/test.jpg
-H "Authorization: AWS4-HMAC-SHA256 Credential=??/20170228/eu-west-1/s3/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-content-sha256;x-amz-date, Signature=43750caa762314eb70aace1f7f8ae34633b93352aa25646433ef21e48dd79429" -H "Content-Length: 64"
-H "Content-Type: application/octet-stream"
-H "x-amz-content-sha256: e8379a31b13fb9423928fe28dd41a5e3204a52072634503c31e8b3ea42605b46"
-H "x-amz-date: 20170228T111828Z"
-d "e8379a31b13fb9423928fe28dd41a5e3204a52072634503c31e8b3ea42605b46"
-X PUT
Generated canonical request:
PUT
/recordtest/test.jpg
content-length:64
content-type:application/octet-stream
host:??.s3.amazonaws.com
x-amz-content-sha256:e8379a31b13fb9423928fe28dd41a5e3204a52072634503c31e8b3ea42605b46
x-amz-date:20170228T111200Z
content-length;content-type;host;x-amz-content-sha256;x-amz-date
e8379a31b13fb9423928fe28dd41a5e3204a52072634503c31e8b3ea42605b46

Setting --data-binary using libcurl in c++

I'm using the curl api from http://curl.haxx.se/ in a c++ application. I'm trying to write the equivalent of:
curl -v -X POST -H "Content-type: application/json; charset=utf-16" --data-binary #some data.json
I'm using curl_easy_setopt() to set up the various options but the bit I can't figure out is the --data-binary flag. Which CURLOPT setting would I use to set this?
Thanks in advance
You should use CURLOPT_POSTFIELDS:
specify data to POST to server [...] Pass a char * as parameter, pointing to the full data to send in a HTTP POST operation. You must make sure that the data is formatted the way you want the server to receive it.
Note: you can easily discover this by using the --libcurl option:
--libcurl FILE Dump libcurl equivalent code of this command line

converting curl api call into restful api call

i am very new to the java EE.
I have built some simple restful client for calling rest api's from java using jersey framework.
I have one curl request which i have to call from java
curl -i -X POST 'https://{url}' -H 'Content-Type: application/x-www-form-urlencoded' -H "Accept: application/vnd.newbay.dv-1.8+json" -H "X-HTTP-METHOD-OVERRIDE: DELETE" -H "X-Client-Platform: SERVICES" -H "X-Client-Identifier: IL" --insecure --data 'path=%2FMy+Samsung+GT-I9082%2F20130822_172409.jpg&path=%2FMy+Samsung+GT-I9082%2F20130905_085407.jpg'
Here the paths of file are passed as form post in curl using --data.
Can anyone tell me how to call this api from my java program using jersey framework? I am having difficulty because of this --data part? how exactly i will pass that in my restful java client that i cant understand.
Also can anyone tell me how to pass form post --data part while calling the same api from soapUI?
In your cURL command, --data provides post data in a classic html form format. You shoud alrealy be aware of that because the content-type is explicit : application/x-www-form-urlencoded
Jersey can handle this encoding for you, using MultivaluedMapImpl.
see Sending Name Value Pair in POST using Jersey Client