Multipart form data with vibe.d - d

I'm trying to submit a multipart form containing an image with vibe.d
My code looks like this:
auto f = File("image.jpg");
auto data = new char[f.size];
f.rawRead(data);
string boundary = randomUUID.toString.toUpper();
string h = "--Boundary-" ~ boundary ~ "\r\n";
h ~= `Content-Disposition: form-data; name="type"` ~ "\r\n\r\n";
h ~= "photo\r\n";
h ~= "--Boundary-" ~ boundary ~ "\r\n";
h ~= `Content-Disposition: form-data; name="photo"; filename="558704D0-2855-4689-996C-F556BE4A3872.jpg"` ~ "\r\n";
h ~= "Content-Type: image/jpeg\r\n\r\n";
h ~= data ~ "\r\n";
h ~= "--Boundary-" ~ boundary ~ "\r\n";
h ~= `Content-Disposition: form-data; name="photo_ids"` ~ "\r\n\r\n";
h ~= `["55227F15-36D2-4A04-A4D9-FB23C00627D1"]` ~ "\r\n";
h ~= "--Boundary-" ~ boundary ~ "\r\n";
auto response = requestHTTP("https://httpbin.org/anything", (scope req) {
req.method = HTTPMethod.POST;
req.headers["content-type"] = "multipart/form-data; boundary=Boundary-" ~ boundary;
req.headers["content-length"] = to!string(h.length);
req.bodyWriter.write(h);
}).bodyReader.readAllUTF8();
logInfo(response);
but httpbin tells me that I posted nothing:
[main(----) INF] {
[main(----) INF] "args": {},
[main(----) INF] "data": "",
[main(----) INF] "files": {},
[main(----) INF] "form": {},
[main(----) INF] "headers": {
[main(----) INF] "Accept-Encoding": "gzip, deflate",
[main(----) INF] "Content-Length": "58038",
[main(----) INF] "Content-Type": "multipart/form-data; boundary=Boundary-76CCC942-83EB-4339-BB6B-2C7D5BF027B6",
[main(----) INF] "Host": "httpbin.org",
[main(----) INF] "User-Agent": "vibe.d/1.7.0 (HTTPClient, +http://vibed.org/)"
[main(----) INF] },
[main(----) INF] "json": null,
[main(----) INF] "method": "POST",
[main(----) INF] "origin": "",
[main(----) INF] "url": "https://httpbin.org/anything"
[main(----) INF] }
I have no idea what I'm doing wrong. Would appreciate any help

It is radically simpler now with the current version of Vibe.d.
Make sure you state 'multipart/form-data' in the enctype of your form:
form.form-grid(method="post", action="new_employee", enctype="multipart/form-data")
Then a field in that form should include an input field of type 'file', something like this:
input(type="file", name="picture")
In the postNewEmployee() method of your web framework class, get the file through request.files:
auto pic = "picture" in request.files;
Here is a sample postNewEmployee() method being passed an Employee struct:
void postNewEmployee(string _authUser, Employee e)
{
import std.file;
import std.path;
import std.algorithm;
string photopath = "No photo submitted";
auto pic = "picture" in request.files;
if(pic !is null)
{
string ext = extension(pic.filename.name);
string[] exts = [".jpg", ".jpeg", ".png", ".gif"];
if(canFind(exts, ext))
{
photopath = "uploads/photos/" ~ e.fname ~ "_" ~ e.lname ~ ext;
string dir = "./public/uploads/photos/";
mkdirRecurse(dir);
string fullpath = dir ~ e.fname ~ "_" ~ e.lname ~ ext;
try moveFile(pic.tempPath, NativePath(fullpath));
catch (Exception ex) copyFile(pic.tempPath, NativePath(fullpath));
}
}
e.photo = photopath;
if(e.phone.length == 0) e.phone = "(123) 456 7890";
if(e.paygd.length == 0) e.paygd = "none yet";
if(e.postcode.length == 0) e.postcode = "A1A 1A1";
e.pword = createDigestPassword(realm, e.email, e.pword);
empModel.addEmployee(e);
redirect("list_employees");
}
When I tried to learn Vibe.d again, I again became aware of the dearth of tutorials, so I wrote a tutorial myself while everything is fresh as a learner:
https://github.com/reyvaleza/vibed
Hope you find this useful.

Your multipart data is malformed. For the last boundary you have to append a "--" string to indicate the end. So instead of the last
h ~= "--Boundary-" ~ boundary ~ "\r\n";
it needs to be
h ~= "--Boundary-" ~ boundary ~ "--\r\n";`
and then it will work.
See https://www.freesoft.org/CIE/RFC/1521/16.htm
Eventually this should be supported as API inside vibe.d, and there are 2 open PRs for this, but for now you have to work around it like you did.
https://github.com/vibe-d/vibe.d/pull/1178
https://github.com/vibe-d/vibe.d/pull/1876

Related

Odd ColdFusion cfscript syntax issue

I'm having a very odd syntax error in my cfscript.
stFields = {
"EligibilityQuery": {
"Patient": {
"FirstName": arguments.data.lname,
"MiddleName": "",
"LastName": arguments.data.fname,
"DateOfBirth": dateformat(arguments.data.dob,'yyyy-mm-dd'),
"Gender": arguments.data.gender,
"SSN": arguments.data.SSN,
"Address": {
"FirstLine": "",
"SecondLine": "",
"ZipCode": arguments.data.ZipCode
}
},
"NPI": "1111111"
}
};
// call API
var authorization = "Basic: " & ToBase64('username:password');
cfhttp(method="POST", url="https://mysite/api/myAPI/", result="apiResult"){
cfhttpparam(name="Authorization", type="header", value="#authorization#");
cfhttpparam(name="Content-Type", type="header", value="application/json");
cfhttpparam(type="body", value="#serializeJSON(stFields)#");
}
apiResult = deserializeJSON(apiResult.fileContent);
It's returning error on cfhttp (A script statement must end with ";".)
Error - The CFML compiler was processing:
cfhttp(method="POST", url="https://mysite/api/myAPI/", result="apiResult")
Where am I missing the ";"?
Expects a ; after cfhttp(method="POST", url="https://mysite/api/myAPI/", result="apiResult").
Are you on CF9 or CF10? Try this:
// call API
var authorization = "Basic: " & ToBase64('username:password');
httpService = new http(method = "POST", charset = "utf-8", url = "https://mysite/api/myAPI/");
httpService.addParam(name = "Authorization", type = "header", value = "#authorization#");
httpService.addParam(name = "Content-Type", type = "header", value = "application/json");
httpService.addParam(type = "body", value = "#serializeJSON(stFields)#");
apiResult = httpService.send().getPrefix();
apiResult = deserializeJSON(apiResult.fileContent);

Repeatedly send API request based on parameters in Postman

I have a database table with approx 4,000 records (currently). A response to an API call (POST, JSON) gives me data of this table for a maximum of 1,000 records per API call. A parameter ‘PageNo’ defines which of the 4,000 records are selected (e.g. PageNo = 1 gives me record 1-1000). The header data of the response includes a ‘PageCount’, in my example 4. I am able to retrieve that ‘PageCount’ and the test below loops through the PageNo (result in Postman Console = 1 2 3 4).
How I can call the same request repeatedly in a loop and use the values of the PageNo (i) as a parameter for that request like so:
{{baseUrl}}/v1/Units/Search?PageNo={{i}}
In my example I would expect the request to run 4 times with PageNo2 = 1, 2, 3, 4.
I am aware that I can use a CSV file and loop through the request in Collection Runner but PageCount changes (i.e. the number of records in the table change) and I need to run this loop frequently so creating a new CSV file for each loop is not really an option.
Postman Test:
pm.environment.set('Headers2', JSON.stringify(pm.response.headers));
var Headers2 = JSON.stringify(pm.response.headers);
pm.environment.set('PageCount2', JSON.parse(Headers2)[10].value);
var i;
for (i = 1; [i] <= pm.environment.get('PageCount2'); i++) {
console.log(i);
postman.setNextRequest('custom fields | json Copy');
}
Postman Request:
{
"Location":"{{TestingLocation}}",
"Fields":[
"StockNo",
"BrandDesc"
],
"Filters": {
"StatusCode":"{{TestingUnitSearchStatusCode}}"
},
"PageSize":1000,
"PageNo" : "{{i}}"
}
With postman.setNextRequest() you can set the calling request as the same request. But you need an exit strategy, otherwise that request would be called infinite times. This only works with the collection runner.
On your first(!) request, store the amount of pages and create a counter.
Increment the counter. If it is smaller than the amount of pages, set the current request as the next request.
Else, do not set a next request. The collection runner will continue with its normal flow. Optionally remove the pages and counter variables.
You can let the request detect that it is its first iteration by not having initialized the amount of pages and counter variables.
Performed a POC for your use-case using postman-echo API.
Pre-req script --> Takes care of the initial request to the endpoint to retrieve the PageSize and set it as an env var. Also initializes the iterationCount to 1 (as env var)
Test Script --> Checks for the current iteration number and performs tests.
Here's a working postman collection.
If you're familiar with newman, you can just run:
newman run <collection-name>
You can also import this collection in Postman app and use collection runner.
{
"info": {
"_postman_id": "1d7669a6-a1a1-4d01-a162-f8675e01d1c7",
"name": "Loop Req Collection",
"schema": "https://schema.getpostman.com/json/collection/v2.1.0/collection.json"
},
"item": [
{
"name": "Run_Req",
"event": [
{
"listen": "test",
"script": {
"id": "dac5e805-548b-4bdc-a26e-f56500414208",
"exec": [
"var iterationCount = pm.environment.get(\"iterationCount\");",
"if (pm.environment.get(\"iterationCount\") <= pm.environment.get(\"pageSize\")) {",
" console.log(\"Iteration Number: \" + iterationCount);",
"",
" pm.test(\"Status code is 200\", function() {",
" pm.response.to.have.status(200);",
" });",
"",
" pm.test(\"Check value of pageNo in body is equal to iterationCount\", function() {",
" var jsonData = pm.response.json();",
" return jsonData.json.PageNo == iterationCount;",
"",
" });",
" iterationCount++;",
" pm.environment.set(\"iterationCount\", iterationCount);",
" postman.setNextRequest('Run_Req');",
"}",
"if (pm.environment.get(\"iterationCount\") > pm.environment.get(\"pageSize\")) {",
" console.log(\"Cleanup\");",
" pm.environment.unset(\"pageSize\");",
" pm.environment.set(\"iterationCount\", 1);",
" postman.setNextRequest('');",
" pm.test(\"Cleanup Success\", function() {",
" if (pm.environment.get(\"pageSize\") == null && pm.environment.get(\"iterationCount\") == null) {",
" return true;",
" }",
" });",
"}"
],
"type": "text/javascript"
}
},
{
"listen": "prerequest",
"script": {
"id": "3d43c6c7-4a9b-46cf-bd86-c1823af5a68e",
"exec": [
"if (pm.environment.get(\"pageSize\") == null) {",
" pm.sendRequest(\"https://postman-echo.com/response-headers?pageSize=4\", function(err, response) {",
" var pageSize = response.headers.get('pageSize');",
" var iterationCount = 1;",
" console.log(\"pre-req:pageSize= \" + pageSize);",
" console.log(\"pre-req:iterationCount= \" + iterationCount);",
" pm.environment.set(\"pageSize\", pageSize);",
" pm.environment.set(\"iterationCount\", iterationCount);",
" });",
"",
"}"
],
"type": "text/javascript"
}
}
],
"request": {
"method": "POST",
"header": [
{
"key": "Content-Type",
"name": "Content-Type",
"value": "application/json",
"type": "text"
}
],
"body": {
"mode": "raw",
"raw": "{\n \"Location\":\"{{TestingLocation}}\",\n \"Fields\":[\n \"StockNo\",\n \"BrandDesc\"\n ],\n \"Filters\": {\n \"StatusCode\":\"{{TestingUnitSearchStatusCode}}\"\n },\n \"PageSize\":1000,\n \"PageNo\" : \"{{iterationCount}}\"\n}"
},
"url": {
"raw": "https://postman-echo.com/post",
"protocol": "https",
"host": [
"postman-echo",
"com"
],
"path": [
"post"
]
}
},
"response": []
}
]
}

How to split json value in log file using grok/regular expression

I have one log file which I need to extract the json content from the file and I need to parse it using logstash json filter. I wrote one grok pattern but which it is not working properly. Below is my log file.
2016-12-18 12:13:52.313 -08:00 [Information] 636176600323139749 1b2c4c40-3da6-46ff-b93f-0eb07a57f2a3 18 - API: GET https://aaa.com/o/v/S?$filter=uid eq '9'&$expand=org($filter=org eq '0')
{
"Id": "1b",
"App": "D",
"User": "",
"Machine": "DC",
"RequestIpAddress": "xx.xxx.xxx",
"RequestHeaders": {
"Cache-Control": "no-transform",
"Connection": "close",
"Accept": "application/json"
},
"RequestTimestamp": "2016-12-18T12:13:52.2609587-08:00",
"ResponseContentType": "application/json",
"ResponseContentBody": {
"#od","value":[
{
"uid":"","sId":"10,org":[
{
"startDate":"2015-02-27T08:00:00Z","Code":"0","emailId":"xx#gg.COM"
}
]
}
]
},
"ResponseStatusCode": 200,
"ResponseHeaders": {
"Content-Type": "application/json;"
},
"ResponseTimestamp": "2016-12-18T12:13:52.3119655-08:00"
}
My Grok pattern
grok {
match => [ "message","%{TIMESTAMP_ISO8601:exclude}%{GREEDYDATA:exclude1}(?<exclude2>[\s])(?<json_value>[\W\w]+)"]
}
Assuming this is all one message (it's not multiline, or has been combined before now) and there's a space between the URI and the json, this grok pattern should work:
%{TIMESTAMP_ISO8601} %{NOTSPACE:timezone} \[%{WORD:severity}] %{WORD:field1} %{UUID:field2} %{NUMBER:field3} - API: %{WORD:verb} (?<field4>[^\{]*) %{GREEDYDATA:json}
It would have been nice to use %{URI}, but the string you have is not a valid URI (it contains unescaped spaces).

Google Maps Geolocation API returns the latitude and longitude of my server

During my testing the Geolocation API use to return proper result.
But in deployment on AWS it is returning me the IP address of Virginia where my EC2 instances are located.
"cellId": 27193,
"locationAreaCode": 17007,
"mobileCountryCode": 404,
"mobileNetworkCode": 20
Result:
"lat": 19.2019619,
"lng": 73.1063466
But on AWs it returns:
"lat": 39.043756699999996
"lng": -77.4874416
params = {
"key": SERVER_KEY
}
data = json.dumps({
"cellTowers": [
{
"cellId": cid,
"locationAreaCode": lac,
"mobileCountryCode": mcc,
"mobileNetworkCode": mnc
}
]
})
log.info(params)
log.info(data)
request = requests.post(HOST_URL, params=params, data=data)
# print(request.text)
# print(request.status_code)
if request.status_code == 200:
response = json.loads(request.text)
latitude = response["location"]["lat"]
longitude = response["location"]["lng"]
return {
"latitude": latitude,
"longitude": longitude
}
else:
error_data = request.text
log.error("Error Occurred in Finding Cell Tower Location")
log.error(request.text)
log.error(json.dumps(params))
send_error_notification(error_data)
return None
Answer is so simple and how silly of me assuming that not adding header would not affect me.
By just adding the header with application/json everything is working like a charm.
params = {
"key": SERVER_KEY
}
data = json.dumps({
"considerIp": False,
"cellTowers": [
{
"cellId": cid,
"locationAreaCode": lac,
"mobileCountryCode": mcc,
"mobileNetworkCode": mnc
}
]
})
headers = {
'content-type': "application/json",
'cache-control': "no-cache"
}
log.info(params)
log.info(data)
request = requests.post(HOST_URL, params=params, data=data, headers=headers)

YouTube API returns zero results

I'm using YouTube API to retrieve videos data. I have a list of videos IDs.
For some of the videos the results that returns are normal and for some of them the API returns zero results. Via my browser all the results are valid.
Here is my code:
def getVideoDuration(self,videoId):
try:
content = urllib2.urlopen("https://www.googleapis.com/youtube/v3/videos?part=statistics%2C+contentDetails&id=" + videoId +"&key=" + self.DEVELOPER_KEY).read()
jsonContent= json.loads(content)
duration = jsonContent['items'][0].values()[0]['duration']
if len(duration) == 7:
minutes = jsonContent['items'][0].values()[0]['duration'][2]
seconds = jsonContent['items'][0].values()[0]['duration'][4:6]
if len(duration) == 5:
minutes = 0
seconds = jsonContent['items'][0].values()[0]['duration'][2:4]
print minutes,seconds
totalTime = str(minutes) + "." + str(seconds)
return float(totalTime)
except:
return 0.0
For the ID: 'fu5K2cOeD4M' my code return zero results, but via my browser the results are normal (JSON response attached):
{
"kind": "youtube#videoListResponse",
"etag": "\"0KG1mRN7bm3nResDPKHQZpg5-do/wTtZkXqw81l7Hq6-GrLwJ3wRQ5w\"",
"pageInfo": {
"totalResults": 1,
"resultsPerPage": 1
},
"items": [
{
"kind": "youtube#video",
"etag": "\"0KG1mRN7bm3nResDPKHQZpg5-do/jrxp-dHXG3s3ujaIjyq15GWV7V8\"",
"id": "fu5K2cOeD4M",
"contentDetails": {
"duration": "PT8M15S",
"dimension": "2d",
"definition": "sd",
"caption": "false",
"licensedContent": false
},
"statistics": {
"viewCount": "18358",
"likeCount": "166",
"dislikeCount": "1",
"favoriteCount": "0",
"commentCount": "33"
}
}
]
}
I tried to delay between my requests using time.sleep(), but it didn't help me.
The problem was that i sent the request i added an extra character