Making a POST request in C++ with curlpp - c++

I am attempting to make a POST request using curlpp in C++ to Statistics Canada with their getDataFromVectorsAndLatestNPeriods. I can't seem to get a result from the request.
#include <stdlib.h>
#include <stdio.h>
#include <curlpp/cURLpp.hpp>
#include <curlpp/Easy.hpp>
#include <curlpp/Options.hpp>
int main()
{
curlpp::Cleanup cleanup;
curlpp::Easy request;
curlpp::Forms form;
request.setOpt(curlpp::options::Url(std::string("https://www150.statcan.gc.ca/t1/wds/rest/getDataFromVectorsAndLatestNPeriods")));
request.setOpt(curlpp::options::Verbose(true));
form.push_back(new curlpp::FormParts::Content("vectorID:54325508","latestN:1"));
request.setOpt(new curlpp::options::HttpPost(form));
request.setOpt(new curlpp::options::WriteStream(&std::cout));
request.perform();
return 0;
}
I compiled it with g++ -std=gnu++11 -lcurl -lcurlpp cry.cpp
And when the output when verbose is set to true is:
* Trying 205.193.226.160...
* TCP_NODELAY set
* Connected to www150.statcan.gc.ca (205.193.226.160) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/cert.pem
CApath: none
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: C=CA; ST=Ontario; L=Ottawa; jurisdictionCountryName=CA; O=Statistics Canada; businessCategory=Government Entity; serialNumber=1970-01-01; CN=www150.statcan.gc.ca
* start date: Oct 4 16:33:01 2019 GMT
* expire date: Jan 3 17:02:58 2022 GMT
* subjectAltName: host "www150.statcan.gc.ca" matched cert's "www150.statcan.gc.ca"
* issuer: C=US; O=Entrust, Inc.; OU=See www.entrust.net/legal-terms; OU=(c) 2014 Entrust, Inc. - for authorized use only; CN=Entrust Certification Authority - L1M
* SSL certificate verify ok.
> POST /t1/wds/rest/getDataFromVectorsAndLatestNPeriods HTTP/1.1
Host: www150.statcan.gc.ca
Accept: */*
Content-Length: 161
Content-Type: multipart/form-data; boundary=------------------------8fe530d4d57d4b83
* We are completely uploaded and fine
< HTTP/1.1 415
< Date: Sat, 06 Nov 2021 03:39:47 GMT
< Content-Length: 0
< Connection: keep-alive
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Content-Security-Policy: default-src 'self' 'unsafe-inline' *.statcan.gc.ca *.statcan.ca *.stc.ca *.demdex.net *.omtrdc.net *.everesttech.net blob:; style-src 'self' 'unsafe-inline' *.statcan.gc.ca *.statcan.ca https://fonts.googleapis.com blob:; script-src 'self' 'unsafe-inline' 'unsafe-eval' *.statcan.gc.ca *.statcan.ca *.googletagmanager.com *.adobedtm.com *.jsdelivr.net *.mathjax.org cdnjs.cloudflare.com blob:; connect-src 'self' *.statcan.gc.ca *.statcan.ca *.stc.ca *.demdex.net *.omtrdc.net https://api.mapbox.com/ https://events.mapbox.com/; img-src 'self' *.statcan.gc.ca *.statcan.ca *.stc.ca *.demdex.net *.omtrdc.net *.everesttech.net *.jsdelivr.net data: blob:; font-src 'self' *.statcan.gc.ca *.statcan.ca https://fonts.gstatic.com; worker-src 'self' 'unsafe-inline' 'unsafe-eval' *.statcan.gc.ca *.statcan.ca blob:; frame-src 'self' 'unsafe-inline' *.statcan.gc.ca *.statcan.ca *.stc.ca https://dv-vd.shinyapps.io *.demdex.net blob:;
< Strict-Transport-Security: max-age=31536000; includeSubDomains
< Strict-Transport-Security: max-age=31536000
<
* Connection #0 to host www150.statcan.gc.ca left intact
* Closing connection 0
What is happening and how can I get it to do what I actually want?

I haven't used libcurlpp, but for libcurl a natural way of making a POST request is through the CURLOPT_POST and CURLOPT_POST_FIELDS options, see for example How to use libcurl for HTTP post?. This leas to this simple main:
int main()
{
curlpp::Cleanup cleanup;
curlpp::Easy request;
request.setOpt(curlpp::options::Url(std::string("https://www150.statcan.gc.ca/t1/wds/rest/getDataFromVectorsAndLatestNPeriods")));
// request.setOpt(curlpp::options::Verbose(true));
std::list<std::string> header =
{
"Content-Type: application/json",
"accept: application/json"
};
request.setOpt(new curlpp::options::HttpHeader(header));
std::string query = "[{\"vectorId\":54325508, \"latestN\":1}]";
request.setOpt(new curlpp::options::PostFields(query));
request.setOpt(new curlpp::options::WriteStream(&std::cout));
request.perform();
}
The part setting the HTTP header can actually be skipped for the server you connect with. The solution is in complete agreement with example 12 from curlpp documentation, https://github.com/jpbarrette/curlpp/blob/master/examples/example12.cpp .

Related

ModSecurity / CRS: Need help writing a custom rule to whitelist false-positives - Rule 951220/951120 (user listing an SQL programming book for sale)

ModSecurity 3.0.8
ModSecurity-Nginx 1.0.3
CRS: 4.0.0-rc1
I have a marketplace where users list various things for sale. One such item is a used Microsoft SQL book. ModSecurity is complaining about this:
---CBSB85oo---A--
[10/Nov/2022:01:04:59 +0000] 166804229999.091612 66.249.70.172 53858 ***.***.***.*** 443
---CBSB85oo---B--
GET /sale/microsoft-sql-server-2008-r2-standard-w-5-cals-1-install-download-delivery-1634636/ HTTP/1.1
Host: www.*********.com
AMP-Cache-Transform: google;v="1..8"
Connection: keep-alive
Accept: text/html,application/xhtml+xml,application/signed-exchange;v=b3,application/xml;q=0.9,*/*;q=0.8
From: googlebot(at)googlebot.com
User-Agent: Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.5304.87 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Accept-Encoding: gzip, deflate, br
---CBSB85oo---F--
HTTP/1.1 403
Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
Cache-Control: no-store, no-cache, must-revalidate
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Vary: Accept-Encoding
X-XSS-Protection: 1; mode=block
Set-Cookie: NEWSESSID=n2b1ct0ipgh8qj5ti8ia0en134; path=/; HttpOnly
Pragma: no-cache
Connection: keep-alive
Content-Type: text/html; charset=utf-8
Date: Thu, 10 Nov 2022 01:04:59 GMT
Server: nginx
X-Frame-Options: SAMEORIGIN
X-Content-Type-Options: nosniff
Referrer-Policy: strict-origin
Permissions-Policy: geolocation=(),midi=(),sync-xhr=(),microphone=(),camera=(),magnetometer=(),gyroscope=(),fullscreen=(self),payment=()
---CBSB85oo---H--
ModSecurity: Warning. Matched "Operator `Rx' with parameter `(?i)(?:System\.Data\.OleDb\.OleDbException|\[Microsoft\]\[ODBC SQL Server Driver\]|\[Macromedia\]\[SQLServer JDBC Driver\]|\[SqlException|System\.Data\.SqlClient\.SqlException|Unclosed quotation mark (641 characters omitted)' against variable `RESPONSE_BODY' (Value: `<!DOCTYPE html>\x0d\x0a<html lang="en" prefix="og: http://ogp.me/ns#">\x0d\x0a<head>\x0d\x0a <meta (74840 characters omitted)' ) [file "/etc/nginx/modsec/coreruleset-4.0.0-rc1/rules/RESPONSE-951-DATA-LEAKAGES-SQL.conf"] [line "312"] [id "951220"] [rev ""] [msg "mssql SQL Information Leakage"] [data "Matched Data: SQL Server 2008 R2 Standard w/5 cals -1 Install (Download Delivery) For Sale</title>\x0d\x0a <meta name="Description" content="For Sale: Microsoft SQL Server 2008 R2 Standard w/5 cals -1 Inst (64617 characters omitted)"] [severity "2"] [ver "OWASP_CRS/4.0.0-rc1"] [maturity "0"] [accuracy "0"] [tag "application-multi"] [tag "language-multi"] [tag "platform-mssql"] [tag "attack-disclosure"] [tag "paranoia-level/1"] [tag "OWASP_CRS"] [tag "capec/1000/118/116/54"] [hostname "***.***.***.***"] [uri "/sale/microsoft-sql-server-2008-r2-standard-w-5-cals-1-install-download-delivery-1634636/"] [unique_id "166804229999.091612"] [ref "o179,64776v572,65688"]
ModSecurity: Access denied with code 403 (phase 4). Matched "Operator `Ge' with parameter `4' against variable `TX:BLOCKING_OUTBOUND_ANOMALY_SCORE' (Value: `5' ) [file "/etc/nginx/modsec/coreruleset-4.0.0-rc1/rules/RESPONSE-959-BLOCKING-EVALUATION.conf"] [line "186"] [id "959100"] [rev ""] [msg "Outbound Anomaly Score Exceeded (Total Score: 5)"] [data ""] [severity "0"] [ver "OWASP_CRS/4.0.0-rc1"] [maturity "0"] [accuracy "0"] [tag "anomaly-evaluation"] [hostname "***.***.***.***"] [uri "/sale/microsoft-sql-server-2008-r2-standard-w-5-cals-1-install-download-delivery-1634636/"] [unique_id "166804229999.091612"] [ref ""]
I'm fairly inexperienced with writing custom CRS exclusion rules, but I'm sure someone has had to deal with this same issue, where various programming books are listing for sale and ModSecurity gets thrown off by it with rule 921220.
In addition, there were a few other listings, such as the following that trigger a similar rule 921120:
---4N6WNSFo---A--
[09/Nov/2022:02:08:59 +0000] 166795973938.894013 159.148.57.6 25181 ***.***.***.*** 443
---4N6WNSFo---B--
GET /sale/yamaha-g100iiib100iiien-operating-guide-by-download-mauritron-203723-200170/ HTTP/1.1
Host: www.**********e.com
User-Agent: Mozilla/5.0 (compatible; GeedoBot; +http://www.geedo.com/bot.html)
Accept: text/html, application/xhtml+xml, application/xml; q=0.9, */*; q=0.8
Accept-Encoding: gzip
---4N6WNSFo---F--
HTTP/1.1 403
Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
Cache-Control: no-store, no-cache, must-revalidate
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Vary: Accept-Encoding
X-XSS-Protection: 1; mode=block
Set-Cookie: NEWSESSID=07h58j65hep9tfduebjbn25qg5; path=/; HttpOnly
Pragma: no-cache
Connection: keep-alive
Content-Type: text/html; charset=utf-8
Date: Wed, 09 Nov 2022 02:08:59 GMT
Server: nginx
X-Frame-Options: SAMEORIGIN
X-Content-Type-Options: nosniff
Referrer-Policy: strict-origin
Permissions-Policy: geolocation=(),midi=(),sync-xhr=(),microphone=(),camera=(),magnetometer=(),gyroscope=(),fullscreen=(self),payment=()
---4N6WNSFo---H--
ModSecurity: Warning. Matched "Operator `Rx' with parameter `(?i:ORA-[0-9][0-9][0-9][0-9]|java\.sql\.SQLException|Oracle error|Oracle.*Driver|Warning.*oci_.*|Warning.*ora_.*)' against variable `RESPONSE_BODY' (Value: `<!DOCTYPE html>\x0d\x0a<html lang="en" prefix="og: http://ogp.me/ns#">\x0d\x0a<head>\x0d\x0a <meta (68173 characters omitted)' ) [file "/etc/nginx/modsec/coreruleset-4.0.0-rc1/rules/RESPONSE-951-DATA-LEAKAGES-SQL.conf"] [line "69"] [id "951120"] [rev ""] [msg "Oracle SQL Information Leakage"] [data "Matched Data: ora-7025 found within RESPONSE_BODY"] [severity "2"] [ver "OWASP_CRS/4.0.0-rc1"] [maturity "0"] [accuracy "0"] [tag "application-multi"] [tag "language-multi"] [tag "platform-oracle"] [tag "attack-disclosure"] [tag "paranoia-level/1"] [tag "OWASP_CRS"] [tag "capec/1000/118/116/54"] [hostname "***.***.***.***"] [uri "/sale/yamaha-g100iiib100iiien-operating-guide-by-download-mauritron-203723-200170/"] [unique_id "166795973938.894013"] [ref "o37082,8v298,59690"]
ModSecurity: Access denied with code 403 (phase 4). Matched "Operator `Ge' with parameter `4' against variable `TX:BLOCKING_OUTBOUND_ANOMALY_SCORE' (Value: `5' ) [file "/etc/nginx/modsec/coreruleset-4.0.0-rc1/rules/RESPONSE-959-BLOCKING-EVALUATION.conf"] [line "186"] [id "959100"] [rev ""] [msg "Outbound Anomaly Score Exceeded (Total Score: 5)"] [data ""] [severity "0"] [ver "OWASP_CRS/4.0.0-rc1"] [maturity "0"] [accuracy "0"] [tag "anomaly-evaluation"] [hostname "***.***.***.***"] [uri "/sale/yamaha-g100iiib100iiien-operating-guide-by-download-mauritron-203723-200170/"] [unique_id "166795973938.894013"] [ref ""]
How should I deal with listings like this, and how should I craft the custom CRS rule to prevent users from being blocked from pages like these? Also, I do not use an Microsoft or Oracle server, but it would be nice to keep as many of the CRS rules as possible without having to turn the Engine off for whole website directories such as "/sale/" (every item listed for sale).
Thanks
Try this exclusion rule (for both cases):
SecRule REQUEST_FILENAME "#beginsWith /sale/" \
"id:1000,\
phase:1,\
pass,\
t:none,\
nolog,\
ctl:ruleRemoveTargetById=951120;RESPONSE_BODY,\
ctl:ruleRemoveTargetById=951220;RESPONSE_BODY"

APIGateway Lambda integration do not serve binary file

I have an api rest gateway and a lambda fetching an image from s3 after doing some checks.
That serving binary files is not so easy I noticed and found a lot of problems about this topic on SO or on other sites.
What I understand and already configured is the following:
my lambda is returning an JSON object with isBase64Encoded
return {
statusCode: 200,
headers: {
"Content-Type": mimeType, // mimeType is here image/jpeg
},
body: data.toString("base64"), // data is a buffer;
// I also tried data.toString("binary") and data.toString()
isBase64Encoded: true,
}
In my API gateway I set the binary Media Types to the following
Resources:
MyApiGateway:
Type: AWS::ApiGateway::RestApi
Properties:
...
BinaryMediaTypes:
- "application/octet"
- "image/jpeg"
- "image/png"
- "image/gif"
- "image/*"
This settings I see also in the AWS Console.
When I now query me API Gateway I only receive a BASE64 encoded image.
❯ curl https://myurl/test/profile-picture?size=small -H "Accept: image/jpeg" -v
* TCP_NODELAY set
* Connected to xxx port 443 (#0)
...
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x558f76b7fea0)
> GET /test/profile-picture?size=small HTTP/2
> Host: xxx
> user-agent: curl/7.68.0
> accept: image/jpeg
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 200
< date: Sun, 28 Feb 2021 21:37:04 GMT
< content-type: image/jpeg
< content-length: 3940
< x-amzn-requestid: bc54ee15-5c41-4828-a7b2-c4978af07176
< x-amzn-trace-id: Root=1-603c0d00-2f468936671986a95532641e;Sampled=0
<
/9j/4AAQSkZJRgABAQAAAQABAAD/2wCEAAEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAf/AABEIAD8AZAMBEQACEQEDEQH/xAGiAAABBQEBAQEBAQAAAAAAAAAAAQIDBAUGBwgJCgsQAAIBAwMCBAMFBQQEAAABfQECAwAEEQUSITFBBhNRYQcicRQygZGhCCNCscEVUtHwJDNicoIJChYXGBkaJSYnKCkqNDU2Nzg5OkNERUZHSElKU1RVVldYWVpjZGVmZ2hpanN0dXZ3eHl6g4SFhoeIiYqSk5SVlpeYmZqio6Slpqeoqaqys7S1tre4ubrCw8TFxsfIycrS09TV1tfY2drh4uPk5ebn6Onq8fLz9PX29/j5+gEAAwEBAQEBAQEBAQAAAAAAAAECAwQFBgcICQoLEQACAQIEBAMEBwUEBAABAncAAQIDEQQFITEGEkFRB2FxEyIygQgUQpGhscEJIzNS8BVictEKFiQ04SXxFxgZGiYnKCkqNTY3ODk6Q0RFRkdISUpTVFVWV1hZWmNkZWZnaGlqc3R1dnd4eXqCg4SFhoeIiYqSk5SVlpeYmZqio6Slpqeoqaqys7S1tre4ubrCw8TFxsfIycrS09TV1tfY2dri4+Tl5ufo6ery8/T19vf4+fr/2gAMAwEAAhEDEQA/AP8AP/oAKACgAoAKACgAoAKACgAoAKAHKMnHPTtQA/YPf9P8KAIqACgAoAKACgAoAKACgD2H4E/BPxX+0H8SNF+GXg6/8P6Vq+ssMar4pvbuw0PT43ubWxhkvZtO0/VtSdrnUL6x0+0ttP0y+vLq9vLaCG3YuSvicQ57h+HMrrZricLj8bSozo0/q+W4dYnEzlWqKnBqM6lGjTpxcr1K1etSo046ymm0n914dcA5l4l8UYbhXKc04fyjGYnC43GRxnEuaRyrL1SwGHlia1KFX2VevisbVpwccJgcHh8Ri8TU92nScYzlH6w+LP8AwTz8a/B/U/C+l694k13Uv7emEOo6vpvwl8dadonh9UurG3nvtQfxfF4Y1uHSVS/juLbVNR0HSrW+ijne2LRwzSR/M4fxAwGIfK/7LwVT2aqSoZhxBl9PEwT25qWEWNjZ6LmjUlG7Svc/dF9FXihYSti6ceOM6pQxM8NSxPCnhfnmdZfiZU4qVR0sdjMxyWKlFqpF0qlKNRezcnFRaZn/AAg/Ye0r4ueJdJ8K2nxqtfD+p3vi8eFtRl1P4e6xPY6Ravs2+JnmtdbM2o6EhFybiXTba4nhjgiYQySX9hFcebmXiU8phTrYzKsFDDVK0aUcRT4goV4JSteq/YYGo3CHNFzUVKok0+Rn3fDv0MaHFc8wwmR+IHENTNsvymrmNXK8x8KcxyWvLE0/aezy3nzTirDQp18S6co0K9Xkw0pRnB1FODi6n7Qn7B3iT4A+L9T8JP8AFPwF43uNM1q80aS50NdTthMltELiHVUtriKWWPTb+1P2m0Nw0N6EGy6sraVkjesu8UMvxuIxNCplmJprDqE4YnDYnD4zC4mlVjzU6lGa9hVtOPvKM6MJqLTaXMr8PEP0JeKcpyfJc3y7jPJ8Y849tSnlWbZRmeR5rl+Mwk1Tx2DxUKc83y+c8HVfsqtbD5jVoTqqUaVSfJUdOlpX7NPwrt7C2/4SL4m6lqGpTxi4eXw5ZR2VhCj/ACi2eHXNO+2faYZEkEkqs1vLGYpIypZ400xPiG/af7BllStRUYqUq/tKc41bXlBKnCpGUUnFqSkrttWsk3jlX0Q6lPDzjxXxngcszH6xU9hRy14bG4etgLQ+r4pzxFXDVKVSrNVlKg4S5Iwg3PmlKEPhav00/ioKACgAoAKACgAoAKAOn8GeM/E/w98UaL4y8G6xd6D4k8PahbanpepWbhZIbm1mSZEmicNBd2kxQRXlhdxzWV9bNJa3cE1vLJEwB+/Hgn4m+B/2vfCfxY+MXgfwXqQ+Iw8ArN+1H4M0/Vb2B9B8GgWtne+PfCF7cyC80y7j8R2fhvTPh74uttavfD/ha51rTfCnxK8N+Evhd8PvAc8qq4fC4ylLD4rDUMRRnFKdGvQp1qU1FqVpQnGUZpNRnyyT1jda3Z6GVZ5nnD+Ow+aZFnGaZNmWEm6mEzHKcwxeXY7CVJwnTlUw+KwlalWoSlTqVKUpU6kW6dSUG+WUk/Nfh/8AD7xF4Jmh0W9vJNO1bTPin8KtA0fx3ouqp4dvNf8Ahdqnh6LR/iv8UdWivr7S9R1Gx+FevajoMfitriItceI/GHg+Dwhq9xpEmv6vq/j4zhThrG01h8VkeV1qMFzQpVMJRdNJSTsouCjFSUYtJK2mtnZH3+V+Nni9k1eti8t8SeM8LisQo/WMTTz/ADGWIrOHMo+1rVK86lRw558rlJtOTt8TPBP2ydY8X2nwb+EHifxVb6toXxU1ObwhN4ss9csRpfjLS5dH0rxp4ciufHFqIbeSbW/H83hUeNnsNYgvVstDutETTr69ttR1G5vFh+G8gwdOpRwmTZbhqdSHs5RoYSjRfs3FpRUoQjONk3ZwkmtHF3SZnmXjL4rZxicBi828ROMczrZbiFi8Iswz/McXQo11UjU5lha9eeGnGbhFVadSjOnVjeFWE4ScX8++Av2nfC+n6G9t438M6hcast9MYJPDdroy2H9ntBbeWJF1mSe5S6+1fa2kWJxaiJofKRW8wV8RjvDhyruWXZtiMPQkuaVPES9pJVXKV+WVKnSXs1D2cYqSlO8ZOU3dW/pzhr6YOX/2aocb+H+V5nm9KqqVHF5HSq4PDTwMKFBU3iKWMzLESeMlifrdSrOi6WH9nUpQp0YOEnL4nr9RP4YCgAoAKACgAoAKACgAoA/Rb/glDqniGx/bu+C+maFq6aRaeK4PiF4V8Wm+kmTw5feCNS+Gvi6bxPa+M44bmzMng2DT7NtV8QyN* Connection #0 to host xn5gb6d7e6.execute-api.eu-central-1.amazonaws.com left intact
cwNptjpjazbT217ptrdQJ/qttHvbRgfvPpXwH8QeOPiL4KtPhd4E1aXTfCXhzU/Anw6vNZvEaz8I/A7XoPEVrcPr1rf6gsvivw9pnwq8W+In+JnjHT9X13xdoHxruPCn9qxeJLnRPh9BZ17Re85NaprTS9mm0t7bcyv00tbQTi9Er9Guu/e2t+r0V7+enwX/AMFwfgP8Uvg/4j8K6UPhC3wu+Bvhi1+H1hpvijxVL4C0Dxl8W/H/AIx+H0XjaKztNAs9K8MeOPEcHwM8G6xYfCvxVqepaV4gvPDviazul8X+LtRvPEvh2WeedSb76+ez6u1r6q3W3ezGotK7vbu1a7d3f567aadD+famAUAFABQAUAFABQAUAFABQB9ifsL/ALTfh/8AZC+Pdv8AHHXfAl78RptI8A/Erwdo/hq01jT9BSWb4p+DdV+GHiO6udV1HRPEUdm1v4B8XeMBpFwmjaitt4kk0S6vtP1PSYNR0y9TV9PPXv8ALzvb5DTtr1tp210d/K1/npsfS/xf/wCCsXx48VzySfBU6v8AAa4vIdQk1TxbpfiPSdc+J93r3iDRZdB8aeL7Dx7ongnwFN4f8Y+P7GeYeMPF+i6TaeLbqGQ6JpGv6L4Uz4dJZf1+nb5W/Ky+/XfzPzG8S+KvFXjDVNS1zxd4l8QeKtZ1jV9S8QatrHiPWdR1zVNU17WWgfWNb1LUNTubq7vdX1V7W2bUtSuZZby+a3ga6mlMUe1gc7QAUAFABQAUAFABQAUAFABQA9Mc5xxjGaAJaAEbofoaAIKACgAoAKACgAoAKACgAoAKACgAyR0JFAClie9ACUAFABQAUAFABQAUAFABQAUAFABQAUAFAH//2Q==
Is there something wrong?
I am not able to figre the error out :-(

Can not create shared-domain in Cloud Foundry - 504 Gateway Time-out

I can not create shared-domain in Cloud Foundry, any pushed apps get's health check connection refused.
I had working Cloud Foundry environment based on OpenStack IaaS. Everything worked as expected. I took my deployment files and after some time deployed it successfully in IaaS Vmware VSphere 7. The problem is, that every app that I push has problems with health check:
2020-10-29T16:55:01.43+0000 [CELL/0] OUT Cell 938b869c-5a68-40cc-9486-c5bc0d53a73a successfully destroyed container for instance 44e9c2a6-b54d-4fc4-4118-6d6d
2020-10-29T16:55:36.55+0000 [CELL/0] OUT Cell 938b869c-5a68-40cc-9486-c5bc0d53a73a creating container for instance 17f161a2-9788-426d-414d-6c33
2020-10-29T16:55:37.18+0000 [CELL/0] OUT Cell 938b869c-5a68-40cc-9486-c5bc0d53a73a successfully created container for instance 17f161a2-9788-426d-414d-6c33
2020-10-29T16:55:37.47+0000 [CELL/0] OUT Downloading droplet...
2020-10-29T16:55:37.75+0000 [CELL/0] OUT Downloaded droplet
2020-10-29T16:55:37.75+0000 [CELL/0] OUT Starting health monitoring of container
2020-10-29T16:56:38.45+0000 [HEALTH/0] ERR Failed to make TCP connection to port 8080: connection refused
2020-10-29T16:56:38.45+0000 [CELL/0] ERR Timed out after 1m0s: health check never passed.
2020-10-29T16:56:38.46+0000 [CELL/SSHD/0] OUT Exit status 0
2020-10-29T16:56:38.48+0000 [APP/PROC/WEB/0] OUT Exit status 143
I am also not able to create any shared domains:
bash-5.0# cf create-shared-domain tcp.cf.test-env.net --router-group default-tcp -v
REQUEST: [2020-10-29T17:03:33Z]
GET /v2/info HTTP/1.1
Host: api.cf.test-env.net
Accept: application/json
User-Agent: cf/6.47.2+d526c2cb3.2019-11-05 (go1.12.12; amd64 linux)
RESPONSE: [2020-10-29T17:03:33Z]
HTTP/1.1 200 OK
Content-Length: 561
Content-Type: application/json;charset=utf-8
Date: Thu, 29 Oct 2020 17:03:33 GMT
Server: nginx
X-Content-Type-Options: nosniff
X-Vcap-Request-Id: 4badb79b-2faf-4623-6c3c-ce5fa3223cd5::dc43d2c9-c902-4429-9d65-d9a0060983c5
{
"api_version": "2.144.0",
"app_ssh_endpoint": "ssh.cf.test-env.net:2222",
"app_ssh_host_key_fingerprint": "ae:a3:ed:ad:37:d3:8a:7b:ed:b4:e5:d2:25:e5:8c:d0",
"app_ssh_oauth_client": "ssh-proxy",
"authorization_endpoint": "https://login.cf.test-env.net",
"build": "",
"description": "",
"doppler_logging_endpoint": "wss://doppler.cf.test-env.net:443",
"min_cli_version": null,
"min_recommended_cli_version": null,
"name": "",
"osbapi_version": "2.15",
"routing_endpoint": "https://api.cf.test-env.net/routing",
"support": "",
"token_endpoint": "https://uaa.cf.test-env.net",
"version": 0
}
REQUEST: [2020-10-29T17:03:33Z]
GET /login HTTP/1.1
Host: login.cf.test-env.net
Accept: application/json
Connection: close
User-Agent: cf/6.47.2+d526c2cb3.2019-11-05 (go1.12.12; amd64 linux)
RESPONSE: [2020-10-29T17:03:34Z]
HTTP/1.1 200 OK
Cache-Control: no-store
Content-Language: en-US
Content-Length: 384
Content-Type: application/json;charset=UTF-8
Date: Thu, 29 Oct 2020 17:03:34 GMT
Set-Cookie: X-Uaa-Csrf=NJlSPAjspn7m8oWuQdKsVD; Max-Age=86400; Expires=Fri, 30-Oct-2020 17:03:34 GMT; Path=/; Secure; HttpOnly
Strict-Transport-Security: max-age=31536000
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-Vcap-Request-Id: 577d4d31-ec30-477e-6f44-c0dd9306270d
X-Xss-Protection: 1; mode=block
{
"app": {
"version": "74.12.0"
},
"commit_id": "7311e68",
"entityID": "login.cf.test-env.net",
"idpDefinitions": {},
"links": {
"login": "https://login.cf.test-env.net",
"passwd": "/forgot_password",
"register": "/create_account",
"uaa": "https://uaa.cf.test-env.net"
},
"prompts": {
"password": "[PRIVATE DATA HIDDEN]",
"username": [
"text",
"Email"
]
},
"timestamp": "2019-12-02T22:53:03+0000",
"zone_name": "uaa"
}
Creating shared domain tcp.cf.test-env.net as admin...
REQUEST: [2020-10-29T17:03:34Z]
GET /routing/v1/router_groups?name=default-tcp HTTP/1.1
Host: api.cf.test-env.net
Accept: application/json
Authorization: [PRIVATE DATA HIDDEN]
Connection: close
Content-Type: application/json
User-Agent: cf/6.47.2+d526c2cb3.2019-11-05 (go1.12.12; amd64 linux)
[application/json Content Hidden]
RESPONSE: [2020-10-29T17:03:34Z]
HTTP/1.1 200 OK
Content-Length: 114
Content-Type: application/json
Date: Thu, 29 Oct 2020 17:03:34 GMT
X-Vcap-Request-Id: 9459b068-0987-4f5e-7dee-1efdb5ca6fb8
[
{
"guid": "343ba1e8-88a7-4003-6db6-4feabedd072b",
"name": "default-tcp",
"reservable_ports": "1024-2048",
"type": "tcp"
}
]
REQUEST: [2020-10-29T17:03:34Z]
POST /v2/shared_domains HTTP/1.1
Host: api.cf.test-env.net
Accept: application/json
Authorization: [PRIVATE DATA HIDDEN]
Content-Type: application/json
User-Agent: cf/6.47.2+d526c2cb3.2019-11-05 (go1.12.12; amd64 linux)
{
"internal": false,
"name": "tcp.cf.test-env.net",
"router_group_guid": "343ba1e8-88a7-4003-6db6-4feabedd072b"
}
RESPONSE: [2020-10-29T17:04:04Z]
HTTP/1.0 504 Gateway Time-out
Cache-Control: no-cache
Connection: close
Content-Type: text/html
<html><body><h1>504 Gateway Time-out</h1>
The server didn't respond in time.
</body></html>
Error unmarshalling the following into a cloud controller error: <html><body><h1>504 Gateway Time-out</h1>
The server didn't respond in time.
</body></html>
FAILED
I suspect network configuration issue, that blocks some internal CF parts from connection. There is no any firewall or any rules found in VMware. I can also ping and make ssh connection between bosh created VM's.
Any ideas, what else can I do?
the problem was with DNAT and SNAT rules on VmWare NSX-T. If any internal VM asked about dns name "api.cf.test-env.net" it get's remote (public) IP address as answer. When the connection should be established, the internal VM has been asking api.cf.test-env.net by public IP address, and get's the local one by second stage of TCP three-way-handshake - what caused TCP RST. After creating DNAT and SNAT rules correctly, everything works as expected. I still wondering why "api.cf.test-env.net" is not answered by bosh-dns with internal address. Does anyone know why it so and how it can be changed?

Why end point returning below error while processing request?

Details:- Have added Datamapper in-process module of my wso2 project. But when I send request JSON using command prompt to my back-end service I get below error from the endpoint.
--In console window of Integration studio.
Details:- From below logs, I can say it pass through a log module just before endpoint.
[2020-02-18 15:25:14,521] INFO {org.apache.synapse.mediators.builtin.LogMediator} - message = Routing to clemency medical center
[2020-02-18 15:46:22,301] INFO {org.apache.synapse.mediators.builtin.LogMediator} - message = Routing to clemency medical center
---In Command Prompt getting error:-
F:\WS02\WSO2 Integration Studio\Request_JSON\HelathCare\Transforming Message Content>curl -v -X POST --data #request.json http://localhost:8280/healthcare/categories/surgery/reserve --header "Content-Type:application/json"
Note: Unnecessary use of -X or --request, POST is already inferred.
* Trying ::1...
* TCP_NODELAY set
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8280 (#0)
> POST /healthcare/categories/surgery/reserve HTTP/1.1
> Host: localhost:8280
> User-Agent: curl/7.55.1
> Accept: */*
> Content-Type:application/json
> Content-Length: 200
>
* upload completely sent off: 200 out of 200 bytes
< HTTP/1.1 500 Internal Server Error
< Accept-Ranges: none
< Access-Control-Allow-Methods: POST
< Set-Cookie: SERVERID=s0; path=/
< Access-Control-Allow-Headers: content-type
< Content-Type: application/octet-stream
< Via: HTTP/1.1 forward.http.proxy:8080
< Date: Tue, 18 Feb 2020 10:16:27 GMT
< Transfer-Encoding: chunked
<
Error in executing request: POST /clemency/categories/surgery/reserve* Connection #0 to host localhost left intact
Below are request and response JSON content have used.
Request content (client requested content in below format)
{
"name": "John Doe",
"dob": "1940-03-19",
"ssn": "234-23-525",
"address": "California",
"phone": "8770586755",
"email": "johndoe#gmail.com",
"doctor": "thomas collins",
"hospital": "grand oak community hospital"
}
The response we expect using data mapper from back end service.
{
"patient": {
"name": "John Doe",
"dob": "1990-03-19",
"ssn": "234-23-525",
"address": "California",
"phone": "8770586755",
"email": "johndoe#gmail.com"
},
"doctor": "thomas collins",
"hospital": "grand oak community hospital"
}
In case of Using the data mapper approach just be sure on input and output schema you are providing and after successfully mapping input and output for conversion from XML to JSON vice versa. Please ensure the properties of Data-Mapper make input and output as per your requirements.
By default it set to XML to XML.

401 When trying to create an orgunit using Google API

I'm trying to use Google's Admin SDK to create an orgunit using a shell script. My script is as follows:
# Obtain a token we can use to modify the organisation
auth_header=`oauth2l header --json "..." "admin.directory.orgunit"`
customer_id=...
curl -v -H "Content-Type: application/json" -X POST \
--data-binary "#google-orgunits/technical.json" \
--header "$auth_header" \
"https://www.googleapis.com/admin/directory/v1/customer/$customer_id/orgunits"
This produces the output:
* Trying 216.58.196.138...
* Connected to www.googleapis.com (216.58.196.138) port 443 (#0)
* found 173 certificates in /etc/ssl/certs/ca-certificates.crt
* found 704 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
* server certificate verification OK
* server certificate status verification SKIPPED
* common name: *.googleapis.com (matched)
* server certificate expiration date OK
* server certificate activation date OK
* certificate public key: RSA
* certificate version: #3
* subject: C=US,ST=California,L=Mountain View,O=Google Inc,CN=*.googleapis.com
* start date: Wed, 05 Apr 2017 17:01:30 GMT
* expire date: Wed, 28 Jun 2017 16:56:00 GMT
* issuer: C=US,O=Google Inc,CN=Google Internet Authority G2
* compression: NULL
* ALPN, server accepted to use http/1.1
> POST /admin/directory/v1/customer/.../orgunits HTTP/1.1
> Host: www.googleapis.com
> User-Agent: curl/7.47.0
> Accept: */*
> Content-Type: application/json
> Authorization: Bearer ...
> Content-Length: 157
>
* upload completely sent off: 157 out of 157 bytes
< HTTP/1.1 401 Unauthorized
< Vary: X-Origin
< WWW-Authenticate: Bearer realm="https://accounts.google.com/", error=invalid_token
< Content-Type: application/json; charset=UTF-8
< Date: Sat, 15 Apr 2017 06:26:27 GMT
< Expires: Sat, 15 Apr 2017 06:26:27 GMT
< Cache-Control: private, max-age=0
< X-Content-Type-Options: nosniff
< X-Frame-Options: SAMEORIGIN
< X-XSS-Protection: 1; mode=block
< Server: GSE
< Alt-Svc: quic=":443"; ma=2592000; v="37,36,35"
< Accept-Ranges: none
< Vary: Origin,Accept-Encoding
< Transfer-Encoding: chunked
<
{
"error": {
"errors": [
{
"domain": "global",
"reason": "required",
"message": "Login Required",
"locationType": "header",
"location": "Authorization"
}
],
"code": 401,
"message": "Login Required"
}
}
There must be some problem here: I appear to be obtaining a valid token, (looks like ya29.ElouBGKFig-nXZ9uykyGoDr0hxAxG5PMJTUh3VmtAtj2SAdYEbH2Coumjp5XoaF232oVx3--2EpTyNi5NgFBNrLINJij9tGL3-64MshEXjHhvkH-1NESoxPeVAU). I've followed all of the instructions here, enabled API access, authorized my API client, everything; but still, not working. Where have I gone wrong?
Try checking the documentation about Directory API: Authorize Requests
Every request your application sends to the Directory API must include an authorization token. The token also identifies your application to Google.
Here's the OAuth 2.0 scope information for the Directory API:
https://www.googleapis.com/auth/admin.directory.orgunit - Global scope for access to all organization unit operations.
https://www.googleapis.com/auth/admin.directory.orgunit.readonly -
Scope for only retrieving organization units.
You can check the OAuth 2.0 Playground, an interactive demonstration of using OAuth 2.0 with Google (including the option to use your own client credentials). Also there are many quickstart that can help you on how to properly authorize a request for Admin SDK.
Hope this helps.