I'm calling a REST WS with a JSON payload to subscribe certains events. The server answer with HTTP-Code 201 and a field named Location in the HTTP-Header with the ID of the subscription.
As an example, in curl (-v) we get:
[...]
< HTTP/1.1 201 Created
< Connection: Keep-Alive
< Content-Length: 0
< Location: /v2/subscriptions/5ab386ad4bf6feec37ffe44d
[...]
In C++ using curlpp we want to retrieve that id looking at the response header. Now we have only the body response (in this case empty).
std::ostringstream response;
subRequest.setOpt(new curlpp::options::WriteStream(&response));
// Send request and get a result.
subRequest.perform();
cout << response.str() << endl;
How can we obtain the Location header's field (whose content in the example is "/v2/subscriptions/5ab386ad4bf6feec37ffe44d") in C++ using curlpp?
There are several values you can retrieve using the family of curlpp::infos::*::get functions. For example the HTTP response code:
curlpp::infos::ResponseCode::get(subRequest)
See the Infos.hpp header for a complete list. When you need a value that is not available through one of these infos you can also choose to receive the headers separately from the body in a callback.
subRequest.setOpt(new curlpp::options::HeaderFunction(
[] (char* buffer, size_t size, size_t items) -> size_t {
std::string s(buffer, size * items); // buffer is not null terminated
std::cout << s;
return size * items;
}));
Ok, I found it.
Simply adding
subRequest.setOpt(new curlpp::options::Header(1));
do the trick and stores headers in the response.
Related
Is it possible to use libmodsecurity as a library and process requests on my own? I was messing with the examples in the repo ModSecurity examples, but I cant figure out how to make it take my request. I tried with simple_example_using_c.c but with no success. Is anyone have idea if this is possible?
#include <stdio.h>
#include <stdlib.h>
#include <modsecurity/modsecurity.h>
#include <modsecurity/rules_set.h>
char rulez[] ="basic_rules.conf";
const char *request = "" \
"GET /?test=test HTTP/\n" \
"Host: localhost:9999\n" \
"Content-Length: 27\n" \
"Content-Type: application/x-www-form-urlencoded\n";
int main(){
ModSecurity *modsec;
RulesSet *setRulez;
Transaction *transakcyja;
const char *error;
modsec = msc_init();
printf(msc_who_am_i(modsec));
msc_set_connector_info(modsec, "ModSecurity simple API");
setRulez = msc_create_rules_set();
int rulz = msc_rules_add_file(setRulez, rulez, &error);
if(rulz == -1){
fprintf(stderr, "huston rulez problem \n");
fprintf(stderr, "%s\n", error);
return -1;
}
msc_rules_dump(setRulez);
transakcyja = msc_new_transaction(modsec, setRulez, NULL);
if(transakcyja == NULL){
fprintf(stderr, "Init bad");
return -1;
}
msc_process_connection(transakcyja, "127.0.0.1", 9998, "127.0.0.1", 9999);
msc_process_uri(transakcyja, "http://127.0.0.1:9999/?k=test&test=test", "GET", "1.1");
msc_process_request_body(transakcyja);
msc_process_response_headers(transakcyja, 200, "HTTP 1.3");
msc_process_response_body(transakcyja);
msc_process_logging(transakcyja);
msc_rules_cleanup(setRulez);
msc_cleanup(modsec);
return 0;
}
Edit: I know something more now but, anyone know how to pass request to transaction? I know there is addRequestHeader() but it takes one header at the time, I can't really figure it out.
I think you have to understand how ModSecurity works.
There are five phases of all transaction:
parse request headers
parse request body
parse response headers
parse response body
make logs (and check transaction variables, eg. anomaly scores (in case of CRS))
(And plus the phase 0: process the connection itself.)
In the examples you can see couple of functions for each phases.
This is a common HTTP request:
POST /set.php HTTP/1.1
Host: foobar.com
User-Agent: something
Accept: */*
Content-Type: application/x-www-form-urlencoded
Content-Length: 7
a=1&b=2
Now if you already create a transaction object, you have to add the data of phases to that.
First see the status line, and add the necessary parts - consider your application already has the information, like client IP (std::string cip), port (int cport), and server IP (std::string dip), port (int dport). You also have the URI (std::string uri), method (std::string method) and the version of protocoll (std::string vers). You also need an object with type modsecurity::ModSecurityIntervention *it.
// phase 0
trans->processConnection(cip.c_str(), cport, dip.c_str(), dport);
trans->processURI(uri.c_str(), method.c_str(), vers.c_str());
trans->intervention(it);
Now you have to check the it variable, eg. it.status. For more information, check the source.
Now consider you have a variable (a list) which contains the parsed headers. Now you have to add these headers one by one to the transaction:
for(your_iterator it=headerType.begin(); it != headerType.end(); ++it) {
const std::string key = it->first.as<std::string>(); // key
const std::string val = it->second.as<std::string>(); // val
trans->addRequestHeader(key, val);
}
Now you can process the headers and check the results. Note, that if you process a phase, the engine evaluates all rules, where the phase values is relevant: 1, 2, 3, 4 or 5.
// phase 1
trans->processRequestHeaders();
trans->intervention(it);
In next steps, you have to add the request body and process it, then get the response headers and body (from the upstream), and repeat the steps above...
Hope now you can see how does it works.
I've made a utility, which runs the CRS test cases on libmodecurity3 while it uses CRS rules. The tool available here: ftwrunner.
Hope this helped.
I am trying to write a gSoap server, which needs to save the incoming message. I want to save it in a buffer. Here is how gSoap Documentation does it on a client side to save the outgoing message...
//from gSoap Documentation
//SOURCE: https://www.genivia.com/doc/soapdoc2.html#tth_sEc19.7
int mysend(struct soap *soap, const char *s, size_t n)
{
struct buffer *h = (struct buffer*)soap->user; // get buffer through handle
int m = h->max, k = h->len + n;
// need to increase space?
if (m == 0)
m = 1024;
else
while (k >= m)
m *= 2;
if (m != h->max)
{
char *buf = malloc(m);
memcpy(buf, h->buf, h->len);
h->max = m;
if(h->buf)
free(h->buf);
h->buf = buf;
}
memcpy(h->buf + h->len, s, n);
h->len += n;
return SOAP_OK;
}
this works with some modification, but if I take this same idea to the server side, store to a buffer and end it with this return statement...
size_t myrecv(struct soap *soap, char *s, size_t n){
//do similar to above example...
...
return default_frecv(soap,s,n);
}
it only stores the message going from the server back to the client. I need to save the message coming to the server from the client. I thought recv would give me the incoming message, but this is not the case. Any Ideas? Any help, suggestions, or ideas are appreciated! Thanks in Advance!
source to the mysend example: https://www.genivia.com/doc/soapdoc2.html#tth_sEc19.7
I need to save the message coming to the server from the client. I
thought recv would give me the incoming message, but this is not the
case.
Hmmm. The documentation clearly says that frecv() is the callback that receives all messages, regardless whether it is the client side implementation or the server side implementation. So defining your own frecv() callback should so it. If you get stuck, take a look at the gsoap/plugin/logging.c message logger that implements this callback to capture the message received and sends it to a file/pipe.
I was able to hook into frecv() and do my own additional processing after invoking the default frecv() function. However, a new purpose requires me to parse the buffer before the default frecv(), at which point it doesn't yet exist. Once the default frecv() is called, the buffer exists, but the action taken on the buffer (namely soap_serve()), which I wanted to intercept, has already occurred, so I'm one request behind in my derived context.
My intent was to parse the buffer for the name of the WSDL, in order to assign a unique namespace to soap_serve(), but it's not working.
I'm currently trying to create a http server using Boost.Asio, I made it like this HTTP Server 3.
Currently I just read the Request and always return an OK Message. So nothing special or time consuming.
The Problem I come across is, running the Server with 12 Threads (16 cores # 2.53GHz), the server handles arround 200-300 requests per second.
I did the same in C# using HttpListener, running with 12 Threads, it handles arround 5000-7000 requests.
What the heck is Boost.Asio doing?
Using Instrumentation Profiling with Visual Studio get following "Functions With Most Individual Work":
Name Exclusive Time %
GetQueuedCompletionStatus 44,46
std::_Lockit::_Lockit 14,54
std::_Container_base12::_Orphan_all 3,46
std::_Iterator_base12::~_Iterator_base12 2,06
Edit 1:
if (!err) {
//Add data to client request
if(client_request_.empty())
client_request_ = std::string(client_buffer_.data(), bytes_transferred);
else
client_request_ += std::string(client_buffer_.data(), bytes_transferred);
//Check if headers complete
client_headerEnd_ = client_request_.find("\r\n\r\n");
if(client_headerEnd_ == std::string::npos) {
//Headers not yet complete, read again
client_socket_.async_read_some(boost::asio::buffer(client_buffer_),
boost::bind(&session::handle_client_read_headers, shared_from_this(),
boost::asio::placeholders::error,
boost::asio::placeholders::bytes_transferred));
} else {
//Search Cookie
std::string::size_type loc=client_request_.find("Cookie");
if(loc != std::string::npos) {
//Found Cookie
std::string::size_type locend=client_request_.find_first_of("\r\n", loc);
if(locend != std::string::npos) {
std::string lCookie = client_request_.substr(loc, (locend-loc)); loc = lCookie.find(": "); if(loc != std::string::npos) {
std::string sCookies = lCookie.substr(loc+2);
std::vector<std::string> vCookies;
boost::split(vCookies, sCookies, boost::is_any_of(";"));
for (std::size_t i = 0; i < vCookies.size(); ++i) {
std::vector<std::string> vCookie;
boost::split(vCookie, vCookies[i], boost::is_any_of("="));
if(vCookie[0].compare("sessionid") == 0) {
if(vCookie.size() > 1) {
client_sessionid_ = vCookie[1];
break;
}
}
} }
} }
//Search Content-Length
loc=client_request_.find("Content-Length");
if(loc == std::string::npos) {
//No Content-Length, no Content? -> stop further reading
send_bad_request();
return;
}
else {
//Parse Content-Length, for further body reading
std::string::size_type locend=client_request_.find_first_of("\r\n", loc);
if(locend == std::string::npos) {
//Couldn't find header end, can't parse Content-Length -> stop further reading
send_bad_request();
return;
}
std::string lHeader = client_request_.substr(loc, (locend-loc));
loc = lHeader.find(": ");
if(loc == std::string::npos) {
//Couldn't find colon, can't parse Content-Length -> stop further reading
send_bad_request();
return;
}
//Save Content-Length
client_request_content_length_ = boost::lexical_cast<std::string::size_type>(lHeader.substr(loc+2));
//Check if already read complete body
if((client_request_.size()-(client_headerEnd_)) < client_request_content_length_) {
//Content-Length greater than current body, start reading.
client_socket_.async_read_some(boost::asio::buffer(client_buffer_),
boost::bind(&session::handle_client_read_body, shared_from_this(),
boost::asio::placeholders::error,
boost::asio::placeholders::bytes_transferred));
}
else {
//Body is complete, start handling
handle_request();
}
}
}
}
Edit 2:
Client used for testing is a simple C#-Application which starts 128-Threads each iterate 1000 times without any Sleep.
System.Net.HttpWebRequest req = (System.Net.HttpWebRequest)System.Net.WebRequest.Create(BaseUrl);
req.Method = "POST";
byte[] buffer = Encoding.ASCII.GetBytes("{\"method\":\"User.Login\",\"params\":[]}");
req.GetRequestStream().Write(buffer, 0, buffer.Length);
req.GetRequestStream().Close();
The reason for the slowness probably is that Boost::Asio HTTP Server 3 example always closes the connection after each response, forcing the client to create a new connection for each request. Opening and closing connection on every request takes lots of time. Obviously, this could not outperform any server that supports HTTP/1.1 and Keep-alive (basically, doesn't close client connection and allows client to reuse it for subsequent requests).
Your C# server, System.Net.HttpListener, does support Keep-alive. The client, System.Net.HttpWebRequest, also has Keep-alive enabled by default. So, the connections are reused in this configuration.
Adding keep-alive to HTTP Server 3 example is straightforward:
inside connection::handle_read() check the request if client requested Keep-alive and store this flag within the connection
change connection::handle_write() so that it initiates graceful connection closure only when client doesn't support Keep-alive, otherwise just initiate async_read_some() like you already do in connection::start():
socket_.async_read_some(boost::asio::buffer(buffer_),
strand_.wrap(
boost::bind(&connection::handle_read, shared_from_this(),
boost::asio::placeholders::error,
boost::asio::placeholders::bytes_transferred)));
And don't forget to clear your request/reply and reset the request_parser before calling async_read_some().
it seems that client_request_.find("\r\n\r\n"); is called repeatedly -- hunting for the end tokens from the beginning of the string each loop. use a starting position position. such as client_request_.find("\r\n\r\n", lastposition); (using bytes_transferred)
its possible to use asycn_read_until( ,"\r\n\r\n"); found here
or async_read which should read all (instead of some).
About HTTP server 3 example. Look at the request_parser source code. The methods parse/consume. It is really not optimial cus it getting data from buffer byte-by-byte and working with each byte; pushing into std::string using push_back and so on. Its just an example.
Also, if you are using asio::strand notice that it uses a mutex t lock "strand implementation". For HTTP server its easily possible to remove asio::strand at all, so i recomment to do this. If you want to stay with strands - to avoid delays on locking you can set those defines at compile time:
-DBOOST_ASIO_STRAND_IMPLEMENTATIONS=30000 -DBOOST_ASIO_ENABLE_SEQUENTIAL_STRAND_ALLOCATION
I wrote a Http/Rest Client.
The Main Problem is that i recieve some unkown digits within the requested data.
I really don´t know where they come from..
e0b
<html>
<head>
[...]
</body>
</html>
0
You see the e0b and the 0 at the end.
In big xml files for example i got something like this:
<sometag id="somei
2000
d"><child>
...
</child></some
2000
tag>
It is irreproducible by me.
My Code:
// read the response status code
boost::asio::streambuf httpStreamBufferResponse;
boost::asio::read_until(httpSocket, httpStreamBufferResponse, "\r\n");
// check status code and validate
istream httpResponseIStream(&httpStreamBufferResponse);
// temp var for version
string sHttpVersion;
httpResponseIStream >> sHttpVersion;
// temp var for status code
unsigned int uiStatusCode;
httpResponseIStream >> uiStatusCode;
// fetch status message and switch it
string sStatusMessage;
getline(httpResponseIStream, sStatusMessage);
if(!httpResponseIStream || sHttpVersion.substr(0, 5) != "HTTP/"){
new Note(eNotesType(ERROR), "Request Interrupt", "Invalid Request Response");
Log::write("ERROR: Request Interrupt: Invalid Request Response");
}
// != 200 even means that something is not OK
if(uiStatusCode != 200){
this -> sHttpStatusCode = uiStatusCode;
new Note(eNotesType(WARNING), "Request Response "
+ boost::lexical_cast<string>(uiStatusCode), httpErrorToString.at(uiStatusCode));
Log::write("WARNING: Request Response "
+ boost::lexical_cast<string>(uiStatusCode) + ": " + httpErrorToString.at(uiStatusCode));
}
// Read the response headers, which are terminated by a blank line.
boost::asio::read_until(httpSocket, httpStreamBufferResponse, "\r\n\r\n");
// Process the response header
stringstream responseSStream;
string responseSHeader;
while (getline( httpResponseIStream, responseSHeader ) && responseSHeader != "\r" ) {
responseSStream << responseSHeader;
}
// store header in member variable
this -> sHttpResponseHeader = sHttpVersion + " " + boost::lexical_cast<string>(uiStatusCode) + " "
+ httpErrorToString.at(uiStatusCode) + "\n" + responseSStream.str();
// read until EOF and writing data to output as we go.
ostringstream responseOSStream;
while(boost::asio::read(httpSocket, httpStreamBufferResponse, boost::asio::transfer_at_least(1), error)){
responseOSStream << &httpStreamBufferResponse;
}
// store content in member variable
this -> sHttpResponseContent = responseOSStream.str();
// if there is no EOF
if(error != boost::asio::error::eof){
new Note(eNotesType(ERROR), "Request Interrupt", "Invalid Response End");
Log::write("ERROR: Request Interrupt: Invalid Response End");
}
// catch not known exceptions properly
} catch (exception& e){
string exceptionMessage = e.what();
new Note(eNotesType(ERROR), "Exception", exceptionMessage);
Log::write("ERROR: Exception: " + exceptionMessage);
}
// log http standby
Log::write("http status: standby");
It would be a great pleasure if anybody got ANY idea where this come from..?!
My nerves are on edge..
Your code is claiming HTTP/1.1 compliance and doesn't actually comply with HTTP/1.1's requirements. Either don't claim HTTP/1.1 compliance or make sure your code does everything the standard says a client must do.
All HTTP/1.1 applications MUST be able to receive and decode the "chunked" transfer-coding, and MUST ignore chunk-extension extensions they do not understand. -- HTTP/1.1 specification, section 3.6.1
I'm still working on some kind of client for communication with an IP Camera. Now I have the following issue:
I send a request to the camera ( a RTSP DESCRIBEin particular ). Now I get it's answer which looks like this:
RTSP/1.0 200 OK
CSeq: 2
Date: Thu, Jan 01 1970 00:31:41 GMT
Content-Base: rtsp://192.168.0.42/mpeg4?mode=Live&stream=-1&buffer=0&seek=0&fps=100& metainfo=/
Content-Type: application/sdp
Content-Length: 517
This is the header of the answer, followed by a so called Session Description which has the size shown in the field Content-Length. Actually I don't care much for the Session Description , I'm just interested in the Content-Base field. But still, since there is some communication following on the same socket, I need to get rid of all the data.
For receiving'm using the async_read calls from boost::asio.
My code looks ( simplified ) like this:
CommandReadBuffer::CallbackFromAsyncWrite()
{
boost::asio::async_read_until(*m_Socket, m_ReceiveBuffer,"\r\n\r\n",
boost::bind(&CommandReadBuffer::handle_rtsp_describe, this->shared_from_this(),
boost::asio::placeholders::error,boost::asio::placeholders::bytes_transferred));
}
This one reads at least the header ( shown above ) since its terminated by a blank line. As usual for async_write it just reads some more of the data, but nevermind. Now to the next callback function:
void CommandReadBuffer::handle_rtsp_describe(const boost::system::error_code& err,size_t bytesTransferred)
{
std::istream response_stream(&m_ReceiveBuffer);
std::string header;
// Just dump the data on the console
while (std::getline(response_stream, header))
{
// Normally I would search here for the desired content-base field
std::cout << header << "\n";
}
boost::asio::async_read(*m_Socket, m_ReceiveBuffer, boost::asio::transfer_at_least(1),
boost::bind(&CommandReadBuffer::handle_rtsp_setup, this->shared_from_this(),
boost::asio::placeholders::error, boost::asio::placeholders::bytes_transferred));
}
Now this works fine as well, if I print out the number of received bytes it's always 215.
Now we go on to the critical callback:
void CommandReadBuffer::handle_rtsp_setup(const boost::system::error_code& err, size_t bytesTransferred)
{
std::cout << "Error: " << err.message() << "\n";
if (!err)
{
// Write all of the data that has been read so far.
std::cout << &m_ReceiveBuffer;
// Continue reading remaining data until EOF.
m_DeadlineTimer->async_wait(boost::bind(&CommandReadBuffer::handleTimeout, this->shared_from_this(),boost::asio::placeholders::error));
boost::asio::async_read(*m_Socket, m_ReceiveBuffer, boost::asio::transfer_at_least(1),
boost::bind(&CommandReadBuffer::handle_rtsp_setup, this->shared_from_this(),
boost::asio::placeholders::error, boost::asio::placeholders::bytes_transferred));
}
else if (err != boost::asio::error::eof)
{
std::cout << "Error: " << err.message() << "\n";
}
else
{
std::cout << "End of Frame " << err.message() << "\n";
}
}
This part reads 220 Bytes. If I look at console Output from this call and compare it with the actualy payload of the frame ( as seen in Wireshark ) I can see that all data has been received. Now I would actually assume that async_read would set me the eof error. But instead the return code of error is success and so it calls async_read again. This time there is no data to be received and it never calls the callback function ( since there will be no more incoming data ).
Now I actually don't know how I could determine that all data has been sent. Actually I would expect the error flag to be set.
Now this is very similar to the implementation of the Boost Example for an Async HTTP client. Also it is done the same way in the Example Boost Async HTTP Client. I implemented this in another call and there it actually works.
Now in my opinion it should make no difference for the async_read call wether it is HTTP or RTSP - end of frame is end of frame, if there is no more data to read.
I'm also aware that according to the boost documentation I am using
void async_read(
AsyncReadStream & s,
basic_streambuf< Allocator > & b,
CompletionCondition completion_condition,
ReadHandler handler);
which means the function will continue until
The supplied buffer is full (that is, it has reached maximum size).
The completion_condition function object returns 0.
So if there is no more data to read, it just continues.
But I also tried the overloaded function without the CompletionCondition parameter, which should return when an error occurs ( EOF !!! ) - But this just won't callback either...
Any suggestions? I just don't get what I'm doing wrong...
I have written an RTSP client and server library using boost asio and can offer the following advice:
The RTSP message syntax is generic: there is no need for different DESCRIBE and SETUP handlers. In general
write an RTSP request
to read the response do a boost::asio::async_read_until("\r\n\r\n")
then check for the Content-Length header
if content_length > 0 do a boost::asio::transfer_at_least(content_length)
Further, why are you expecting an EOF? The connection is still open: the server is waiting for either another SETUP or a PLAY request and typically won't close the connection until the RTSP TCP connection has been timed out, which has a default value of 60 seconds according to RFC2326.
If in your application, you have completed interaction with the RTSP server, close the connection after you have read the response.