zlib compression seems to decompress via gunzip but not curl - c++

I've written a http server that only sends back compressed http responses:
https://github.com/ericcurtin/asio/commit/1d37a1d225d1e812a747b595c02f9770ebd75dd0
So if you you use curl to request the data and decompress the response by piping through gunzip it works fine:
curl -x "" 127.0.0.1:5000/main.cpp --output - | gunzip
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 758 100 758 0 0 740k 0 --:--:-- --:--:-- --:--:-- 740k
// g++ -O0 main.cpp server.cpp connection_manager.cpp request_handler.cpp
// connection.cpp reply.cpp mime_types.cpp request_parser.cpp -lboost_system
// -lpthread -lz
//
// run like: ./a.out 0.0.0.0 5000 .
//
// main.cpp
// ~~~~~~~~
//
// Copyright (c) 2003-2017 Christopher M. Kohlhoff (chris at kohlhoff dot com)
//
// Distributed under the Boost Software License, Version 1.0. (See accompanying
// file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
//
#include <iostream>
#include <string>
#include <boost/asio.hpp>
#include "server.hpp"
int main(int argc, char* argv[])
{
try
{
// Check command line arguments.
if (argc != 4)
{
std::cerr << "Usage: http_server <address> <port> <doc_root>\n";
std::cerr << " For IPv4, try:\n";
std::cerr << " receiver 0.0.0.0 80 .\n";
std::cerr << " For IPv6, try:\n";
std::cerr << " receiver 0::0 80 .\n";
return 1;
}
// Initialise the server.
http::server::server s(argv[1], argv[2], argv[3]);
// Run the server until stopped.
s.run();
}
catch (std::exception& e)
{
std::cerr << "exception: " << e.what() << "\n";
}
return 0;
}
But if you use curl using --compressed which works with other http servers like the one at example.com it fails after the first 512 bytes:
curl -x "" 127.0.0.1:5000/main.cpp --compressed
// g++ -O0 main.cpp server.cpp connection_manager.cpp request_handler.cpp
// connection.cpp reply.cpp mime_types.cpp request_parser.cpp -lboost_system
// -lpthread -lz
//
// run like: ./a.out 0.0.0.0 5000 .
//
// main.cpp
// ~~~~~~~~
//
// Copyright (c) 2003-2017 Christopher M. Kohlhoff (chris at kohlhoff dot com)
//
// Distributed under the Boost Software License, Version 1.0. (See accompanying
// file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
//
#include <iostream>
curl: (23) Failed writing received data to disk/application
#include <string
Any idea on how my compression could be fixed?

Related

fatal error: 'grpcpp/grpcpp.h' file not found C++

Context
I have been working with C++ for about the past 5-6 months and I'm beginning to learn gRPC. I have followed many tutorials online to get started, but I want to build a client-server communication app from scratch. Probably a bit too much, but I'm doing my best to understand how to get it all to work from the ground up rather than downloading, typing 'make', and then having a working product that I don't know how to implement into my own projects.
Goal: Create and run a simple C++ gRPC client-server communication
Versions
Using VSCode IDE.
Protoc = libprotoc 3.17.3
gRPC = 1.41.1
make = 3.81
Files
mathtest.proto
syntax = "proto3";
option java_package = "ex.grpc";
package mathtest;
// Defines the service
service MathTest {
// Function invoked to send the request
rpc sendRequest (MathRequest) returns (MathReply) {}
}
// The request message containing requested numbers
message MathRequest {
int32 a = 1;
int32 b = 2;
}
// The response message containing response
message MathReply {
int32 result = 1;
}
server.cpp
#include <string>
#include <grpcpp/grpcpp.h>
#include "mathtest.grpc.pb.h"
using grpc::Server;
using grpc::ServerBuilder;
using grpc::ServerContext;
using grpc::Status;
using mathtest::MathTest;
using mathtest::MathRequest;
using mathtest::MathReply;
class MathServiceImplementation final : public MathTest::Service {
Status sendRequest(
ServerContext* context,
const MathRequest* request,
MathReply* reply
) override {
int a = request->a();
int b = request->b();
reply->set_result(a * b);
return Status::OK;
}
};
void Run() {
std::string address("0.0.0.0:5000");
MathServiceImplementation service;
ServerBuilder builder;
builder.AddListeningPort(address, grpc::InsecureServerCredentials());
builder.RegisterService(&service);
std::unique_ptr<Server> server(builder.BuildAndStart());
std::cout << "Server listening on port: " << address << std::endl;
server->Wait();
}
int main(int argc, char** argv) {
Run();
return 0;
}
client.cpp
#include <string>
#include <grpcpp/grpcpp.h>
#include "mathtest.grpc.pb.h"
using grpc::Channel;
using grpc::ClientContext;
using grpc::Status;
using mathtest::MathTest;
using mathtest::MathRequest;
using mathtest::MathReply;
class MathTestClient {
public:
MathTestClient(std::shared_ptr<Channel> channel) : stub_(MathTest::NewStub(channel)) {}
int sendRequest(int a, int b) {
MathRequest request;
request.set_a(a);
request.set_b(b);
MathReply reply;
ClientContext context;
Status status = stub_->sendRequest(&context, request, &reply);
if(status.ok()){
return reply.result();
} else {
std::cout << status.error_code() << ": " << status.error_message() << std::endl;
return -1;
}
}
private:
std::unique_ptr<MathTest::Stub> stub_;
};
void Run() {
std::string address("0.0.0.0:5000");
MathTestClient client(
grpc::CreateChannel(
address,
grpc::InsecureChannelCredentials()
)
);
int response;
int a = 5;
int b = 10;
response = client.sendRequest(a, b);
std::cout << "Answer received: " << a << " * " << b << " = " << response << std::endl;
}
int main(int argc, char* argv[]){
Run();
return 0;
}
Steps taken for compilation
Use mathtest.proto to create the necessary files via 'protoc' (or protobuf) by executing these: protoc --grpc_out=. --plugin=protoc-gen-grpc=/opt/homebrew/bin/grpc_cpp_plugin mathtest.proto & protoc --cpp_out=. mathtest.proto
This creates the following files:
mathtest.pb.h
mathtest.pb.cc
mathtest.grpc.pb.h
mathtest.grpc.pb.cc
Compile client.cpp & server.cpp files to create executable binaries using these commands: g++ -std=c++17 client.cpp mathtest.pb.cc mathtest.grpc.pb.cc -o client 'pkg-config --libs protobuf grpc++' (NOTE: in this post, I use a single quote in the command line, but in the actual command I use a backtick; just wanted to make that clear)
Errors
As you may notice, I can't get to compiling the server because I can't get past the client compilation first. After executing the above command in step 2 of compilation, this is my output:
g++ -std=c++17 client.cpp mathtest.pb.cc mathtest.grpc.pb.cc -o client `pkg-config --libs protobuf grpc++`
client.cpp:4:10: fatal error: 'grpcpp/grpcpp.h' file not found
#include <grpcpp/grpcpp.h>
^~~~~~~~~~~~~~~~~
1 error generated.
In file included from mathtest.pb.cc:4:
./mathtest.pb.h:10:10: fatal error: 'google/protobuf/port_def.inc' file not found
#include <google/protobuf/port_def.inc>
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.
In file included from mathtest.grpc.pb.cc:5:
./mathtest.pb.h:10:10: fatal error: 'google/protobuf/port_def.inc' file not found
#include <google/protobuf/port_def.inc>
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.
make: *** [client] Error 1
Here's my real confusion...
C++ intellisense has no issues finding these files. My $PATH variables point to these folders, and my VS Code include path also point to these folders. I'm unsure where I am going wrong here...
echo $PATH returns this:
/opt/homebrew/bin:/opt/homebrew/sbin:/opt/homebrew/include:/opt/homebrew/Cellar:/opt/homebrew/opt/libtool/libexec/gnubin:/usr/bin:/bin:/usr/sbin:/sbin:/Users/tzeller/.local/bin
The folders in question ('google' & 'grcpp') live within /opt/homebrew/include and they hold the necessary files as well...
What am I missing??
Change your compile command to
g++ -std=c++17 client.cpp mathtest.pb.cc mathtest.grpc.pb.cc -o client `pkg-config --libs --cflags protobuf grpc++`
The --cflags bit asks pkg-config to spit out the necessary parameters for setting the header search path (on my system -I/opt/homebrew/Cellar/grpc/1.41.1/include and others)

SOAP 1.2 fault SOAP-ENV:Sender[no subcode]

Accessing IIS webservice using Gsoap.
I have similar error as in this link shows that the error was solved compiling with -lssl.
I did the same thing in build as
g++ -o client client.cpp stdsoap2.cpp soapC.cpp soapDataManagementSoapProxy.cpp -I /usr/local/ssl/include -L/home/xavier/GSOAP/lib -lgsoapssl++ -L/usr/local/ssl/lib -lssl
My GSOAP lib was build with OpenSSL.
But I still have error as
SOAP 1.2 fault SOAP-ENV:Sender[no subcode]
"OpenSSL not installed: recompile with -DWITH_OPENSSL"
Detail: [no detail]
My test code is as follow. What could be wrong?
#include "soapDataManagementSoapProxy.h"
#include "DataManagementSoap.nsmap"
const char server[] = "https://XXXXXXX.com/XXXmanagement.asmx";
int main(int argc, char **argv)
{
DataManagementSoapProxy webinf;
webinf.soap_endpoint = server;
_tempuri__ReadTestData* a;
_tempuri__ReadTestDataResponse res;
int ret = webinf.ReadTestData(a, res);
if (webinf.error){
webinf.soap_stream_fault(std::cerr);
}
else{
//printf("result = %g\n", result);
std::cout << "Success " << std::endl;
}
webinf.destroy(); /* clean up mem */
return 0;
}
Solution for this issue is
#include "calc.nsmap"
#include "soapcalcProxy.h" // generated with soapcpp2 -j calc.h
calcProxy calc("https-server-endpoint-URL");
double sum;
soap_ssl_init(); // init SSL (just need to do this once in an application)
// soap_ssl_no_init(); // or prevent init OpenSSL when already initialized elsewhere in an application
if (soap_ssl_client_context(calc.soap,
SOAP_SSL_DEFAULT,
NULL, // no keyfile
NULL, // no keyfile password
"cacerts.pem", // trusted certificates (or use self-signed cacert.pem)
NULL, // no capath to trusted certificates
NULL // no random data to seed randomness
))
{
calc.soap_stream_fault(std::cerr);
exit(EXIT_FAILURE);
}
if (calc.add(1.23, 4.56, sum) == SOAP_OK)

Boost asio async_accept works under Windows but fails under FreeBSD. What's wrong?

EDIT: Looks like it's not my code but the build environment. That's both good and bad because now I know that the code is OK but have no idea how to debug the environment. Any suggestions here? Bear in mind that I do not have admin privileges on this computer.
I'm stuck trying to make simple code work under FreeBSD. It's a call to async_accept from Boost 1.64 asio that does not behave. Same code works fine under Windows but under FreeBSD it accepts the client connection (the connect call on the client end succeeds) but never calls its handler. Not even sure how to approach this. (Note that unlike other related questions I DO call io_service.run()) Please help.
Self contained code that shows the problem:
#include <iostream>
#include <boost/bind.hpp>
#include <boost/asio.hpp>
namespace asio = boost::asio;
namespace ph = asio::placeholders;
namespace sys = boost::system;
using asio::ip::tcp;
static void accept_handler(const sys::error_code& error)
{
// THIS IS NEVER CALLED UNDER FREEBSD
if (error)
std::cout << "failed to connected to server" << std::endl;
else
std::cout << "connected to server" << std::endl;
}
int main(int argc, char* argv[])
{
if (argc < 3)
{
std::cerr << "Usage: accept_test <port> <1 for async and 0 for sync accept>" << std::endl;
return 1;
}
asio::io_service io_service;
tcp::acceptor acceptor(io_service, tcp::endpoint(tcp::v4(), atoi(argv[1])));
std::cout << "waiting for server connection ";
tcp::socket sock(io_service);
if (argv[2][0] == '1')
{
//THIS WORKS UNDER WIN BUT DOESN'T CALL HANDLER UNDER FREEBSD
std::cout << "using async accept..." << std::endl;
acceptor.async_accept(sock, boost::bind(&accept_handler, ph::error));
}
else
{
//THIS WORKS FINE UNDER BOTH WIN AND FREEBSD
std::cout << "using sync accept..." << std::endl;
sys::error_code error;
acceptor.accept(sock, error);
if (error)
std::cout << "failed to connected to server" << std::endl;
else
std::cout << "connected to server" << std::endl;
}
io_service.run();
return 0;
}
I suspect an issue with your build environment. Both sync and async work on:
FreeBSD 12.0-CURRENT (GENERIC) #0 r319859: Mon Jun 12 19:37:22 UTC 2017
FreeBSD clang version 4.0.0 (tags/RELEASE_400/final 297347) (based on LLVM 4.0.0)
Target: x86_64-unknown-freebsd12.0
Thread model: posix
InstalledDir: /usr/bin
Boost 1.64.0 (from ports)
Built using
# clang++ -std=c++11 -o test -isystem /usr/local/include/ ./test.cpp -L /usr/local/lib -lboost_system
Resulting binary:
# ldd test
test:
libboost_system.so.1.64.0 => /usr/local/lib/libboost_system.so.1.64.0 (0x80083c000)
libc++.so.1 => /usr/lib/libc++.so.1 (0x800a3f000)
libcxxrt.so.1 => /lib/libcxxrt.so.1 (0x800d06000)
libm.so.5 => /lib/libm.so.5 (0x800f24000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x801151000)
libc.so.7 => /lib/libc.so.7 (0x801367000)
libthr.so.3 => /lib/libthr.so.3 (0x80172e000)
Rough instructions for build env:
vagrant init freebsd/FreeBSD-12.0-CURRENT
vagrant up --provider virtualbox
vagrant ssh
su
pkg install lang/clang-devel
pkg install devel/boost-all
# copy test.cpp
My guess is that you're running Boost 1.57 or earlier. The kqueue reactor was broken in some releases with exactly the symptoms you are seeing. You should upgrade to a later version. It works fine with 1.59 on FreeBSD for me.

boost program_options custom parsing

Is there a way to have something like
myapp hostname:port
parsed by boost program_options? I'm also using other options and I would love to use boost program_options without having to roll my own parser for argc/argv.
I tried with some combinations of
desc.add_options()
("help", "list all available options")
(new MyCustomValue(&store_var), "")
but it didn't work
As Dan Mašek writes, this looks more fitting for a positional argument. Since you specify a specific structure for the option, though, you might want to add std::regex_match.
Say you start with
#include <string>
#include <iostream>
#include <regex>
#include <boost/program_options.hpp>
using namespace std;
using namespace boost::program_options;
int main(int argc, const char *argv[]) {
try {
options_description desc{"Options"};
desc.add_options()
("help,h", "Help screen")
("ip_port", value<std::string>()->required(), "ip:port");
positional_options_description pos_desc;
pos_desc.add("ip_port", -1);
command_line_parser parser{argc, argv};
parser.options(desc).positional(pos_desc).allow_unregistered();
parsed_options parsed_options = parser.run();
variables_map vm;
store(parsed_options, vm);
notify(vm);
const auto ip_port = vm["ip_port"].as<string>();
At this point, we have the user's input for ip_port. We can define a regex to match it. Note that first part can be a string (e.g., localhost), but the second part must be an integer:
const regex ip_port_re("([^:]+):([[:digit:]]+)");
smatch ip_port_match;
if(!regex_match(ip_port, ip_port_match, ip_port_re))
throw validation_error{validation_error::invalid_option_value, ip_port, "ip_port"};
cout << "ip: " << ip_port_match[1] << " port: " << ip_port_match[2] << endl;
}
catch (const error &ex) {
cerr << ex.what() << '\n';
}
}
When running it, it looks like this:
$ g++ --std=c++11 po.cpp -lboost_program_options && ./a.out 127.0.0.1:30
ip: 127.0.0.1 port: 30
$ g++ --std=c++11 po.cpp -lboost_program_options && ./a.out localhost:30
ip: localhost port: 30
$ g++ --std=c++11 po.cpp -lboost_program_options && ./a.out localhost:30d
the argument for option 'localhost:30d' is invalid
$ g++ --std=c++11 po.cpp -lboost_program_options && ./a.out
the option '--ip_port' is required but missing
$ g++ --std=c++11 po.cpp -lboost_program_options && ./a.out localhost:30 foo
option '--ip_port' cannot be specified more than once

Linux C++ userspace app real time sleep function (POSIX, Raspberry Pi)

I need a function which suspends my program in µs, it should be real time, so if I call it with 50µs the thread should stop for exactly 50µs.
My C++ program is running on a Raspberry Pi with normal Raspbian installed.
I wrote this example program which uses the posix time functions to suspend an measure the suspend time.
#include <cstdlib>
#include "stdint-gcc.h"
#include "signal.h"
#include <time.h>
#include <cerrno>
#include <cstdio>
#include <iostream>
#include <cstring>
#define S_IN_NS 1000000000UL
#define MS_IN_NS 1000000UL
#define US_IN_NS 1000UL
#define GET_TIME_CLOCK CLOCK_MONOTONIC
using namespace std;
int main(int argc, char** argv) {
struct timespec newTimeStamp;
struct timespec oldTimeStamp;
struct timespec sleeptime;
sleeptime.tv_sec = 0;
sleeptime.tv_nsec = 50000; //50us
if (clock_gettime(GET_TIME_CLOCK, &oldTimeStamp) == -1)
cout << "Could not get clock time! ERRNO: " << strerror(errno);
if ((clock_nanosleep(CLOCK_MONOTONIC, 0, &sleeptime, NULL)) == -1)
cout << "Sleep failed! ERRNO: " << strerror(errno);
if (clock_gettime(GET_TIME_CLOCK, &newTimeStamp) == -1)
cout << "Could not get clock time! ERRNO: " << strerror(errno);
uint64_t measuredSec = (newTimeStamp.tv_sec - oldTimeStamp.tv_sec);
int32_t measuredNs = (newTimeStamp.tv_nsec - oldTimeStamp.tv_nsec);
uint64_t diffus = (((measuredSec * S_IN_NS) + measuredNs + 500) / 1000UL);
uint64_t diffns = (((measuredSec * S_IN_NS) + measuredNs));
cout << "Diffns:" << diffns << " Diffus:" << diffus << endl;
return 0;
}
Build commands:
arm-bcm2708hardfp-linux-gnueabi-g++ -lrt -c -g -MMD -MP -MF "build/Debug/GNU_ARM_HARDFP-Linux-x86/main.o.d" -o build/Debug/GNU_ARM_HARDFP-Linux-x86/main.o main.cpp
arm-bcm2708hardfp-linux-gnueabi-g++ -lrt -o dist/Debug/GNU_ARM_HARDFP-Linux-x86/timetest build/Debug/GNU_ARM_HARDFP-Linux-x86/main.o
Result (chrt - manipulate real-time attributes of a process):
pi#raspberrypi ~ $ sudo chrt 99 ./timetest
Diffns:130994 Diffus:131
pi#raspberrypi ~ $ sudo chrt 99 ./timetest
Diffns:135994 Diffus:136
pi#raspberrypi ~ $ sudo chrt 99 ./timetest
Diffns:138993 Diffus:139
The program should sleep for exactly 50us but I measured 130-139us.
If I change the GET_TIME_CLOCK define to CLOCK_PROCESS_CPUTIME_ID the cpu time (excluding the sleeptime) is measured (So as I understand it).
Result:
pi#raspberrypi ~ $ sudo chrt 99 ./timetest
Diffns:89000 Diffus:89
pi#raspberrypi ~ $ sudo chrt 99 ./timetest
Diffns:86000 Diffus:86
pi#raspberrypi ~ $ sudo chrt 99 ./timetest
Diffns:88000 Diffus:88
It takes around 80-90µs to do the clock_nanosleep() function, even if I change the sleeptime to 500µs!
So is there any way to suspend a thread for exactly an amount of time (µs) in a C++ userspace application on Raspbian?
thx
If you need to sleep for such a precise amount of time, you probably need to use a spin loop with a check of the current time. This will consume rather a lot of power (and generate heat), but it's a fairly reliable and portable way to do it. Another idea is to try the ideas on this page: http://blog.regehr.org/archives/794