I have to use class Matrix4x4 from some 3rd party library, and I need to serialize it.
1. Is it ok to create header(3rdparty_serialization.h) that will contain all serialization that is needed for 3rd party libraries, like Matrix4x4:
namespace boost {
namespace serialization {
template<class Archive>
void serialize(Archive & ar, Matrix4x4 & m, const unsigned int version)
{
for(size_t i = 0; i < 4;++i)
for(size_t j = 0; j < 4;++j)
{
auto& e = m[i][j];
ar & BOOST_SERIALIZATION_NVP(e);
}
}
} // namespace serialization
} // namespace boost
2. Is this definition of function "serialize" for Matrix4x4 correct?
3. How to customize formatting of Matrix4x4 serialization?
Now, I have output:
<m class_id="2" tracking_level="0" version="0">
<e>1</e>
<e>0</e>
<e>0</e>
<e>0</e>
<e>0</e>
<e>1</e>
<e>0</e>
<e>0</e>
<e>0</e>
<e>0</e>
<e>1</e>
<e>0</e>
<e>0</e>
<e>0</e>
<e>0</e>
<e>1</e>
</m>
I want something like this:
<m class_id="2" tracking_level="0" version="0">
<e>1;0;0;0</e>
<e>0;1;0;0</e>
<e>0;0;1;0</e>
<e>0;0;0;1</e>
</m>
or other more compact and readable form.
There a couple of ways to do this. The easiest way would be to make a special verision of serialize for xml_archive. This would keep the change from infecting other archive types. Try something like:
template<>
void save(boost::archive::xml_archive & ar, const Matrix4x4 & m, const unsigned int version)
{
std::string e;
for(size_t i = 0; i < 4;++i){
for(size_t j = 0; j < 4;++j)
{
e += std::string(m[i][j]);
e += ';';
}
}
ar << BOOST_SERIALIZATION_NVP(e)
}
// similar for load
Robert Ramey
1. Yes.
2. Yes, assuming Matrix is not within a namespace.
3. You could try serializing std::strings instead of individual elements. However, this is a bit wasteful since you will need to format and parse the strings. Also not optimal for size if you want to use e.g. binary_[io]archive.
Related
In some testing code there's a helper function like this:
auto make_condiment(bool salt, bool pepper, bool oil, bool garlic) {
// assumes that first bool is salt, second is pepper,
// and so on...
//
// Make up something according to flags
return something;
};
which essentially builds up something based on some boolean flags.
What concerns me is that the meaning of each bool is hardcoded in the name of the parameters, which is bad because at the call site it's hard to remember which parameter means what (yeah, the IDE can likely eliminate the problem entirely by showing those names when tab completing, but still...):
// at the call site:
auto obj = make_condiment(false, false, true, true); // what ingredients am I using and what not?
Therefore, I'd like to pass a single object describing the settings. Furthermore, just aggregating them in an object, e.g. std::array<bool,4>.
I would like, instead, to enable a syntax like this:
auto obj = make_smart_condiment(oil + garlic);
which would generate the same obj as the previous call to make_condiment.
This new function would be:
auto make_smart_condiment(Ingredients ingredients) {
// retrieve the individual flags from the input
bool salt = ingredients.hasSalt();
bool pepper = ingredients.hasPepper();
bool oil = ingredients.hasOil();
bool garlic = ingredients.hasGarlic();
// same body as make_condiment, or simply:
return make_condiment(salt, pepper, oil, garlic);
}
Here's my attempt:
struct Ingredients {
public:
enum class INGREDIENTS { Salt = 1, Pepper = 2, Oil = 4, Garlic = 8 };
explicit Ingredients() : flags{0} {};
explicit Ingredients(INGREDIENTS const& f) : flags{static_cast<int>(f)} {};
private:
explicit Ingredients(int fs) : flags{fs} {}
int flags; // values 0-15
public:
bool hasSalt() const {
return flags % 2;
}
bool hasPepper() const {
return (flags / 2) % 2;
}
bool hasOil() const {
return (flags / 4) % 2;
}
bool hasGarlic() const {
return (flags / 8) % 2;
}
Ingredients operator+(Ingredients const& f) {
return Ingredients(flags + f.flags);
}
}
salt{Ingredients::INGREDIENTS::Salt},
pepper{Ingredients::INGREDIENTS::Pepper},
oil{Ingredients::INGREDIENTS::Oil},
garlic{Ingredients::INGREDIENTS::Garlic};
However, I have the feeling that I am reinventing the wheel.
Is there any better, or standard, way of accomplishing the above?
Is there maybe a design pattern that I could/should use?
I think you can remove some of the boilerplate by using a std::bitset. Here is what I came up with:
#include <bitset>
#include <cstdint>
#include <iostream>
class Ingredients {
public:
enum Option : uint8_t {
Salt = 0,
Pepper = 1,
Oil = 2,
Max = 3
};
bool has(Option o) const { return value_[o]; }
Ingredients(std::initializer_list<Option> opts) {
for (const Option& opt : opts)
value_.set(opt);
}
private:
std::bitset<Max> value_ {0};
};
int main() {
Ingredients ingredients{Ingredients::Salt, Ingredients::Pepper};
// prints "10"
std::cout << ingredients.has(Ingredients::Salt)
<< ingredients.has(Ingredients::Oil) << "\n";
}
You don't get the + type syntax, but it's pretty close. It's unfortunate that you have to keep an Option::Max, but not too bad. Also I decided to not use an enum class so that it can be accessed as Ingredients::Salt and implicitly converted to an int. You could explicitly access and cast if you wanted to use enum class.
If you want to use enum as flags, the usual way is merge them with operator | and check them with operator &
#include <iostream>
enum Ingredients{ Salt = 1, Pepper = 2, Oil = 4, Garlic = 8 };
// If you want to use operator +
Ingredients operator + (Ingredients a,Ingredients b) {
return Ingredients(a | b);
}
int main()
{
using std::cout;
cout << bool( Salt & Ingredients::Salt ); // has salt
cout << bool( Salt & Ingredients::Pepper ); // doesn't has pepper
auto sp = Ingredients::Salt + Ingredients::Pepper;
cout << bool( sp & Ingredients::Salt ); // has salt
cout << bool( sp & Ingredients::Garlic ); // doesn't has garlic
}
note: the current code (with only the operator +) would not work if you mix | and + like (Salt|Salt)+Salt.
You can also use enum class, just need to define the operators
I would look at a strong typing library like:
https://github.com/joboccara/NamedType
For a really good video talking about this:
https://www.youtube.com/watch?v=fWcnp7Bulc8
When I first saw this, I was a little dismissive, but because the advice came from people I respected, I gave it a chance. The video convinced me.
If you look at CPP Best Practices and dig deeply enough, you'll see the general advice to avoid boolean parameters, especially strings of them. And Jonathan Boccara gives good reasons why your code will be stronger if you don't directly use the raw types, for the very reason that you've already identified.
My class is receiving a vector of complex floats, but the data stream packets must have the data as short. My solution is to first convert the data to short, then batch it into the packets, and send them. What would be the best method of converting this data?
The complex data should map to two separate variables: for example, vcfSampleData[0].real() -> m_pData[0], and vcfSampleData[0].imag() -> m_pData[1] and so on.
void BufferSampleData(const std::vector<std::complex<float>>& vcfSampleData, const float& fTotalGain_dB, const Frequency& CentreFrequency, const Bandwidth& cBandwidth, const TimeStamp& tCaptureTime, const SampleRate& SampleRate)
{
sBufferData* pWriteElement = m_SampleBuffer.GetWriteElement();
if(pWriteElement != nullptr) {
std::copy(vcfSampleData.begin(), vcfSampleData.end(), pWriteElement->m_pData); //vector<comlex<float>> must become vector<complex<short>> or vector<short>
//other irrelevant stuff here
m_SampleBuffer.CommitWriteElement();
}
}
There isn't a std:: function that does what you are asking for, you'll just have to write a loop
pWriteElement->m_pData.reserve(vcfSampleData.size() * 2);
pWriteElement->m_pData.clear();
for (auto cmpl : vcfSampleData)
{
pWriteElement->m_pData.push_back(cmpl.real());
pWriteElement->m_pData.push_back(cmpl.imag());
}
KISS.
void Transform (const std::vector<std::complex<float>>& vSrc, const std::vector<short>& vDst) {
vDst.resize(2*vSrc.size());
for (std::size_t i=0; i<vSrc.size(); i++) {
vDst[2*i] = vSrc[i].real;
vDst[2*i+1] = vSrc[i].imag;
}
}
I want to create a python wrapper for my C++ library. It would be cool, if there is a automatic conversion of std::vector to python lists and the other way round.
Unfortunatly if I add this code to my Interface-file I still get errors in run-time.
%typemap(in) std::vector<float> value (std::vector<float> vIn) {
int iLen = PySequence_Length($input);
for(unsigned int i = 0; i < iLen; i++) {
PyObject *o = PySequence_GetItem($input, i);
if (PyNumber_Check(o)) {
vIn.push_back((float)PyFloat_AsDouble(o) );
}
}
$1 = vIn;
}
%typemap(out) std::vector<float> {
std::vector<float> vOut = $1;
int iLen = vOut.size();
$result = PyList_New(iLen);
for(unsigned int i = 0; i < iLen; i++) {
double fVal = vOut.at(i);
PyObject *o = PyFloat_FromDouble((double) fVal);
PyList_SetItem($result, i, o);
}
}
Class header:
class TrainingSet {
private:
std::vector<std::vector<float> > m_vInputList;
std::vector<std::vector<float> > m_vOutputList;
public:
void AddInput(const std::vector<float> &vIn);
// ..
Python code:
trainSet = TrainingSet()
trainSet.AddInput([0.5, 0.5, 0.5])
Error:
File "runSOMNet.py", line 9, in <module>
trainSet.AddInput([0.5, 0.5, 0.5])
File "/home/dgrat/annetgpgpu/build/ANNet/ANPyNetCPU.py", line 674, in AddInput
def AddInput(self, *args): return _ANPyNetCPU.TrainingSet_AddInput(self, *args)
NotImplementedError: Wrong number or type of arguments for overloaded function 'TrainingSet_AddInput'.
Possible C/C++ prototypes are:
ANN::TrainingSet::AddInput(std::vector< float,std::allocator< float > > const &)
ANN::TrainingSet::AddInput(float *,unsigned int const &)
The std_vector.i library in SWIG provides support for std::vector.
http://www.swig.org/Doc2.0/Library.html#Library_stl_cpp_library
You just need to tell SWIG about the template instantiations you want it to know about:
%include "std_vector.i"
namespace std {
%template(FloatVector) vector<float>;
}
Note that the following Python code will work, but will incur an array copy:
for x in range(0, 3):
list[x] = x
myModule.myFunction(list)
To do the same thing without incurring a copy, construct the list using the SWIG-generated proxy object constructor:
list = myModule.FloatVector()
for x in range(0, 3):
list[x] = x
myModule.myFunction(list)
If you're going to define the %typemaps manually, you will also need a %typemap(check), something like:
%typemap(typecheck) std::vector<float>& {
$1 = PySequence_Check($input) ? 1 : 0;
}
I believe the rule of thumb is that if you define a %typemap(in), you should also define a %typemap(check) --- otherwise, the generated code never gets to where it's put your %typemap(in).
I am using libsvm version 3.16. I have done some training in Matlab, and created a model. Now I would like to save this model to disk and load this model in my C++ program. So far I have found the following alternatives:
This answer explains how to save a model from C++, which is based on this website. Not exactly what I need, but could be adapted. (This requires development time).
I could find the best training parameters (kernel,C) in Matlab and re-train everything in C++. (Will require doing the training in C++ each time I change a parameter. It's not scalable).
Thus, both of these options are not satisfactory,
Does anyone have an idea?
My solution was to retrain in C++ because I couldn't find a nice way to directly save the model. Here's my code. You'll need to adapt it and clean it up a bit. The biggest change you'll have to make it not hard coding the svm_parameter values like I did. You'll also have to replace FilePath with std::string. I'm copying, pasting and making small edits here in SO so the formatting won't e perfect:
Used like this:
auto targetsPath = FilePath("targets.txt");
auto observationsPath = FilePath("observations.txt");
auto targetsMat = MatlabMatrixFileReader::Read(targetsPath, ',');
auto observationsMat = MatlabMatrixFileReader::Read(observationsPath, ',');
auto v = MiscVector::ConvertVecOfVecToVec(targetsMat);
auto model = SupportVectorRegressionModel{ observationsMat, v };
std::vector<double> observation{ { // 32 feature observation
0.883575729725847,0.919446119013878,0.95359403450317,
0.968233630936732,0.91891307107125,0.887897763183844,
0.937588566544751,0.920582702918882,0.888864454119387,
0.890066735260163,0.87911085669864,0.903745573664995,
0.861069296586979,0.838606194934074,0.856376230548304,
0.863011311537075,0.807688936997926,0.740434984165146,
0.738498042748759,0.736410940165691,0.697228384912424,
0.608527698289016,0.632994967880269,0.66935784966765,
0.647761430696238,0.745961037635717,0.560761134660957,
0.545498063585615,0.590854855113663,0.486827902942118,
0.187128866890822,- 0.0746523069562551
} };
double prediction = model.Predict(observation);
miscvector.h
static vector<double> ConvertVecOfVecToVec(const vector<vector<double>> &mat)
{
vector<double> targetsVec;
targetsVec.reserve(mat.size());
for (size_t i = 0; i < mat.size(); i++)
{
targetsVec.push_back(mat[i][0]);
}
return targetsVec;
}
libsvmtargetobjectconvertor.h
#pragma once
#include "machinelearning.h"
struct svm_node;
class LibSvmTargetObservationConvertor
{
public:
svm_node ** LibSvmTargetObservationConvertor::ConvertObservations(const vector<MlObservation> &observations, size_t numFeatures) const
{
svm_node **svmObservations = (svm_node **)malloc(sizeof(svm_node *) * observations.size());
for (size_t rowI = 0; rowI < observations.size(); rowI++)
{
svm_node *row = (svm_node *)malloc(sizeof(svm_node) * numFeatures);
for (size_t colI = 0; colI < numFeatures; colI++)
{
row[colI].index = colI;
row[colI].value = observations[rowI][colI];
}
row[numFeatures].index = -1; // apparently needed
svmObservations[rowI] = row;
}
return svmObservations;
}
svm_node* LibSvmTargetObservationConvertor::ConvertMatToSvmNode(const MlObservation &observation) const
{
size_t numFeatures = observation.size();
svm_node *obsNode = (svm_node *)malloc(sizeof(svm_node) * numFeatures);
for (size_t rowI = 0; rowI < numFeatures; rowI++)
{
obsNode[rowI].index = rowI;
obsNode[rowI].value = observation[rowI];
}
obsNode[numFeatures].index = -1; // apparently needed
return obsNode;
}
};
machinelearning.h
#pragma once
#include <vector>
using std::vector;
using MlObservation = vector<double>;
using MlTarget = double;
//machinelearningmodel.h
#pragma once
#include <vector>
#include "machinelearning.h"
class MachineLearningModel
{
public:
virtual ~MachineLearningModel() {}
virtual double Predict(const MlObservation &observation) const = 0;
};
matlabmatrixfilereader.h
#pragma once
#include <vector>
using std::vector;
class FilePath;
// Matrix created with command:
// dlmwrite('my_matrix.txt', somematrix, 'delimiter', ',', 'precision', 15);
// In these files, each row is a matrix row. Commas separate elements on a row.
// There is no space at the end of a row. There is a blank line at the bottom of the file.
// File format:
// 0.4,0.7,0.8
// 0.9,0.3,0.5
// etc.
static class MatlabMatrixFileReader
{
public:
static vector<vector<double>> Read(const FilePath &asciiFilePath, char delimiter)
{
vector<vector<double>> values;
vector<double> valueline;
std::ifstream fin(asciiFilePath.Path());
string item, line;
while (getline(fin, line))
{
std::istringstream in(line);
while (getline(in, item, delimiter))
{
valueline.push_back(atof(item.c_str()));
}
values.push_back(valueline);
valueline.clear();
}
fin.close();
return values;
}
};
supportvectorregressionmodel.h
#pragma once
#include <vector>
using std::vector;
#include "machinelearningmodel.h"
#include "svm.h" // libsvm
class FilePath;
class SupportVectorRegressionModel : public MachineLearningModel
{
public:
SupportVectorRegressionModel::~SupportVectorRegressionModel()
{
svm_free_model_content(model_);
svm_destroy_param(¶m_);
svm_free_and_destroy_model(&model_);
}
SupportVectorRegressionModel::SupportVectorRegressionModel(const vector<MlObservation>& observations, const vector<MlTarget>& targets)
{
// assumes all observations have same number of features
size_t numFeatures = observations[0].size();
//setup targets
//auto v = ConvertVecOfVecToVec(targetsMat);
double *targetsPtr = const_cast<double *>(&targets[0]); // why aren't the targets const?
LibSvmTargetObservationConvertor conv;
svm_node **observationsPtr = conv.ConvertObservations(observations, numFeatures);
// setup observations
//svm_node **observations = BuildObservations(observationsMat, numFeatures);
// setup problem
svm_problem problem;
problem.l = targets.size();
problem.y = targetsPtr;
problem.x = observationsPtr;
// specific to out training sets
// TODO: This is hard coded.
// Bust out these values for use in constructor
param_.C = 0.4; // cost
param_.svm_type = 4; // SVR
param_.kernel_type = 2; // radial
param_.nu = 0.6; // SVR nu
// These values are the defaults used in the Matlab version
// as found in svm_model_matlab.c
param_.gamma = 1.0 / (double)numFeatures;
param_.coef0 = 0;
param_.cache_size = 100; // in MB
param_.shrinking = 1;
param_.probability = 0;
param_.degree = 3;
param_.eps = 1e-3;
param_.p = 0.1;
param_.shrinking = 1;
param_.probability = 0;
param_.nr_weight = 0;
param_.weight_label = NULL;
param_.weight = NULL;
// suppress command line output
svm_set_print_string_function([](auto c) {});
model_ = svm_train(&problem, ¶m_);
}
double SupportVectorRegressionModel::Predict(const vector<double>& observation) const
{
LibSvmTargetObservationConvertor conv;
svm_node *obsNode = conv.ConvertMatToSvmNode(observation);
double prediction = svm_predict(model_, obsNode);
return prediction;
}
SupportVectorRegressionModel::SupportVectorRegressionModel(const FilePath & modelFile)
{
model_ = svm_load_model(modelFile.Path().c_str());
}
private:
svm_model *model_;
svm_parameter param_;
};
Option 1 is actually pretty reasonable. If you save the model in libsvm's C format through matlab, then it is straightforward to work with the model in C/C++ using functions provided by libsvm. Trying to work with matlab-formatted data in C++ will probably be much more difficult.
The main function in "svm-predict.c" (located in the root directory of the libsvm package) probably has most of what you need:
if((model=svm_load_model(argv[i+1]))==0)
{
fprintf(stderr,"can't open model file %s\n",argv[i+1]);
exit(1);
}
To predict a label for example x using the model, you can run
int predict_label = svm_predict(model,x);
The trickiest part of this will be to transfer your data into the libsvm format (unless your data is in the libsvm text file format, in which case you can just use the predict function in "svm-predict.c").
A libsvm vector, x, is an array of struct svm_node that represents a sparse array of data. Each svm_node has an index and a value, and the vector must be terminated by an index that is set to -1. For instance, to encode the vector [0,1,0,5], you could do the following:
struct svm_node *x = (struct svm_node *) malloc(3*sizeof(struct svm_node));
x[0].index=2; //NOTE: libsvm indices start at 1
x[0].value=1.0;
x[1].index=4;
x[1].value=5.0;
x[2].index=-1;
For SVM types other than the classifier (C_SVC), look at the predict function in "svm-predict.c".
You can specify one buffer for your file stream like that:
char buf[BUFFER_SIZE];
std::ofstream file("file", std::ios_base::binary | std::ios_base::out);
if (file.is_open())
{
file.rdbuf()->pubsetbuf(buf, BUFFER_SIZE);
file << "abcd";
}
What I want to do now, is using more than just one buffer:
char* buf[] = { new char[BUFFER_SIZE], new char[BUFFER_SIZE], new char[BUFFER_SIZE], };
Is it possible without creating a custom derivation of std::streambuf?
EDIT:
I think I need to explain what I want to do in more detail. Please consider the following situation:
- The file(s) I want to read won't fit into memory
- The file while be accessed by some kind of a binary jump search
So, if you split the file into logical pages of a specific size, then I would like to provide multiple buffers which are representing specific pages. This would increase performance when a file location is read and the related page is already in a buffer.
I gather from the comment that you want to do a kind of scatter-gather I/O. I'm pretty sure there's no support for that in the C++ standard I/O streams library, so you'll have to roll your own.
If you want to do this efficiently, you can use OS support for scatter-gather. E.g., POSIX/Unix-like systems have writev for this purpose.
There's nothing like this provided by Standard. However, depending on your platform, you can use Memory Mapped Files, which provide the same functionality. Windows and Linux both provide them.
I will take a look at boost::iostreams::mapped_file, but I think my requirement is much simpler. I've created a custom class derived from basic_filebuf.
template<typename char_type>
class basic_filemultibuf : public std::basic_filebuf<char_type/*, std::char_traits<char_type>*/>
{
private:
char_type** m_buffers;
std::ptrdiff_t m_buffer_count,
m_curent_buffer;
std::streamsize m_buffer_size;
protected:
virtual int_type overflow(int_type meta = traits_type::eof())
{
if (this->m_buffer_count > 0)
{
if (this->m_curent_buffer == this->m_buffer_count)
this->m_curent_buffer = 0;
this->basic_filebuf::setbuf(this->m_buffers[this->m_curent_buffer++], this->m_buffer_size);
}
return this->basic_filebuf::overflow(meta);
}
public:
basic_filemultibuf(basic_filebuf const& other)
: basic_filebuf(other),
m_buffers(NULL),
m_buffer_count(0),
m_curent_buffer(-1),
m_buffer_size(0)
{
}
basic_filemultibuf(basic_filemultibuf const& other)
: basic_filebuf(other),
m_buffers(other.m_buffers),
m_buffer_count(other.m_buffer_count),
m_curent_buffer(other.m_curent_buffer),
m_buffer_size(other.m_buffer_size)
{
}
basic_filemultibuf(FILE* f = NULL)
: basic_filemultibuf(basic_filebuf(f))
{
}
basic_filemultibuf* pubsetbuf(char** buffers, std::ptrdiff_t buffer_count, std::streamsize buffer_size)
{
if ((this->m_buffers = buffers) != NULL)
{
this->m_buffer_count = buffer_count;
this->m_buffer_size = buffer_size;
this->m_curent_buffer = 0;
}
else
{
this->m_buffer_count = 0;
this->m_buffer_size = 0;
this->m_curent_buffer = -1;
}
this->basic_filebuf::setbuf(NULL, 0);
return this;
}
};
Example usage:
typedef basic_filemultibuf<char> filemultibuf;
std::fstream file("file", std::ios_base::binary | std::ios_base::in | std::ios_base::out);
char** buffers = new char*[2];
for (int i = 0; i < n; ++i)
buffers[i] = new char[4096];
filemultibuf multibuf(*file.rdbuf());
multibuf.pubsetbuf(buffers, 2, 4096);
file.set_rdbuf(&multibuf);
//
// do awesome stuff with file ...
//
for (int i = 0; i < n; ++i)
delete[] buffers[i];
That's pretty much it. The only thing I would really like to do is offer this functionally for other streambufs, because the usage of multiple buffers should not be restricted to filebuf. But it seems to me it isn't possible without rewriting the file specific functions.
What do you think about that?