I am implementing serialization that's supposed to work by recursive overloading until serializing data breaks down to primitive types. Here it's relevant code at serialization.hpp:
#pragma once
#include <fstream>
#include <ranges>
#include <concepts>
using namespace std;
void write( fstream & f, const integral auto & data );
void write( fstream & f, const pair<auto,auto> & p );
void write( fstream & f, const ranges::range auto & data );
void write( fstream & f, const integral auto & data ) {
f.write( (const char*)&data, sizeof( data ) );
}
void write( fstream & f, const pair<auto,auto> & p ) {
write( f, p.first );
write( f, p.second );
}
void write( fstream & f, const ranges::range auto & data ) {
const uint64_t size = ranges::size( data );
write( f, size );
for ( const auto & i : data ) {
write( f, i );
}
}
And it works well for most types providing serialization for free even for std::map, std::list, etc. If user needs to serialize any custom type then it's needed to just provide a void write( fstream & f, const CustomType & t ) { ... } function and it supposed to get serialized even inside any data structure. And it works well until CustomData gets wrapped into a namespace. main.cpp:
#include "serialization.hpp"
#include <fstream>
#include <map>
using namespace std;
struct MyData {
int custom_data[ 666 ];
};
void write( fstream & f, const MyData & d ) {
write( f, d.custom_data[ 123 ] );
}
namespace my_namespace {
struct NamespacedData {
int custom_data[ 777 ];
};
}//namespace my_namespace
void write( fstream & f, const my_namespace::NamespacedData & d ) {
write( f, d.custom_data[ 7 ] );
}
int main() {
auto f = fstream( "my_data.binary", ios::out | ios::binary );
f.exceptions( fstream::badbit | fstream::failbit | fstream::eofbit );
map< int, MyData > my_map;
//DOES work:
write( f, my_map );
map< int, my_namespace::NamespacedData > namespaced_map;
//does NOT work:
write( f, namespaced_map );
}
Compilation fails with error: serialization.hpp:16:10: error: no matching function for call to ‘write(std::fstream&, const my_namespace::NamespacedData&)’ ...
How function argument's full name matters here? Isn't it's irrelevant when compiler should try to match any type when looking for fitting overloaded function? Or it's just gcc bug?
Command: g++ main.cpp -std=gnu++20 -fconcepts
g++ version 11.3.0
Related
Here is data serialization program which compiles with gcc 11.1:
#include <fstream>
#include <ranges>
#include <concepts>
#include <map>
using namespace std;
void write( fstream & f, const integral auto & data ) {
f.write( (const char*)&data, sizeof( data ) );
}
void write( fstream & f, const pair<auto,auto> & p ) {
write( f, p.first );
write( f, p.second );
}
template< ranges::range T >
void write( fstream & f, const T & data ) {
const uint64_t size = ranges::size( data );
write( f, size );
for ( const auto & i : data )
write( f, i );
}
int main() {
auto f = fstream( "spagetti", ios::out | ios::binary );
const bool pls_compile = true;
if constexpr (pls_compile) {
write( f, pair< int, int >( 123, 777 ) );
write( f, map< int, int >{ { 1, 99 }, { 2, 98 } } );
}
else
write( f, pair< map< int, int >, int >( { { 1, 99 }, { 2, 98 } }, 777 ) );
}
which made me a happy developer of serialization library which provides functions to serialize to file anything that is integral || pair || range. But it turned out that if you set pls_compile = false, then compilation fails with spagetti.cpp:12:10: error: no matching function for call to 'write(std::fstream&, const std::map<int, int>&)'. It can't be fixed by moving declaration of write( pair ) past write( range ) because pls_compile = true will stop compiling then.
What is the best way to fix it and why compiler forgets about existence of write( range ) while generating implementation of write( pair< map, int > ) based on write( pair ) template? It clearly should be already familiar with write( range ) since it already proceeded down to write( pair< map, int > ).
In order for a function to participate in overload resolution, it must be declared before the point of the use in the compilation unit. If it is not declared until after the point of use, it will not be a possible overload, even if the compile has proceeded past the point of definition (eg, if the use is in a instatiation of a template defined before the declation but triggered by a use of a template afterwards, as you have here.)
The easiest fix is to just forward declare your templates/functions. You can't go wrong forward declaring everything at the top of the file:
void write( fstream & f, const integral auto & data );
void write( fstream & f, const pair<auto,auto> & p );
template< ranges::range T >
void write( fstream & f, const T & data );
As the other answer stated, you could declare all your functions first before define any of them. However, this answer is about OP's another question
Is there a way to somehow force compiler to respect all the declarations including future ones?
Normally, you could use some Customisation Point technique to achieve your goal, such as C++20 CPO (niebloids), and plain old ADL.
But in your particular example, all types are the STL library types and inside namespace std. Thus you cannot define your functions inside the std to make them part of ADL. Thus C++20 CPO and plain ADL won't work.
There are several approaches which can make your example work.
tag_invoke
#include <fstream>
#include <ranges>
#include <concepts>
#include <map>
using namespace std;
namespace lib{
inline constexpr struct write_fn{
// skip noexcept and requires for simplicity
decltype(auto) operator()(std::fstream& f, const auto& t) const {
return tag_invoke(*this, f, t);
}
} write{};
void tag_invoke(write_fn, std::fstream & f, const integral auto & data){
f.write( (const char*)&data, sizeof( data ) );
}
void tag_invoke(write_fn, std::fstream & f, const pair<auto,auto> & p ) {
lib::write( f, p.first );
lib::write( f, p.second );
}
void tag_invoke(write_fn, std::fstream & f, const std::ranges::range auto& data) {
const uint64_t size = std::ranges::size( data );
lib::write(f, size );
for (const auto& i : data){
lib::write( f, i );
}
}
}
int main() {
auto f = std::fstream( "spagetti", std::ios::out | std::ios::binary );
lib::write(f, std::pair< int, int >( 123, 777 ) );
lib::write(f, std::map< int, int >{ { 1, 99 }, { 2, 98 } } );
lib::write(f, std::pair<std::map< int, int >, int >( { { 1, 99 }, { 2, 98 } }, 777 ) );
}
godbolt link
This works also because of ADL. as the tag write_fn is inside lib namespace
Class Template Specialisation
#include <fstream>
#include <ranges>
#include <concepts>
#include <map>
using namespace std;
namespace lib{
template <class T>
struct write_impl;
inline constexpr struct write_fn{
// skip noexcept and requires for simplicity
decltype(auto) operator()(std::fstream& f, const auto& t) const {
return write_impl<std::decay_t<decltype(t)>>::apply(f,t);
}
} write{};
template <integral T>
struct write_impl<T>{
static void apply(std::fstream & f, const T& data){
f.write( (const char*)&data, sizeof( data ) );
}
};
template <class T, class U>
struct write_impl<std::pair<T, U>>{
static void apply(std::fstream & f, const std::pair<T, U>& p){
lib::write( f, p.first );
lib::write( f, p.second );
}
};
template <std::ranges::range T>
struct write_impl<T>{
static void apply(std::fstream & f, const T& data) {
const uint64_t size = std::ranges::size( data );
lib::write(f, size );
for (const auto& i : data){
lib::write( f, i );
}
}
};
}
int main() {
auto f = std::fstream( "spagetti", std::ios::out | std::ios::binary );
lib::write(f, std::pair< int, int >( 123, 777 ) );
lib::write(f, std::map< int, int >{ { 1, 99 }, { 2, 98 } } );
lib::write(f, std::pair<std::map< int, int >, int >( { { 1, 99 }, { 2, 98 } }, 777 ) );
}
[godbolt link] (https://godbolt.org/z/n7Gbosscn)
Language support
http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2022/p2547r0.pdf
However, I don't think the authors have implemented a compiler that supports it
#include <iostream>
#include <string>
struct foo
{
void operator()( const int& ) { }
void operator()( const float& ) { }
void operator()( const std::string& ) { }
};
template<typename... Types>
void bar_recursive()
{
std::cout << "Iterating over overloads";
}
template<typename Fn>
void bar( Fn&& fn )
{
// Somehow i want to take first argument types of callable operators and pass them to bar_recursive function
bar_recursive< FirstArgumentTypesOfOverloadedCallableOperators... >( );
}
int main(int argc, char *argv[])
{
// I want bar() function to call bar_recursive< int , float , std::string >()
bar( foo {} );
return 0;
}
Those overloaded function call operators don't have arbitrary number of arguments. They all will have exactly one parameter.
Is it possible to deduce those types ?
#include <functional>
#include <iostream>
template<typename T>
void test( std::function< void ( const T& ) > f )
{
T val {};
f( val );
std::cout << "std::function" << std::endl;
}
template<typename T>
void test( void(*f) ( const T& ) )
{
T val {};
f( val );
std::cout << "function pointer" << std::endl;
}
int main()
{
auto capturing_var { 0 };
// Works because implicit conversion to function pointer isn't applied when lambda is capturing
test< int >( [ capturing_var ]( const int& x ) { } );
// Doesn't work because implicitly convertible to function pointer and makes ambiguity
// I want this line to work somehow, how can i make it worked with same client code ? Is it possible ?
test< int >( []( const int& x ) { } );
// This line is finer if it works, but in this case compiler cannot deduce T and also ambiguous if it could
test( []( const int& x ) { } );
// Works because of unary + operator so conversion to function ptr and i dont need to specify T
test( +[]( const int& x ) { } );
return 0;
}
Actually I just want this code to work without changing anything in main(), but I'm not sure whether it is possible.
I can omit overloaded function which takes function pointer, then it would work but in that case I need to specify what T is.
Note: I need T to be deduced.
You can use technique described in this question with some small postprocessing.
Minimal example:
template<typename Ret, typename Arg>
Arg argument_type(Ret(*)(Arg));
template<typename Ret, typename Fn, typename Arg>
Arg argument_type(Ret(Fn::*)(Arg) const);
template<typename Fn>
auto argument_type(Fn) -> decltype(argument_type(&Fn::operator()));
template<typename Arg, typename Fn>
struct argument {
static_assert(std::is_invocable_v<Fn, Arg>);
using type = Arg;
};
template<typename Fn>
struct argument<void, Fn>{
using type = decltype(argument_type(std::declval<Fn>()));
};
template<typename T = void, typename Fn>
void test(Fn fn) {
using Arg = std::decay_t<typename argument<T, Fn>::type>;
std::cout << "Arg = " << boost::typeindex::type_id_with_cvr<Arg>().pretty_name()
<< std::endl;
}
int main() {
int capturing_var;
test<int>([capturing_var](const int& x) {}); // Arg = int
test<int>([](const int& x) {}); // Arg = int
test([](const int& x) {}); // Arg = int
test(+[](const int& x) {}); // Arg = int
test<int>([](auto x) {}); // Arg = int
}
If the argument type cannot be deduced, e.g., for a variadic lambda, it has to be provided (last example).
I have a series of functions that takes a stream as input and writes a transformation to an output stream. Right now the interface looks like this:
void phase1(std::istream& in, std::ostream& out);
void phase2(std::istream& in, std::ostream& out);
std::istream data = get_initial_data();
std::stringstream first_result;
phase1(data, first_result);
std::stringstream second_result;
phase2(first_result, second_result);
Is there an easier/more natural way to chain these calls without using Boost (sorry)?
I think you'd want to do:
(phase1 | phase2 | phase3)( in, out );
where all the glue happens for you. What more,
auto first_part = phase1|phase2;
auto second_part = phase3|phase4;
(first_part | second_part)( in, out );
should also work.
namespace stream {
template<class F=std::function<void(std::istream&, std::ostream&)>>
struct pipe {
F f;
void operator()( std::istream& in, std::ostream& out ) const {
f(in,out);
}
template<class O,
std::enable_if_t< !std::is_same<O, F>{} && std::is_convertible<O, F>{}, bool> = true
>
pipe ( pipe <O> o ):
f(std::move(o.f))
{}
pipe (F fin):
f(std::move(fin))
{}
};
template<class F>
pipe (F)->pipe <F>;
template<class First, class Second>
auto operator|( pipe <First> first, pipe <Second> second )
{
return pipe {[=](auto& in, auto& out){
std::stringstream intermediate;
first( in, intermediate );
second( intermediate, out );
}};
}
}
and now you can do:
std::istream data = get_initial_data();
( pipe {phase1} | pipe {phase2} )( data, out );
we can extend this to sources and sinks, allowing things to be glued to the input, but that often requires continuation passing style to handle lifetime issues.
You an also use pipe <> to handle any stream pipe object in a type-erased manner.
Live example.
If you want sources and sinks it looks like this:
namespace stream {
template<class Sig, class F=std::function<Sig>>
struct operation;
template<class R, class...Unused, class F>
struct operation<R(Unused...), F>
{
F f;
static_assert(
std::is_convertible< std::result_of_t< F const&(Unused...) >, R >{}
);
template<class...Args>
R operator()( Args&&...args ) const {
return static_cast<R>(f(std::forward<Args>(args)...));
}
template<class O,
std::enable_if_t< !std::is_same<O, F>{} && std::is_convertible<O, F>{}, bool> = true
>
operation ( operation<R(Unused...), O> o ):
f(std::move(o.f))
{}
operation (F fin):
f(std::move(fin))
{}
};
template<class F=std::function<void(std::istream&, std::ostream&)>>
struct pipe:operation<void(std::istream&, std::ostream&), F> {
using operation<void(std::istream&, std::ostream&), F>::operation;
};
template<class F>
pipe (F)->pipe <F>;
template<class First, class Second>
auto operator|( pipe <First> first, pipe <Second> second )
{
return pipe {[=](auto& in, auto& out){
std::stringstream intermediate;
first( in, intermediate );
second( intermediate, out );
}};
}
template<class F=std::function< void(std::function< void(std::ostream&)>) >>
struct source:operation<void(std::function< void(std::istream&)>), F> {
using operation<void(std::function< void(std::istream&)>), F>::operation;
};
template<class F>
source(F)->source<F>;
template<class F=std::function< void(std::function< void(std::ostream&)>) >>
struct sink:operation<void(std::function< void(std::ostream&)>), F> {
using operation<void(std::function< void(std::ostream&)>), F>::operation;
};
template<class F>
sink(F)->sink<F>;
template<class First, class Second>
auto operator|( source<First> src, pipe<Second> p ) {
return source{[=]( auto&& f ){
src([&](auto&& in){
std::stringstream ss;
p( in, ss );
f( ss );
});
}};
}
template<class First, class Second>
auto operator|( pipe<First> p, sink<Second> snk ) {
return sink{[=]( auto&& f ){
snk([&](auto&& out){
std::stringstream ss;
f(ss);
p(ss, out);
});
}};
}
void copy_f( std::istream& is, std::ostream& os ) {
char c;
while (is.get(c)) {
os << c;
}
}
inline pipe copy{copy_f};
template<class First, class Second>
void operator|( source<First> src, sink<Second> snk ) {
src([&](auto&& in){
snk([&](auto&& out){
copy( in, out );
});
});
}
}
you can then do:
using namespace stream;
auto src = source{[](auto&& f){
std::stringstream ss;
ss << "Hello world\n";
f(ss);
}};
auto snk = sink{[](auto&& f){
f(std::cout);
}};
src|copy|copy|copy|snk;
Live example
A source is a function object that in turn takes a function object, that it passes an istream& to.
A sink is a function object that in turn takes a function object, that it passes a ostream& to.
This double-function syntax deals with annoying lifetime issues, and lets you do cleanup before/after the client stream-user does stuff with the stream.
And a slightly more insane version that supports direct piping to/from streams is here.
I am writing template classes for de/serialization of stl containers.
Let's say I want to serialize a set<int>. I have a base serialisation class and the following template:
template<typename T>class serialiser;
template<>class serialiser<int>:public serialisation<int>{
public:
void serialise ( int t );
};
To serialize the set, I have:
template<typename T>class Container_serialiser:public serialisation<T>{
public:
void serialise ( T t );
private:
/* Notice that I must declare a serialiser.
*/
serialiser<typename T :: value_type>value_serialiser;
};
template<typename T>void Container_serialiser<T>::serialise ( T t ){
for(typename T :: const_iterator t_iterator = t . begin ( );t_iterator != t . end ( );++ t_iterator){
value_serialiser . serialise ( * t_iterator );
}
}
This only works for containers with elements that do not themselves comprise containers.
The Problem
What if I want to serialize a map<int,set<int> >?
Since each element's type is pair<int,set<int> >, I need the following class:
template<>class serialiser<pair<int,set<int> > >:public serialisation<pair<int,set<int> > >{
public:
void serialise ( const pair<int,set<int> >t );
private:
serialiser<int>t_first_serialiser;
/* Notice that I must declare a Container_serialiser.
*/
Container_serialiser<set<int> >t_second_serialiser;
};
However, Container_serialiser requires serialiser<T> to be a complete type. Thus, it cannot be defined until after the above class. Likewise, the above class requires Container_serialiser<T> to be a complete type and cannot be defined before it.
The compiler gives the following error:
prog.cpp: In instantiation of 'class Container_serialiser<std::map<int, std::set<int> > >':
prog.cpp:73:44: required from here
prog.cpp:40:39: error: 'Container_serialiser<T>::value_serialiser' has incomplete type
serialiser<typename T :: value_type>value_serialiser;
^
prog.cpp:25:27: note: declaration of 'class serialiser<std::pair<const int, std::set<int> > >'
template<typename T>class serialiser;
Problem Code
#include <iostream>
#include <map>
#include <set>
#include <utility>
using std :: cout;
using std :: map;
using std :: set;
using std :: make_pair;
using std :: pair;
template<typename T>class serialisation{
public:
virtual ~serialisation ( );
protected:
virtual void serialise ( const T t ) = 0;
};
template<typename T>serialisation<T> :: ~serialisation ( ){
}
template<typename T>class serialiser;
template<>class serialiser<int>:public serialisation<int>{
public:
void serialise ( const int t );
};
void serialiser<int>:: serialise ( const int t ){
cout << t << '\n';
}
template<typename T>class Container_serialiser:public serialisation<T>{
public:
void serialise ( const T t );
private:
serialiser<typename T :: value_type>value_serialiser;
};
template<typename T>void Container_serialiser<T>:: serialise ( const T t ){
for(typename T :: const_iterator t_iterator = t . begin ( );t_iterator != t . end ( );++ t_iterator){
value_serialiser . serialise ( * t_iterator );
}
}
template<>class serialiser<pair<int,set<int> > >:public serialisation<pair<int,set<int> > >{
public:
void serialise ( const pair<int,set<int> >t );
private:
serialiser<int>t_first_serialiser;
Container_serialiser<set<int> >t_second_serialiser;
};
void serialiser<pair<int,set<int> > >:: serialise ( const pair<int,set<int> >t ){
t_first_serialiser . serialise ( t . first );
t_second_serialiser . serialise ( t . second );
}
int main ( ){
set<int>t;
t . insert ( 2 );
t . insert ( 3 );
Container_serialiser<set<int> >t_serialiser;
t_serialiser . serialise ( t );
map<int,set<int> >u;
u . insert ( make_pair ( 5,t ) );
u . insert ( make_pair ( 7,t ) );
Container_serialiser<map<int,set<int> > >u_serialiser;
u_serialiser . serialise ( u );
}
Ideone
A Hack
I can make this work by writing four templates:
serialiser0<T> for all T that are not containers and do not comprise containers
Container_serialiser0<T> for all T whose value_type is the argument of a serialiser0<T>
serialiser1<T> for all T that are not containers but comprise containers of elements that themselves do not comprise containers
Container_serialiser1<T> for all T whose value_type is the argument of a serialiser1<T>
This design is repetitive, confusing, and has to be extended manually for each new level of containers. Is there a better pattern?
Unfortunately, due to the project I am working on, I must avoid external dependencies such as boost.serialization even at the cost of taking a performance hit and write for the C++98 standard.
Hack Code
#include <iostream>
#include <map>
#include <set>
#include <utility>
using std :: cout;
using std :: map;
using std :: set;
using std :: make_pair;
using std :: pair;
template<typename T>class serialisation{
public:
virtual ~serialisation ( );
protected:
virtual void serialise ( const T t ) = 0;
};
template<typename T>serialisation<T> :: ~serialisation ( ){
}
template<typename T>class serialiser0;
template<>class serialiser0<int>:public serialisation<int>{
public:
void serialise ( const int t );
};
void serialiser0<int>:: serialise ( const int t ){
cout << t << '\n';
}
template<typename T>class Container_serialiser0:public serialisation<T>{
public:
void serialise ( const T t );
private:
serialiser0<typename T :: value_type>value_serialiser;
};
template<typename T>void Container_serialiser0<T>:: serialise ( const T t ){
for(typename T :: const_iterator t_iterator = t . begin ( );t_iterator != t . end ( );++ t_iterator){
value_serialiser . serialise ( * t_iterator );
}
}
template<typename T>class serialiser1;
template<>class serialiser1<pair<const int,set<int> > >:public serialisation<pair<int,set<int> > >{
public:
void serialise ( const pair<int,set<int> >t );
private:
serialiser0<int>t_first_serialiser;
Container_serialiser0<set<int> >t_second_serialiser;
};
void serialiser1<pair<const int,set<int> > >:: serialise ( const pair<int,set<int> >t ){
t_first_serialiser . serialise ( t . first );
t_second_serialiser . serialise ( t . second );
}
/* This is the same as Container_serialiser0!
*/
template<typename T>class Container_serialiser1:public serialisation<T>{
public:
void serialise ( const T t );
private:
serialiser1<typename T :: value_type>value_serialiser;
};
template<typename T>void Container_serialiser1<T>:: serialise ( const T t ){
for(typename T :: const_iterator t_iterator = t . begin ( );t_iterator != t . end ( );++ t_iterator){
value_serialiser . serialise ( * t_iterator );
}
}
int main ( ){
set<int>t;
t . insert ( 2 );
t . insert ( 3 );
Container_serialiser0<set<int> >t_serialiser;
t_serialiser . serialise ( t );
map<int,set<int> >u;
u . insert ( make_pair ( 5,t ) );
u . insert ( make_pair ( 7,t ) );
Container_serialiser1<map<int,set<int> > >u_serialiser;
u_serialiser . serialise ( u );
}
Ideone
(I must admit that I had forgotten about this question.)
Thanks to #michael-gopshtein's comment, I eventually stumbled upon partial template specialization, which solves this problem quite well. Note that I must use both template <typename SerialisedType> class Serialiser and template <typename SerialisedType> inline void serialise(const SerialisedType &x). The function itself cannot be partially-specialized, but most specializations of Serialiser::serialise depend upon a general call to ::serialise.
To serialize a std::map<int, std::set<int>>, I would write the following:
#include <map>
#include <set>
#include <utility>
// For the sake of having a MWE, I'll just write int-s to stdout.
#include <iostream>
template <typename SerialisedType> class Serialiser;
template <typename SerialisedType>
inline void serialise(const SerialisedType &x) {
Serialiser<SerialisedType>::serialise(x);
}
template <> class Serialiser<int> {
public:
inline static void serialise(const int &x);
};
template <typename Key> class Serialiser<std::set<Key>> {
public:
inline static void serialise(const std::set<Key> &x);
};
template <typename T1, typename T2> class Serialiser<std::pair<T1, T2>> {
public:
inline static void serialise(const std::pair<T1, T2> &x);
};
template <typename Key, typename T>
class Serialiser<std::map<Key, T>> {
public:
inline static void serialise(const std::map<Key, T> &x);
};
void Serialiser<int>::serialise(const int &x) {
std::cout << x << "\n";
}
template <typename Key>
void Serialiser<std::set<Key>>::serialise(const std::set<Key> &x) {
{
const typename std::set<Key>::const_iterator x_cend = x.cend();
for (typename std::set<Key>::const_iterator x_iterator = x.cbegin();
x_iterator != x_cend; ++x_iterator)
::serialise(*x_iterator);
}
}
template <typename T1, typename T2>
void Serialiser<std::pair<T1, T2>>::serialise(const std::pair<T1, T2> &x) {
::serialise(x.first);
::serialise(x.second);
}
template <typename Key, typename T>
void Serialiser<std::map<Key, T>>::serialise(const std::map<Key, T> &x) {
{
const typename std::map<Key, T>::const_iterator x_cend = x.cend();
for (typename std::map<Key, T>::const_iterator x_iterator = x.cbegin();
x_iterator != x_cend; ++x_iterator)
::serialise(*x_iterator);
}
}
int main() {
std::map<int, std::set<int>> u {
{5, {2, 3}},
{7, {2, 3}}
};
serialise(u);
}
Output:
5
2
3
7
2
3