Related
I am trying to create a data structure for arrays of dynamic-sized arrays. Multiple choices are possible, the simplest one being std::vector<std::vector<T>>. However it is often not efficient and we would like to compress the data of all the inner vectors into one big vector, and have a vector of offsets to tell where each element begins.
Example:
// encoding of : | 4.,5.,1. | 7.,8.,9.,2 |
std::vector<double> v = {4.,5.,1., 7.,8.,9.,2};
std::vector<int> offsets = {0 , 3 , 7};
Let's encapsulate it ! Consider the following data structure:
(note: the code is neither complete, general or precise, at this point this is just to give an idea of what is going on):
class vblock_vector {
private:
std::vector<double> v;
std::vector<int> offsets;
public:
using iterator = vblock_iterator;
auto begin() -> iterator {
return {v.data(),offsets.data()};
}
auto end() -> iterator {
return {v.data(),offsets.data()+offsets.size()};
}
};
An basic implementation of the iterator type is the following:
struct vblock_iterator {
private:
double* ptr;
int* offsets_ptr;
public:
using reference = span_ref<double>; // see notes (0) and (1)
// using value_type = ???; // See below
auto operator++() {
++offsets_ptr;
return *this;
}
auto operator*() const {
return span_ref<double,int>(ptr+offsets_ptr[0],ptr+offsets_ptr[1]);
}
auto operator<=>(const vblock_iterator&) const = default;
// ... other iterator interface stuff that is trivial
};
This iterator works with e.g. std::copy. (4)
Now let's say that I want to replace my old std::copy calls with std::ranges::copy. For that, vblock_iterator needs to satisfy the std::input_iterator concept. In order to do that, vblock_iterator needs to have an associated value_type (required by the intermediate std::indirectly_readable concept).
An obvious choice would be using value_type = std::vector<double>(2), but I surely don't want to give std::ranges::copy the freedom to use this type at its discretion in its implementation: it would be inefficient.
My question is the following : why does std::input_iterator<In> requires In to have a value_type? At least for copying it is not needed (the fact that I can use std::copy and that it does the right thing proves it). Of course, one can say : "define value_type to be anything, it won't be used by std::range::copy implementations anyway", but then why require it?
I am currently under the impression that value_type is mandatory for e.g. std::swappable, but not for std::input_iterator (nor even std::random_access_iterator dare I say). But the standard committee decided otherwise: what is the reason behind this choice? (3)
Notes:
(0) span_ref is just like a std::span with reference semantics (its operator= is "assign-through" and not "rebind to new array").
(1) In reality, the reference type needs to be a tad more complex to account for offsets, but it is not the subject here. Suffice to say, that it is possible to have an efficient reference type for this structure.
(2) And I think this is the only reasonable choice. At least a container is needed (vector, deque...). E.g. a std::span won't do because if we bother to save the value pointed to by the iterator, it is because we will modify the original memory, and std::span won't help us with that.
(3) In the presentation of the std::indirectly_readable concept (then called Readable), Eric Niebler goes into some detail of why we need value_type to be related in some form to reference to work well with proxy references, but I still don't see why we would would even need value_type for algorithms that don't need to swap elements (or store them somewhere). Yes, there is mathematically a value_type for vblock_iterator, but why require it if it is not meant to be used? (similarly, there is also mathematical operator+= for forward ranges : but since it is inefficient, it is simply not required).
(4) And other algorithms: std::move, std::find, std::find_if, std::any_of, std::partition_point, std::lower_bound, std::unique... So I think that there is something more fundamental going on than: "we are just lucky with std::copy".
std::copy requires a LegacyInputIterator for its iterator types. It does not check this requirement. If you fail to provide a LegacyInputIterator, your program is ill-formed, no diagnostic required.
A LegacyInputIterator requires that std::iterator_traits<X>::value_type exists because it subsumes LegacyIterator.
So your program was ill-formed once you passed it to std::copy. The behavior of your ill-formed program is not determined by the C++ standard in any way; the compiler can legally provide you a program that emails your browser history to your great aunt Eustice and be standard compliant. Or it could do something that happens to align with what you think the program "should" do. Or it could fail to compile.
The std::ranges algorithms have slightly different requirements. These requirements are far more likely to be checked by concepts than the old style algorithms are, telling the user with a compile time error.
You are running into such a case.
To be even more clear, you cannot rely on the implementation of std code to enforce the standard.
These types are required partly to make it easier to talk about the types in question and what operations on them mean, semantically.
Beyond the simple requirements like std::iterator_traits<X>::value_type exist, there are semantic requirements on what *it does, what x = *it++ does, etc. Most of those requirements cannot be checked by the compiler (due to Rice's theorem, they cannot be checked in theory); but the algorithms in the std namespace rely on those semantic meanings being correct for any iterator passed in.
Because the compiler can assume the semantic meanings are correct, the algorithms can be cleaner, simpler and faster than if they had to check them. And it means that multiple different compiler vendors can write different std algorithm implementations, improving the algorithm over each other, and there is an objective standard to argue against.
For a LegacyInputIterator and types value_type and reference from std::iterator_traits<X>, we must have:
value_type v = *it;
is a valid expression, *it must return reference, and
*it++
must return a type convertible to value_type.
Not every algorithm need use every property of every iterator it requires that iterator to have. The goal here is to have semantically meaningful categories that do not demand too much in the way of overhead.
Requiring that an iterator over stuff actually have a type it is an iterator over is not a large overhead. And it makes talking about that the iterator is insanely easier.
You could refactor it and remove that concept, or cut the concept up into smaller pieces so that the value_type is only required in the narrow cases where it is required, but that would make the concepts harder to write about and harder to understand.
This is a C++ / D cross-over question. The D programming language has ranges that -in contrast to C++ libraries such as Boost.Range- are not based on iterator pairs. The official C++ Ranges Study Group seems to have been bogged down in nailing a technical specification.
Question: does the current C++11 or the upcoming C++14 Standard have any obstacles that prevent adopting D ranges -as well as a suitably rangefied version of <algorithm>- wholesale?
I don't know D or its ranges well enough, but they seem lazy and composable as well as capable of providing a superset of the STL's algorithms. Given their claim to success for D, it would seem very nice to have as a library for C++. I wonder how essential D's unique features (e.g. string mixins, uniform function call syntax) were for implementing its ranges, and whether C++ could mimic that without too much effort (e.g. C++14 constexpr seems quite similar to D compile-time function evaluation)
Note: I am seeking technical answers, not opinions whether D ranges are the right design to have as a C++ library.
I don't think there is any inherent technical limitation in C++ which would make it impossible to define a system of D-style ranges and corresponding algorithms in C++. The biggest language level problem would be that C++ range-based for-loops require that begin() and end() can be used on the ranges but assuming we would go to the length of defining a library using D-style ranges, extending range-based for-loops to deal with them seems a marginal change.
The main technical problem I have encountered when experimenting with algorithms on D-style ranges in C++ was that I couldn't make the algorithms as fast as my iterator (actually, cursor) based implementations. Of course, this could just be my algorithm implementations but I haven't seen anybody providing a reasonable set of D-style range based algorithms in C++ which I could profile against. Performance is important and the C++ standard library shall provide, at least, weakly efficient implementations of algorithms (a generic implementation of an algorithm is called weakly efficient if it is at least as fast when applied to a data structure as a custom implementation of the same algorithm using the same data structure using the same programming language). I wasn't able to create weakly efficient algorithms based on D-style ranges and my objective are actually strongly efficient algorithms (similar to weakly efficient but allowing any programming language and only assuming the same underlying hardware).
When experimenting with D-style range based algorithms I found the algorithms a lot harder to implement than iterator-based algorithms and found it necessary to deal with kludges to work around some of their limitations. Of course, not everything in the current way algorithms are specified in C++ is perfect either. A rough outline of how I want to change the algorithms and the abstractions they work with is on may STL 2.0 page. This page doesn't really deal much with ranges, however, as this is a related but somewhat different topic. I would rather envision iterator (well, really cursor) based ranges than D-style ranges but the question wasn't about that.
One technical problem all range abstractions in C++ do face is having to deal with temporary objects in a reasonable way. For example, consider this expression:
auto result = ranges::unique(ranges::sort(std::vector<int>{ read_integers() }));
In dependent of whether ranges::sort() or ranges::unique() are lazy or not, the representation of the temporary range needs to be dealt with. Merely providing a view of the source range isn't an option for either of these algorithms because the temporary object will go away at the end of the expression. One possibility could be to move the range if it comes in as r-value, requiring different result for both ranges::sort() and ranges::unique() to distinguish the cases of the actual argument being either a temporary object or an object kept alive independently. D doesn't have this particular problem because it is garbage collected and the source range would, thus, be kept alive in either case.
The above example also shows one of the problems with possibly lazy evaluated algorithm: since any type, including types which can't be spelled out otherwise, can be deduced by auto variables or templated functions, there is nothing forcing the lazy evaluation at the end of an expression. Thus, the results from the expression templates can be obtained and the algorithm isn't really executed. That is, if an l-value is passed to an algorithm, it needs to be made sure that the expression is actually evaluated to obtain the actual effect. For example, any sort() algorithm mutating the entire sequence clearly does the mutation in-place (if you want a version doesn't do it in-place just copy the container and apply the in-place version; if you only have a non-in-place version you can't avoid the extra sequence which may be an immediate problem, e.g., for gigantic sequences). Assuming it is lazy in some way the l-value access to the original sequence provides a peak into the current status which is almost certainly a bad thing. This may imply that lazy evaluation of mutating algorithms isn't such a great idea anyway.
In any case, there are some aspects of C++ which make it impossible to immediately adopt the D-sytle ranges although the same considerations also apply to other range abstractions. I'd think these considerations are, thus, somewhat out of scope for the question, too. Also, the obvious "solution" to the first of the problems (add garbage collection) is unlikely to happen. I don't know if there is a solution to the second problem in D. There may emerge a solution to the second problem (tentatively dubbed operator auto) but I'm not aware of a concrete proposal or how such a feature would actually look like.
BTW, the Ranges Study Group isn't really bogged down by any technical details. So far, we merely tried to find out what problems we are actually trying to solve and to scope out, to some extend, the solution space. Also, groups generally don't get any work done, at all! The actual work is always done by individuals, often by very few individuals. Since a major part of the work is actually designing a set of abstractions I would expect that the foundations of any results of the Ranges Study Group is done by 1 to 3 individuals who have some vision of what is needed and how it should look like.
My C++11 knowledge is much more limited than I'd like it to be, so there may be newer features which improve things that I'm not aware of yet, but there are three areas that I can think of at the moment which are at least problematic: template constraints, static if, and type introspection.
In D, a range-based function will usually have a template constraint on it indicating which type of ranges it accepts (e.g. forward range vs random-access range). For instance, here's a simplified signature for std.algorithm.sort:
auto sort(alias less = "a < b", Range)(Range r)
if(isRandomAccessRange!Range &&
hasSlicing!Range &&
hasLength!Range)
{...}
It checks that the type being passed in is a random-access range, that it can be sliced, and that it has a length property. Any type which does not satisfy those requirements will not compile with sort, and when the template constraint fails, it makes it clear to the programmer why their type won't work with sort (rather than just giving a nasty compiler error from in the middle of the templated function when it fails to compile with the given type).
Now, while that may just seem like a usability improvement over just giving a compilation error when sort fails to compile because the type doesn't have the right operations, it actually has a large impact on function overloading as well as type introspection. For instance, here are two of std.algorithm.find's overloads:
R find(alias pred = "a == b", R, E)(R haystack, E needle)
if(isInputRange!R &&
is(typeof(binaryFun!pred(haystack.front, needle)) : bool))
{...}
R1 find(alias pred = "a == b", R1, R2)(R1 haystack, R2 needle)
if(isForwardRange!R1 && isForwardRange!R2 &&
is(typeof(binaryFun!pred(haystack.front, needle.front)) : bool) &&
!isRandomAccessRange!R1)
{...}
The first one accepts a needle which is only a single element, whereas the second accepts a needle which is a forward range. The two are able to have different parameter types based purely on the template constraints and can have drastically different code internally. Without something like template constraints, you can't have templated functions which are overloaded on attributes of their arguments (as opposed to being overloaded on the specific types themselves), which makes it much harder (if not impossible) to have different implementations based on the genre of range being used (e.g. input range vs forward range) or other attributes of the types being used. Some work has been being done in this area in C++ with concepts and similar ideas, but AFAIK, C++ is still seriously lacking in the features necessary to overload templates (be they templated functions or templated types) based on the attributes of their argument types rather than specializing on specific argument types (as occurs with template specialization).
A related feature would be static if. It's the same as if, except that its condition is evaluated at compile time, and whether it's true or false will actually determine which branch is compiled in as opposed to which branch is run. It allows you to branch code based on conditions known at compile time. e.g.
static if(isDynamicArray!T)
{}
else
{}
or
static if(isRandomAccessRange!Range)
{}
else static if(isBidirectionalRange!Range)
{}
else static if(isForwardRange!Range)
{}
else static if(isInputRange!Range)
{}
else
static assert(0, Range.stringof ~ " is not a valid range!");
static if can to some extent obviate the need for template constraints, as you can essentially put the overloads for a templated function within a single function. e.g.
R find(alias pred = "a == b", R, E)(R haystack, E needle)
{
static if(isInputRange!R &&
is(typeof(binaryFun!pred(haystack.front, needle)) : bool))
{...}
else static if(isForwardRange!R1 && isForwardRange!R2 &&
is(typeof(binaryFun!pred(haystack.front, needle.front)) : bool) &&
!isRandomAccessRange!R1)
{...}
}
but that still results in nastier errors when compilation fails and actually makes it so that you can't overload the template (at least with D's implementation), because overloading is determined before the template is instantiated. So, you can use static if to specialize pieces of a template implementation, but it doesn't quite get you enough of what template constraints get you to not need template constraints (or something similar).
Rather, static if is excellent for doing stuff like specializing only a piece of your function's implementation or for making it so that a range type can properly inherit the attributes of the range type that it's wrapping. For instance, if you call std.algorithm.map on an array of integers, the resultant range can have slicing (because the source range does), whereas if you called map on a range which didn't have slicing (e.g. the ranges returned by std.algorithm.filter can't have slicing), then the resultant ranges won't have slicing. In order to do that, map uses static if to compile in opSlice only when the source range supports it. Currently, map 's code that does this looks like
static if (hasSlicing!R)
{
static if (is(typeof(_input[ulong.max .. ulong.max])))
private alias opSlice_t = ulong;
else
private alias opSlice_t = uint;
static if (hasLength!R)
{
auto opSlice(opSlice_t low, opSlice_t high)
{
return typeof(this)(_input[low .. high]);
}
}
else static if (is(typeof(_input[opSlice_t.max .. $])))
{
struct DollarToken{}
enum opDollar = DollarToken.init;
auto opSlice(opSlice_t low, DollarToken)
{
return typeof(this)(_input[low .. $]);
}
auto opSlice(opSlice_t low, opSlice_t high)
{
return this[low .. $].take(high - low);
}
}
}
This is code in the type definition of map's return type, and whether that code is compiled in or not depends entirely on the results of the static ifs, none of which could be replaced with template specializations based on specific types without having to write a new specialized template for map for every new type that you use with it (which obviously isn't tenable). In order to compile in code based on attributes of types rather than with specific types, you really need something like static if (which C++ does not currently have).
The third major item which C++ is lacking (and which I've more or less touched on throughout) is type introspection. The fact that you can do something like is(typeof(binaryFun!pred(haystack.front, needle)) : bool) or isForwardRange!Range is crucial. Without the ability to check whether a particular type has a particular set of attributes or that a particular piece of code compiles, you can't even write the conditions which template constraints and static if use. For instance, std.range.isInputRange looks something like this
template isInputRange(R)
{
enum bool isInputRange = is(typeof(
{
R r = void; // can define a range object
if (r.empty) {} // can test for empty
r.popFront(); // can invoke popFront()
auto h = r.front; // can get the front of the range
}));
}
It checks that a particular piece of code compiles for the given type. If it does, then that type can be used as an input range. If it doesn't, then it can't. AFAIK, it's impossible to do anything even vaguely like this in C++. But to sanely implement ranges, you really need to be able to do stuff like have isInputRange or test whether a particular type compiles with sort - is(typeof(sort(myRange))). Without that, you can't specialize implementations based on what types of operations a particular range supports, you can't properly forward the attributes of a range when wrapping it (and range functions wrap their arguments in new ranges all the time), and you can't even properly protect your function against being compiled with types which won't work with it. And, of course, the results of static if and template constraints also affect the type introspection (as they affect what will and won't compile), so the three features are very much interconnected.
Really, the main reasons that ranges don't work very well in C++ are the some reasons that metaprogramming in C++ is primitive in comparison to metaprogramming in D. AFAIK, there's no reason that these features (or similar ones) couldn't be added to C++ and fix the problem, but until C++ has metaprogramming capabilities similar to those of D, ranges in C++ are going to be seriously impaired.
Other features such as mixins and Uniform Function Call Syntax would also help, but they're nowhere near as fundamental. Mixins would help primarily with reducing code duplication, and UFCS helps primarily with making it so that generic code can just call all functions as if they were member functions so that if a type happens to define a particular function (e.g. find) then that would be used instead of the more general, free function version (and the code still works if no such member function is declared, because then the free function is used). UFCS is not fundamentally required, and you could even go the opposite direction and favor free functions for everything (like C++11 did with begin and end), though to do that well, it essentially requires that the free functions be able to test for the existence of the member function and then call the member function internally rather than using their own implementations. So, again you need type introspection along with static if and/or template constraints.
As much as I love ranges, at this point, I've pretty much given up on attempting to do anything with them in C++, because the features to make them sane just aren't there. But if other folks can figure out how to do it, all the more power to them. Regardless of ranges though, I'd love to see C++ gain features such as template constraints, static if, and type introspection, because without them, metaprogramming is way less pleasant, to the point that while I do it all the time in D, I almost never do it in C++.
Does the standard require that some_container<T>::value_type be T?
I am asking because I am considering different approaches to implementing an STL-compliant 2d dynamic array. One of them is to have 2Darray<T>::value_type be 2Darray_row<T> or something like that, where the array would be iterated as a collection of rows (a little simplified. My actual implementation allows iteration in 3 directions)
The container requirements are a bit funky in the sense that they are actually not used by any generic algorithm. In that sense, it doesn't really matter much.
That said, the requirements are on the interface for containers not on how the container is actually instantiated. Even non-template classes can conform to the various requirements and, in fact, do. The requirement is that value_type is present; what it is defined to depends entirely on the container implementation.
Table 96 in ยง23.2.1 in the standard (c++11) requires a container class X containing objects of type T to return T for X::value_type.
So, if your some_container stores objects of type T, then value_type has to be T.
Either have a nested container (so colArray<rowArray<T> >) or have a single wrapping (2dArray<T>), but don't try to mix them. The nested approach allows you to use STL all the way down (vector<vector<T> >), but can be confusing and doesn't allow you column iterators etc, which you seem to want.
This SO answer addresses using ublas, and another suggests using Boost multi-arrays.
Generally, go for the STL or Boost option if you can. You are unlikely to write something as well by yourself.
std::set_union and its kin take two pairs of iterators for the sets to be operated on. That's great in that it's the most flexible thing to do. However they very easily could have made an additional convenience functions which would be more elegant for 80% of typical uses.
For instance:
template<typename ContainerType, typename OutputIterator>
OutputIterator set_union( const ContainerType & container1,
const ContainerType & container2,
OutputIterator & result )
{
return std::set_union( container1.begin(), container1.end(),
container2.begin(), container2.end(),
result );
}
would turn:
std::set_union( mathStudents.begin(), mathStudents.end(),
physicsStudents.begin(), physicsStudents.end(),
students.begin() );
into:
std::set_union( mathStudents, physicsStudents, students.begin() );
So:
Are there convenience functions like this hiding somewhere that I just haven't found?
If not, can anyone thing of a reason why it would be left out of STL?
Is there perhaps a more full featured set library in boost? (I can't find one)
I can of course always put my implementations in a utility library somewhere, but it's hard to keep such things organized so that they're used across all projects, but not conglomerated improperly.
Are there convenience functions like this hiding somewhere that I just haven't found?
Not in the standard library.
If not, can anyone thing of a reason why it would be left out of STL?
The general idea with algorithms is that they work with iterators, not containers. Containers can be modified, altered, and poked at; iterators cannot. Therefore, you know that, after executing an algorithm, it has not altered the container itself, only potentially the container's contents.
Is there perhaps a more full featured set library in boost?
Boost.Range does this. Granted, Boost.Range does more than this. It's algorithms don't take "containers"; they take iterator ranges, which STL containers happen to satisfy the conditions for. They also have lazy evaluation, which can be nice for performance.
One reason for working with iterators is of course that it is more general and works on ranges that are not containers, or just a part of a container.
Another reason is that the signatures would be mixed up. Many algorithms, like std::sort have more than one signature already:
sort(Begin, End);
sort(Begin, End, Compare);
Where the second one is for using a custom Compare when sorting on other than standard less-than.
If we now add a set of sort for containers, we get these new functions
sort(Container);
sort(Container, Compare);
Now we have the two signatures sort(Begin, End) and sort(Container, Compare) which both take two template parameters, and the compiler will have problems resolving the call.
If we change the name of one of the functions to resolve this (sort_range, sort_container?) it will not be as convenient anymore.
I agree, STL should take containers instead of iterators-pairs for the following reasons;
Simpler code
Algorithms could be overloaded for specified containers, ie, could use the map::find algorithm instead of std::find -> More general code
A subrange could easily be wrapped into a container, as is done in boost::range
#Bo Persson has pointed to a problem with ambiguity, and I think that's quite valid.
I think there's a historical reason that probably prevented that from ever really even being considered though.
The STL was introduced into C++ relatively late in the standardization process. Shortly after it was accepted, the committee voted against even considering any more new features for addition into C++98 (maybe even at the same meeting). By the time most people had wrapped their head around the existing STL to the point of recognizing how much convenience you could get from something like ranges instead of individual iterators, it was too late to even be considered.
Even if the committee was still considering new features, and somebody had written a proposals to allow passing containers instead of discrete iterators, and had dealt acceptably with the potential for ambiguity, I suspect the proposal would have been rejected. Many (especially the C-oriented people) saw the STL as a huge addition to the standard library anyway. I'm reasonably certain quite a few people would have considered it completely unacceptable to add (lots) more functions/overloads/specializations just to allowing passing one parameter in place of two.
Using the begin & end elements for iteration allows one to use non-container types as inputs. For example:
ContainerType students[10];
vector<ContainerType> physicsStudents;
std::set_union(physicsStudents.begin(), physicsStudents.end(),
&students[0], &students[10],
physicsStudents.begin());
Since they are such simple implementations, I think it makes sense not to add them to the std library and allow authors to add their own. Especially given that they are templates, thus potentially increasing the lib size of the code and adding convenience functions across std would lead to code bloat.
Why would you need to overload the [] operator? I have never come across a practical scenario where this was necessary. Can somebody tell me a practical use case for this.
Err.. std::vector<t>, std::basic_string<t>, std::map<k, v>, and std::deque<t> ?
I used this for a class representing a registry key, where operator[] returned an object representing a registry value with the string between []s.
See also, the Spirit Parser Framework, which uses [] for semantic actions.
Any indexable container can usefully define operator[] to become usable in any template that uses []-syntax indexing.
You don't need that syntax sugar if you're not doing generic programming -- it may look nice, but, cosmetics apart, you could always define specific named methods such as getAt, setAt, and the like, with similar and simpler-to-code functionality.
However, generic programming is at the core of modern C++... and it bears an eerie resemblance to "compile-time, type-safe duck typing" (I'm biased towards such peculiar terminology, of course, having had a part in shaping it -- cfr wikipedia;-).
Just as you should try to use, e.g., prefix-* to mean "dereferencing" for all kinds of iterators and other pointer-like types (so they can be duck-typingly substituted for pointers in a template!), so similarly you should strive to define operator[] in container types where it makes sense, just so they can be duck-typingly substituted for arrays in appropriate templates.
It is useful if you implement almost any type of container that provides random access (or at least some form of keyed access) to its elements (e.g., consider std::vector).
If you write a class that inherits from another class that implements the [] operator, you might want to overwrite the [] operator, such as std::vector or std::string. If you don't do this, your class may not work as the user expects, as your class will implicitly inherit the parent's implementation of [].
Well, several STL containers give some examples - vector<> overloads it to make it act like an array. map<> for example provides the operator[] overload to provide an 'associative array'.
While it is not strictly necessary, it is incredibly useful in making user-defined containers or strings behave like builtin arrays or C strings. This cuts down on verbosity a lot (for example, in Java, you would have to use x.getElementAt(i) while in C++ you can use x[i]; similarly, in Java you need x.compareTo(y)<0, while in C++ you can achieve the same thing using x < y). It is syntactic sugar... but it is very, very tasty.