This question already has an answer here:
How to correcly check whether a pointer belongs within an allocated block?
(1 answer)
Closed 1 year ago.
I am working with some code that checks if std::vector contains a given element in constant time by comparing its address to those describing the extent of the vector's data. However I suspect that, although it works, it relies on undefined behaviour. If the element is not contained by the vector then the pointer comparisons are not permitted.
bool contains(const std::vector<T>& v, const T& a) {
return (v.data() <= &a) && (&a < v.data() + v.size());
}
Am I right in believing it is undefined behaviour? If so, is there any way to do the same thing without drastically changing the time complexity of the code?
You can use std::less
A specialization of std::less for any pointer type yields the implementation-defined strict total order, even if the built-in < operator does not.
Update:
The standard doesn't guarantee that this will actually work for contains though. If you have say two vectors a and b, the total order is permitted to be &a[0], &b[0], &a[1], &b[1], &a[2], &b[2], ..., i.e., with the elements interleaved.
As pointed out in the comments, the standard only guarantees that std::less yields the implementation-defined strict total order, which is is consistent with the partial order imposed by the builtin operators. However, the standard doesn't guarantee the order of pointers pointing to different objects or arrays. Releated: https://devblogs.microsoft.com/oldnewthing/20170927-00/?p=97095
One interesting thing is that there's a similar usage in Herb Sutter's gcpp library(link). There's a comment saying that it is portable, the library is experimental though.
// Return whether p points into this page's storage and is allocated.
//
inline
bool gpage::contains(gsl::not_null<const byte*> p) const noexcept {
// Use std::less<> to compare (possibly unrelated) pointers portably
auto const cmp = std::less<>{};
auto const ext = extent();
return !cmp(p, ext.data()) && cmp(p, ext.data() + ext.size());
}
Yes, the comparisons as written are not permitted if the reference doesn't reference something that is already an element of the vector.
You can make the behavior defined by casting all pointers to uintptr_t and comparing those. This will work on all architectures with continuous memory (i.e. possibly not old 16-bit x86), although I don't know if the specific semantics are guaranteed.
As a side note, I would always interpret the name contains to be about the value, and thus be very surprised if the semantics are anything other than std::find(v.begin(), v.end(), a) != v.end(). Consider using a more expressive name.
Related
Item 18 of Scott Meyers's book Effective STL: 50 Specific Ways to Improve Your Use of the Standard Template Library says to avoid vector <bool> as it's not an STL container and it doesn't really hold bools.
The following code:
vector <bool> v;
bool *pb =&v[0];
will not compile, violating a requirement of STL containers.
Error:
cannot convert 'std::vector<bool>::reference* {aka std::_Bit_reference*}' to 'bool*' in initialization
vector<T>::operator [] return type is supposed to be T&, but why is it a special case for vector<bool>?
What does vector<bool> really consist of?
The Item further says:
deque<bool> v; // is a STL container and it really contains bools
Can this be used as an alternative to vector<bool>?
Can anyone please explain this?
For space-optimization reasons, the C++ standard (as far back as C++98) explicitly calls out vector<bool> as a special standard container where each bool uses only one bit of space rather than one byte as a normal bool would (implementing a kind of "dynamic bitset"). In exchange for this optimization it doesn't offer all the capabilities and interface of a normal standard container.
In this case, since you can't take the address of a bit within a byte, things such as operator[] can't return a bool& but instead return a proxy object that allows to manipulate the particular bit in question. Since this proxy object is not a bool&, you can't assign its address to a bool* like you could with the result of such an operator call on a "normal" container. In turn this means that bool *pb =&v[0]; isn't valid code.
On the other hand deque doesn't have any such specialization called out so each bool takes a byte and you can take the address of the value return from operator[].
Finally note that the MS standard library implementation is (arguably) suboptimal in that it uses a small chunk size for deques, which means that using deque as a substitute isn't always the right answer.
The problems is that vector<bool> returns a proxy reference object instead of a true reference, so that C++98 style code bool * p = &v[0]; won't compile. However, modern C++11 with auto p = &v[0]; can be made to compile if operator& also returns a proxy pointer object. Howard Hinnant has written a blog post detailing the algorithmic improvements when using such proxy references and pointers.
Scott Meyers has a long Item 30 in More Effective C++ about proxy classes. You can come a long way to almost mimic the builtin types: for any given type T, a pair of proxies (e.g. reference_proxy<T> and iterator_proxy<T>) can be made mutually consistent in the sense that reference_proxy<T>::operator&() and iterator_proxy<T>::operator*() are each other's inverse.
However, at some point one needs to map the proxy objects back to behave like T* or T&. For iterator proxies, one can overload operator->() and access the template T's interface without reimplementing all the functionality. However, for reference proxies, you would need to overload operator.(), and that is not allowed in current C++ (although Sebastian Redl presented such a proposal on BoostCon 2013). You can make a verbose work-around like a .get() member inside the reference proxy, or implement all of T's interface inside the reference (this is what is done for vector<bool>::bit_reference), but this will either lose the builtin syntax or introduce user-defined conversions that do not have builtin semantics for type conversions (you can have at most one user-defined conversion per argument).
TL;DR: no vector<bool> is not a container because the Standard requires a real reference, but it can be made to behave almost like a container, at least much closer with C++11 (auto) than in C++98.
vector<bool> contains boolean values in compressed form using only one bit for value (and not 8 how bool[] arrays do). It is not possible to return a reference to a bit in c++, so there is a special helper type, "bit reference", which provides you a interface to some bit in memory and allows you to use standard operators and casts.
Many consider the vector<bool> specialization to be a mistake.
In a paper "Deprecating Vestigial Library Parts in C++17"
There is a proposal to
Reconsider vector Partial Specialization.
There has been a long history of the bool partial specialization of
std::vector not satisfying the container requirements, and in
particular, its iterators not satisfying the requirements of a random
access iterator. A previous attempt to deprecate this container was
rejected for C++11, N2204.
One of the reasons for rejection is that it is not clear what it would
mean to deprecate a particular specialization of a template. That
could be addressed with careful wording. The larger issue is that the
(packed) specialization of vector offers an important
optimization that clients of the standard library genuinely seek, but
would not longer be available. It is unlikely that we would be able to
deprecate this part of the standard until a replacement facility is
proposed and accepted, such as N2050. Unfortunately, there are no such
revised proposals currently being offered to the Library Evolution
Working Group.
Look at how it is implemented. the STL builds vastly on templates and therefore the headers do contain the code they do.
for instance look at the stdc++ implementation here.
also interesting even though not an stl conforming bit vector is the llvm::BitVector from here.
the essence of the llvm::BitVector is a nested class called reference and suitable operator overloading to make the BitVector behaves similar to vector with some limitations. The code below is a simplified interface to show how BitVector hides a class called reference to make the real implementation almost behave like a real array of bool without using 1 byte for each value.
class BitVector {
public:
class reference {
reference &operator=(reference t);
reference& operator=(bool t);
operator bool() const;
};
reference operator[](unsigned Idx);
bool operator[](unsigned Idx) const;
};
this code here has the nice properties:
BitVector b(10, false); // size 10, default false
BitVector::reference &x = b[5]; // that's what really happens
bool y = b[5]; // implicitly converted to bool
assert(b[5] == false); // converted to bool
assert(b[6] == b[7]); // bool operator==(const reference &, const reference &);
b[5] = true; // assignment on reference
assert(b[5] == true); // and actually it does work.
This code actually has a flaw, try to run:
std::for_each(&b[5], &b[6], some_func); // address of reference not an iterator
will not work because assert( (&b[5] - &b[3]) == (5 - 3) ); will fail (within llvm::BitVector)
this is the very simple llvm version. std::vector<bool> has also working iterators in it.
thus the call for(auto i = b.begin(), e = b.end(); i != e; ++i) will work. and also std::vector<bool>::const_iterator.
However there are still limitations in std::vector<bool> that makes it behave differently in some cases.
This comes from http://www.cplusplus.com/reference/vector/vector-bool/
Vector of bool This is a specialized version of vector, which is used
for elements of type bool and optimizes for space.
It behaves like the unspecialized version of vector, with the
following changes:
The storage is not necessarily an array of bool values, but the library implementation may optimize storage so that each value is
stored in a single bit.
Elements are not constructed using the allocator object, but their value is directly set on the proper bit in the internal storage.
Member function flip and a new signature for member swap.
A special member type, reference, a class that accesses individual bits in the container's internal storage with an interface that
emulates a bool reference. Conversely, member type const_reference is
a plain bool.
The pointer and iterator types used by the container are not necessarily neither pointers nor conforming iterators, although they
shall simulate most of their expected behavior.
These changes provide a quirky interface to this specialization and
favor memory optimization over processing (which may or may not suit
your needs). In any case, it is not possible to instantiate the
unspecialized template of vector for bool directly. Workarounds to
avoid this range from using a different type (char, unsigned char) or
container (like deque) to use wrapper types or further specialize for
specific allocator types.
bitset is a class that provides a similar functionality for fixed-size
arrays of bits.
Item 18 of Scott Meyers's book Effective STL: 50 Specific Ways to Improve Your Use of the Standard Template Library says to avoid vector <bool> as it's not an STL container and it doesn't really hold bools.
The following code:
vector <bool> v;
bool *pb =&v[0];
will not compile, violating a requirement of STL containers.
Error:
cannot convert 'std::vector<bool>::reference* {aka std::_Bit_reference*}' to 'bool*' in initialization
vector<T>::operator [] return type is supposed to be T&, but why is it a special case for vector<bool>?
What does vector<bool> really consist of?
The Item further says:
deque<bool> v; // is a STL container and it really contains bools
Can this be used as an alternative to vector<bool>?
Can anyone please explain this?
For space-optimization reasons, the C++ standard (as far back as C++98) explicitly calls out vector<bool> as a special standard container where each bool uses only one bit of space rather than one byte as a normal bool would (implementing a kind of "dynamic bitset"). In exchange for this optimization it doesn't offer all the capabilities and interface of a normal standard container.
In this case, since you can't take the address of a bit within a byte, things such as operator[] can't return a bool& but instead return a proxy object that allows to manipulate the particular bit in question. Since this proxy object is not a bool&, you can't assign its address to a bool* like you could with the result of such an operator call on a "normal" container. In turn this means that bool *pb =&v[0]; isn't valid code.
On the other hand deque doesn't have any such specialization called out so each bool takes a byte and you can take the address of the value return from operator[].
Finally note that the MS standard library implementation is (arguably) suboptimal in that it uses a small chunk size for deques, which means that using deque as a substitute isn't always the right answer.
The problems is that vector<bool> returns a proxy reference object instead of a true reference, so that C++98 style code bool * p = &v[0]; won't compile. However, modern C++11 with auto p = &v[0]; can be made to compile if operator& also returns a proxy pointer object. Howard Hinnant has written a blog post detailing the algorithmic improvements when using such proxy references and pointers.
Scott Meyers has a long Item 30 in More Effective C++ about proxy classes. You can come a long way to almost mimic the builtin types: for any given type T, a pair of proxies (e.g. reference_proxy<T> and iterator_proxy<T>) can be made mutually consistent in the sense that reference_proxy<T>::operator&() and iterator_proxy<T>::operator*() are each other's inverse.
However, at some point one needs to map the proxy objects back to behave like T* or T&. For iterator proxies, one can overload operator->() and access the template T's interface without reimplementing all the functionality. However, for reference proxies, you would need to overload operator.(), and that is not allowed in current C++ (although Sebastian Redl presented such a proposal on BoostCon 2013). You can make a verbose work-around like a .get() member inside the reference proxy, or implement all of T's interface inside the reference (this is what is done for vector<bool>::bit_reference), but this will either lose the builtin syntax or introduce user-defined conversions that do not have builtin semantics for type conversions (you can have at most one user-defined conversion per argument).
TL;DR: no vector<bool> is not a container because the Standard requires a real reference, but it can be made to behave almost like a container, at least much closer with C++11 (auto) than in C++98.
vector<bool> contains boolean values in compressed form using only one bit for value (and not 8 how bool[] arrays do). It is not possible to return a reference to a bit in c++, so there is a special helper type, "bit reference", which provides you a interface to some bit in memory and allows you to use standard operators and casts.
Many consider the vector<bool> specialization to be a mistake.
In a paper "Deprecating Vestigial Library Parts in C++17"
There is a proposal to
Reconsider vector Partial Specialization.
There has been a long history of the bool partial specialization of
std::vector not satisfying the container requirements, and in
particular, its iterators not satisfying the requirements of a random
access iterator. A previous attempt to deprecate this container was
rejected for C++11, N2204.
One of the reasons for rejection is that it is not clear what it would
mean to deprecate a particular specialization of a template. That
could be addressed with careful wording. The larger issue is that the
(packed) specialization of vector offers an important
optimization that clients of the standard library genuinely seek, but
would not longer be available. It is unlikely that we would be able to
deprecate this part of the standard until a replacement facility is
proposed and accepted, such as N2050. Unfortunately, there are no such
revised proposals currently being offered to the Library Evolution
Working Group.
Look at how it is implemented. the STL builds vastly on templates and therefore the headers do contain the code they do.
for instance look at the stdc++ implementation here.
also interesting even though not an stl conforming bit vector is the llvm::BitVector from here.
the essence of the llvm::BitVector is a nested class called reference and suitable operator overloading to make the BitVector behaves similar to vector with some limitations. The code below is a simplified interface to show how BitVector hides a class called reference to make the real implementation almost behave like a real array of bool without using 1 byte for each value.
class BitVector {
public:
class reference {
reference &operator=(reference t);
reference& operator=(bool t);
operator bool() const;
};
reference operator[](unsigned Idx);
bool operator[](unsigned Idx) const;
};
this code here has the nice properties:
BitVector b(10, false); // size 10, default false
BitVector::reference &x = b[5]; // that's what really happens
bool y = b[5]; // implicitly converted to bool
assert(b[5] == false); // converted to bool
assert(b[6] == b[7]); // bool operator==(const reference &, const reference &);
b[5] = true; // assignment on reference
assert(b[5] == true); // and actually it does work.
This code actually has a flaw, try to run:
std::for_each(&b[5], &b[6], some_func); // address of reference not an iterator
will not work because assert( (&b[5] - &b[3]) == (5 - 3) ); will fail (within llvm::BitVector)
this is the very simple llvm version. std::vector<bool> has also working iterators in it.
thus the call for(auto i = b.begin(), e = b.end(); i != e; ++i) will work. and also std::vector<bool>::const_iterator.
However there are still limitations in std::vector<bool> that makes it behave differently in some cases.
This comes from http://www.cplusplus.com/reference/vector/vector-bool/
Vector of bool This is a specialized version of vector, which is used
for elements of type bool and optimizes for space.
It behaves like the unspecialized version of vector, with the
following changes:
The storage is not necessarily an array of bool values, but the library implementation may optimize storage so that each value is
stored in a single bit.
Elements are not constructed using the allocator object, but their value is directly set on the proper bit in the internal storage.
Member function flip and a new signature for member swap.
A special member type, reference, a class that accesses individual bits in the container's internal storage with an interface that
emulates a bool reference. Conversely, member type const_reference is
a plain bool.
The pointer and iterator types used by the container are not necessarily neither pointers nor conforming iterators, although they
shall simulate most of their expected behavior.
These changes provide a quirky interface to this specialization and
favor memory optimization over processing (which may or may not suit
your needs). In any case, it is not possible to instantiate the
unspecialized template of vector for bool directly. Workarounds to
avoid this range from using a different type (char, unsigned char) or
container (like deque) to use wrapper types or further specialize for
specific allocator types.
bitset is a class that provides a similar functionality for fixed-size
arrays of bits.
Is it valid to create an iterator to end(str)+1 for std::string?
And if it isn't, why isn't it?
This question is restricted to C++11 and later, because while pre-C++11 the data was already stored in a continuous block in any but rare POC toy-implementations, the data didn't have to be stored that way.
And I think that might make all the difference.
The significant difference between std::string and any other standard container I speculate on is that it always contains one element more than its size, the zero-terminator, to fulfill the requirements of .c_str().
21.4.7.1 basic_string accessors [string.accessors]
const charT* c_str() const noexcept;
const charT* data() const noexcept;
1 Returns: A pointer p such that p + i == &operator[](i) for each i in [0,size()].
2 Complexity: Constant time.
3 Requires: The program shall not alter any of the values stored in the character array.
Still, even though it should imho guarantee that said expression is valid, for consistency and interoperability with zero-terminated strings if nothing else, the only paragraph I found casts doubt on that:
21.4.1 basic_string general requirements [string.require]
4 The char-like objects in a basic_string object shall be stored contiguously. That is, for any basic_string object s, the identity &*(s.begin() + n) == &*s.begin() + n shall hold for all values of n such that 0 <= n < s.size().
(All quotes are from C++14 final draft (n3936).)
Related: Legal to overwrite std::string's null terminator?
TL;DR: s.end() + 1 is undefined behavior.
std::string is a strange beast, mainly for historical reasons:
It attempts to bring C compatibility, where it is known that an additional \0 character exists beyond the length reported by strlen.
It was designed with an index-based interface.
As an after thought, when merged in the Standard library with the rest of the STL code, an iterator-based interface was added.
This led std::string, in C++03, to number 103 member functions, and since then a few were added.
Therefore, discrepancies between the different methods should be expected.
Already in the index-based interface discrepancies appear:
§21.4.5 [string.access]
const_reference operator[](size_type pos) const;
reference operator[](size_type pos);
1/ Requires: pos <= size()
const_reference at(size_type pos) const;
reference at(size_type pos);
5/ Throws: out_of_range if pos >= size()
Yes, you read this right, s[s.size()] returns a reference to a NUL character while s.at(s.size()) throws an out_of_range exception. If anyone tells you to replace all uses of operator[] by at because they are safer, beware the string trap...
So, what about iterators?
§21.4.3 [string.iterators]
iterator end() noexcept;
const_iterator end() const noexcept;
const_iterator cend() const noexcept;
2/ Returns: An iterator which is the past-the-end value.
Wonderfully bland.
So we have to refer to other paragraphs. A pointer is offered by
§21.4 [basic.string]
3/ The iterators supported by basic_string are random access iterators (24.2.7).
while §17.6 [requirements] seems devoid of anything related. Thus, strings iterators are just plain old iterators (you can probably sense where this is going... but since we came this far let's go all the way).
This leads us to:
24.2.1 [iterator.requirements.general]
5/ Just as a regular pointer to an array guarantees that there is a pointer value pointing past the last element of the array, so for any iterator type there is an iterator value that points past the last element of a corresponding sequence. These values are called past-the-end values. Values of an iterator i for which the expression *i is defined are called dereferenceable. The library never assumes that past-the-end values are dereferenceable. [...]
So, *s.end() is ill-formed.
24.2.3 [input.iterators]
2/ Table 107 -- Input iterator requirements (in addition to Iterator)
List for pre-condition to ++r and r++ that r be dereferencable.
Neither the Forward iterators, Bidirectional iterators nor Random iterator lift this restriction (and all indicate they inherit the restrictions of their predecessor).
Also, for completeness, in 24.2.7 [random.access.iterators], Table 111 -- Random access iterator requirements (in addition to bidirectional iterator) lists the following operational semantics:
r += n is equivalent to [inc|dec]rememting r n times
a + n and n + a are equivalent to copying a and then applying += n to the copy
and similarly for -= n and - n.
Thus s.end() + 1 is undefined behavior.
Returns: A pointer p such that p + i == &operator[](i) for each i in [0,size()].
std::string::operator[](size_type i) is specified to return "a reference to an object of type charT with value charT() when i == size(), so we know that that pointer points to an object.
5.7 states that "For the purposes of [operators + and -], a pointer to a nonarray object behaves the same as a pointer to the first element of an array of length one with the type of the object as its element type."
So we have a non-array object and the spec guarantees that a pointer one past it will be representable. So we know std::addressof(*end(str)) + 1 has to be representable.
However that's not a guarantee on std::string::iterator, and there is no such guarantee anywhere in the spec, which makes it undefined behavior.
(Note that this is not the same as 'ill-formed'. *end(str) + 1 is in fact well-formed.)
Iterators can and do implement checking logic that produce various errors when you do things like increment the end() iterator. This is in fact what Visual Studios debug iterators do with end(str) + 1.
#define _ITERATOR_DEBUG_LEVEL 2
#include <string>
#include <iterator>
int main() {
std::string s = "ssssssss";
auto x = std::end(s) + 1; // produces debug dialog, aborts program if skipped
}
And if it isn't, why isn't it?
for consistency and interoperability with zero-terminated strings if nothing else
C++ specifies some specific things for compatibility with C, but such backwards compatibility is limited to supporting things that can actually be written in C. C++ doesn't necessarily try to take C's semantics and make new constructs behave in some analogous way. Should std::vector decay to an iterator just to be consistent with C's array decay behavior?
I'd say end(std) + 1 is left as undefined behavior because there's no value in trying to constrain std::string iterators this way. There's no legacy C code that does this that C++ needs to be compatible with and new code should be prevented from doing it.
New code should be prevented from relying on it... why? [...] What does not allowing it buy you in theory, and how does that look in practice?
Not allowing it means implementations don't have to support the added complexity, complexity which provides zero demonstrated value.
In fact it seems to me that supporting end(str) + 1 has negative value since code that tries to use it will essentially be creating the same problem as C code which can't figure out when to account for the null terminator or not. C has enough off by one buffer size errors for both languages.
A std::basic_string<???> is a container over its elements. Its elements do not include the trailing null that is implicitly added (it can include embedded nulls).
This makes lots of sense -- "for each character in this string" probably shouldn't return the trailing '\0', as that is really an implementation detail for compatibility with C style APIs.
The iterator rules for containers were based off of containers that don't shove an extra element at the end. Modifying them for std::basic_string<???> without motivation is questionable; one should only break a working pattern if there is a payoff.
There is every reason to think that pointers to .data() and .data() + .size() + 1 are allowed (I could imagine a twisted interpretation of the standard that would make it not allowed). So if you really need read-only iterators into the contents of a std::string, you can use pointer-to-const-elements (which are, after all, a kind of iterator).
If you want editable ones, then no, there is no way to get a valid iterator to one-past-the-end. Neither can you get a non-const reference to the trailing null legally. In fact, such access is clearly a bad idea; if you change the value of that element, you break the std::basic_string's invariant null-termination.
For there to be an iterator to one-past-the-end, the const and non-const iterators to the container would have to have a different valid range, or a non-const iterator to the last element that can be dereferenced but not written to must exist.
I shudder at making such standard wording watertight.
std::basic_string is already a mess. Making it even stranger would lead to standard bugs and would have a non-trivial cost. The benefit is really low; in the few cases where you want access to said trailing null in an iterator range, you can use .data() and use the resulting pointers as iterators.
I can't find a definitive answer, but indirect evidence points at end()+1 being undefined.
[string.insert]/15
constexpr iterator insert(const_iterator p, charT c);
Preconditions: p is a valid iterator on *this.
It would be unreasonable to expect this to work with end()+1 as the iterator, and it indeed causes a crash on both libstdc++ and libc++.
This means end()+1 is not a valid iterator, meaning end() is not incrementable.
It's well known that std::vector<bool> does not satisfy the Standard's container requirements, mainly because the packed representation prevents T* x = &v[i] from returning a pointer to a bool.
My question is: can this be remedied/mitigated when the reference_proxy overloads the address-of operator& to return a pointer_proxy?
The pointer-proxy could contain the same data as the reference_proxy in most implementations, namely a pointer into the packed data and a mask to isolate the particular bit inside the block pointed to. Indirection of the pointer_proxy would then yield the reference_proxy. Essentially both proxies are "fat" pointers, which are, however, still rather light-weight compared to disk-based proxy containers.
Instead of T* x = &v[0] one could then do auto x = &v[0], and use x like if(*x) without problems. I would also like to be able to write for(auto b: v) { /* ... */ }
Questions: would such a multi-proxy approach work with the STL's algorithms? Or do some algorithms really rely on the requirement that x needs to be a real bool*? Or are there too many consecutive user-defined conversions required that prevent this to work? I'd like to know any of such obstructions before trying to fully complete the above implementation sketch.
UPDATE (based on #HowardHinnant 's answer and this ancient discussion on comp.std.c++)
You can come a long way to almost mimic the builtin types: for any given type T, a pair of proxies (e.g. reference_proxy and iterator_proxy) can be made mutually consistent in the sense that reference_proxy::operator&() and iterator_proxy::operator*() are each other's inverse.
However, at some point one needs to map the proxy objects back to behave like T* or T&. For iterator proxies, one can overload operator->() and access the template T's interface without reimplementing all the functionality. However, for reference proxies, you would need to overload operator.(), and that is not allowed in current C++ (although Sebastian Redl presented such a proposal on BoostCon 2013). You can make a verbose work-around like a .get() member inside the reference proxy, or implement all of T's interface inside the reference (this is what is done for vector::bit_reference), but this will either lose the builtin syntax or introduce user-defined conversions that do not have builtin semantics for type conversions (you can have at most one user-defined conversion per argument).
My question is: can this be remedied/mitigated when the
reference_proxy overloads the address-of operator& to return a
pointer_proxy?
libc++ actually does this.
#include <vector>
#include <cassert>
int main()
{
std::vector<bool> v(1);
std::vector<bool>::pointer pb = &v[0];
assert(*pb == false);
*pb = true;
assert(v[0] == true);
std::vector<bool>::const_pointer cbp = pb;
assert(*cbp == true);
v[0] = false;
assert(*cbp == false);
}
It even extends to const_pointer and const_reference in ways that mimic the same types for vector<int>. This is a non-conforming extension for libc++. But it makes writing generic code which might be instantiated on vector<bool> far more likely to compile and behave correctly.
Questions: would such a multi-proxy approach work with the STL's
algorithms? Or do some algorithms really rely on the requirement that
x needs to be a real bool*? Or are there too many consecutive
user-defined conversions required that prevent this to work?
All of libc++'s algorithms work with vector<bool>. Some of them with quite spectacular performance. One algorithm in particular must have special treatment which the standard unfortunately does not mandate:
#include <vector>
#include <cassert>
int main()
{
std::vector<bool> v(1);
bool b = true;
assert(v[0] == false);
assert(b == true);
std::swap(b, v[0]);
assert(v[0] == true);
assert(b == false);
}
This is very easy for the implementation to accomplish. One simply needs to make sure swap works for any combination of bool and vector<bool>::reference. But I don't know if any implementation besides libc++ does this, and it is not mandated by C++11.
An array of bits is a wonderful data structure. But unfortunately it is poorly specified in the C++ standard. libc++ has gone somewhat outlaw to demonstrate that this can be a very useful and high performance data structure. The hope is that a future C++ standard may migrate in this direction to the benefit of the C++ programmer.
Offhand I would say first, that it will actually depend more on the particulars of each individual STL implementation since it doesn't officially conform to the standard requirement that a *reference_type to be lvalue of T*. So regarding potential implementation issues:
The main reason any piece of code would be explicitly dependent on the container's pointer being a real bool* is if the algo was using pointer arithmetic, in which case the size of the pointer type becomes relevant. Pointer arithmetic though would bypass the iterator interface and thus defeat the main purpose of the whole STL container-by-iterator design. std::vector<> itself is guaranteed to be contiguous in C++11, which allows optimized specializations of both STL algos and compiler for(:), both of which may use pointer arithmetic internally. If your type isn't derived from std::vector then that shouldn't be an issue; everything should just assume the iterator method instead.
However! STL code may still take pointers not for the purpose of pointer arithmetic but rather for some other purpose. In this case the problem is C++ syntax. Eg, quoting your own question:
Instead of T* x = &v[0] one could then do auto x = &v[0]
Any templated code in the STL will also have to do the same thing... and that seems entirely unlikely at this point that STL implementations will be making wide use of auto. There may be other situations were the STL attempts to do clever r-value casting tricks that end up failing because it isn't expecting mismatched reference types.
Regarding for(auto b: v) { /* ... */ }: I see no reason that shouldn't work. I think it will generate code that will be far less efficient than the same version you could just roll yourself in 15 mins (or less). I only bring it up since you mention intrinsics in the OP, which imples some consideration for performance. You won't be able to help it out using intrinsics either. There's nothing an intrinsic can do that somehow surpasses a simple bitwise shift for sequentially traversing an array of bits. Most of the added overhead will be from the compiler generating code to update the iterator pointer and mask values, and then reload the mask value on the next iteration. It won't be able to magically deduce what you're trying to do and turn it into a sequential shift op for you. It may at least be able to optimize out the pointer update+writeback stage by caching it into a register outside the loop, though honestly I'd be very skeptical based on my experiences.
Here's one way for going through bits from start to end, just for sake of comparison (a version capable of starting at any arbitrary point in the bitstream would require a little extra setup logic):
uint64_t* pBitSet = &v[-1]; // gets incremented on first iteration through loop.
uint64_t curBitSet = v[0];
for (int i=0; i<v.length(); ++i) {
if ((i % 64) == 0) {
curBitSet = *(++pBitSet);
}
int bit = curBitSet & 1;
curBitSet >>= 1;
// do stuff based on 'bit' here.
}
Consider this hypothetical implementation of vector:
template<class T> // ignore the allocator
struct vector
{
typedef T* iterator;
typedef const T* const_iterator;
template<class It>
void insert(iterator where, It begin, It end)
{
...
}
...
}
Problem
There is a subtle problem we face here:
There is the possibility that begin and end refer to items in the same vector, after where.
For example, if the user says:
vector<int> items;
for (int i = 0; i < 1000; i++)
items.push_back(i);
items.insert(items.begin(), items.end() - 2, items.end() - 1);
If It is not a pointer type, then we're fine.
But we don't know, so we must check that [begin, end) does not refer to a range already inside the vector.
But how do we do this? According to C++, if they don't refer to the same array, then pointer comparisons would be undefined!
So the compiler could falsely tell us that the items don't alias, when in fact they do, giving us unnecessary O(n) slowdown.
Potential solution & caveat
One solution is to copy the entire vector every time, to include the new items, and then throw away the old copy.
But that's very slow in scenarios such as in the example above, where we'd be copying 1000 items just to insert 1 item, even though we might clearly already have enough capacity.
Is there a generic way to (correctly) solve this problem efficiently, i.e. without suffering from O(n) slowdown in cases where nothing is aliasing?
You can use the predicates std::less etc, which are guaranteed to give a total order, even when the raw pointer comparisons do not.
From the standard [comparisons]/8:
For templates greater, less, greater_equal, and less_equal, the specializations for any pointer type yield a total order, even if the built-in operators <, >, <=, >= do not.
But how do we do this? According to C++, if they don't refer to the same array, then pointer comparisons would be undefined!
Wrong. The pointer comparisons are unspecified, not undefined. From C++03 §5.9/2 [expr.rel]:
[...] Pointers to objects or functions of the same type (after pointer conversions) can be compared, with a result defined as follows:
[...]
-Other pointer comparisons are unspecified.
So it's safe to test if there is an overlap before doing the expensive-but-correct copy.
Interestingly, C99 differs from C++ in this, in that pointer comparisons between unrelated objects is undefined behavior. From C99 §6.5.8/5:
When two pointers are compared, the result depends on the relative locations in the address space of the objects pointed to. [...] In all other cases, the behavior is undefined.
Actually, this would be true even if they were regular iterators. There's nothing stopping anyone doing
std::vector<int> v;
// fill v
v.insert(v.end() - 3, v.begin(), v.end());
Determining if they alias is a problem for any implementation of iterators.
However, the thing you're missing is that you're the implementation, you don't have to use portable code. As the implementation, you can do whatever you want. You could say "Well, in my implementation, I follow x86 and < and > are fine to use for any pointers.". And that would be fine.