The following code doesn't compile
#include <vector>
int main()
{
std::vector<bool> enable(10);
enable[0] |= true;
return 0;
}
giving the error
no match for ‘operator|=’ (operand types are ‘std::vector<bool>::reference {aka std::_Bit_reference}’ and ‘bool’)
In my real life code I have a bit field with values I want to |= with the result of a function.
There are easy way to express the same idea, but is there any good reason for such an operator not to be available ?
The main reason would be that std::vector<bool> is special, and its specification specifically permits an implementation to minimise memory usage.
For vectors of anything other than bool, the reference type can actually be a true reference (i.e. std::vector<int>::reference can actually be an int &) - usually directly referencing an element of the vector itself. So it makes sense for the reference type to support all operations that the underlying type can. This works because vector<int> effectively manages a contiguous array of int internally. The same goes for all types other than bool.
However, to minimise memory usage, a std::vector<bool> may not (in fact probably will not) work internally with an actual array of bool. Instead it might use some packed data structure, such as an array of unsigned char internally, where each unsigned char is a bitfield containing 8 bits. So a vector<bool> of length 800 would actually manage an array of 100 unsigned char, and the memory it consumes would be 100 bytes (assuming no over-allocation). If the vector<bool> actually contained an array of 800 bool, its memory usage would be a minimum of 800 bytes (since sizeof(bool) must be at least 1, by definition).
To permit such memory optimisation by implementers of vector<bool>, the return type of vector<bool>::operator[] (i.e. std::vector<bool>::reference) cannot simply be a bool &. Internally, it would probably contain a reference to the underlying type (e.g. a unsigned char) and information to track what bit it actually affects. This would make all op= operators (+=, -=, |=, etc) somewhat expensive operations (e.g. bit fiddling) on the underlying type.
The designers of std::vector<bool> would then have faced a choice between
specify that std::vector<bool>::reference support all the
op= and hear continual complaints about runtime inefficiency from
programmers who use those operators
Don't support those op= and field complaints from programmers who think such things are okay ("cleaner code", etc) even though they will be inefficient.
It appears the designers of std::vector<bool> opted for option 2. A consequence is that the only assignment operators supported by std::vector<bool>::reference are the stock standard operator=() (with operands either of type reference, or of type bool) not any of the op=. The advantage of this choice is that programmers get a compilation error if trying to do something which is actually a poor choice in practice.
After all, although bool supports all the op= using them doesn't achieve much anyway. For example, some_bool |= true has the same net effect as some_bool = true.
Why don't you just do the following?
enable[0] = enable[0] | true;
You should be able to make one yourself pretty easily. Something like:
std::vector<bool>::reference& operator |= (std::vector<bool>::reference& a, bool b)
{
if (b)
a = true;
return a;
}
Alternatively, std::bitset is a good fit.
Short and sweet answer: std::vector<bool> should be avoided. Use vector<wchar> instead. You actually get a container back in which the bools are packed in bits, which gives different behaviour from other vectors, slow code and no-one cares a bout memory anyway. I guess by now no-one likes this anymore, but turning back the clock would break too much code...
Related
Item 18 of Scott Meyers's book Effective STL: 50 Specific Ways to Improve Your Use of the Standard Template Library says to avoid vector <bool> as it's not an STL container and it doesn't really hold bools.
The following code:
vector <bool> v;
bool *pb =&v[0];
will not compile, violating a requirement of STL containers.
Error:
cannot convert 'std::vector<bool>::reference* {aka std::_Bit_reference*}' to 'bool*' in initialization
vector<T>::operator [] return type is supposed to be T&, but why is it a special case for vector<bool>?
What does vector<bool> really consist of?
The Item further says:
deque<bool> v; // is a STL container and it really contains bools
Can this be used as an alternative to vector<bool>?
Can anyone please explain this?
For space-optimization reasons, the C++ standard (as far back as C++98) explicitly calls out vector<bool> as a special standard container where each bool uses only one bit of space rather than one byte as a normal bool would (implementing a kind of "dynamic bitset"). In exchange for this optimization it doesn't offer all the capabilities and interface of a normal standard container.
In this case, since you can't take the address of a bit within a byte, things such as operator[] can't return a bool& but instead return a proxy object that allows to manipulate the particular bit in question. Since this proxy object is not a bool&, you can't assign its address to a bool* like you could with the result of such an operator call on a "normal" container. In turn this means that bool *pb =&v[0]; isn't valid code.
On the other hand deque doesn't have any such specialization called out so each bool takes a byte and you can take the address of the value return from operator[].
Finally note that the MS standard library implementation is (arguably) suboptimal in that it uses a small chunk size for deques, which means that using deque as a substitute isn't always the right answer.
The problems is that vector<bool> returns a proxy reference object instead of a true reference, so that C++98 style code bool * p = &v[0]; won't compile. However, modern C++11 with auto p = &v[0]; can be made to compile if operator& also returns a proxy pointer object. Howard Hinnant has written a blog post detailing the algorithmic improvements when using such proxy references and pointers.
Scott Meyers has a long Item 30 in More Effective C++ about proxy classes. You can come a long way to almost mimic the builtin types: for any given type T, a pair of proxies (e.g. reference_proxy<T> and iterator_proxy<T>) can be made mutually consistent in the sense that reference_proxy<T>::operator&() and iterator_proxy<T>::operator*() are each other's inverse.
However, at some point one needs to map the proxy objects back to behave like T* or T&. For iterator proxies, one can overload operator->() and access the template T's interface without reimplementing all the functionality. However, for reference proxies, you would need to overload operator.(), and that is not allowed in current C++ (although Sebastian Redl presented such a proposal on BoostCon 2013). You can make a verbose work-around like a .get() member inside the reference proxy, or implement all of T's interface inside the reference (this is what is done for vector<bool>::bit_reference), but this will either lose the builtin syntax or introduce user-defined conversions that do not have builtin semantics for type conversions (you can have at most one user-defined conversion per argument).
TL;DR: no vector<bool> is not a container because the Standard requires a real reference, but it can be made to behave almost like a container, at least much closer with C++11 (auto) than in C++98.
vector<bool> contains boolean values in compressed form using only one bit for value (and not 8 how bool[] arrays do). It is not possible to return a reference to a bit in c++, so there is a special helper type, "bit reference", which provides you a interface to some bit in memory and allows you to use standard operators and casts.
Many consider the vector<bool> specialization to be a mistake.
In a paper "Deprecating Vestigial Library Parts in C++17"
There is a proposal to
Reconsider vector Partial Specialization.
There has been a long history of the bool partial specialization of
std::vector not satisfying the container requirements, and in
particular, its iterators not satisfying the requirements of a random
access iterator. A previous attempt to deprecate this container was
rejected for C++11, N2204.
One of the reasons for rejection is that it is not clear what it would
mean to deprecate a particular specialization of a template. That
could be addressed with careful wording. The larger issue is that the
(packed) specialization of vector offers an important
optimization that clients of the standard library genuinely seek, but
would not longer be available. It is unlikely that we would be able to
deprecate this part of the standard until a replacement facility is
proposed and accepted, such as N2050. Unfortunately, there are no such
revised proposals currently being offered to the Library Evolution
Working Group.
Look at how it is implemented. the STL builds vastly on templates and therefore the headers do contain the code they do.
for instance look at the stdc++ implementation here.
also interesting even though not an stl conforming bit vector is the llvm::BitVector from here.
the essence of the llvm::BitVector is a nested class called reference and suitable operator overloading to make the BitVector behaves similar to vector with some limitations. The code below is a simplified interface to show how BitVector hides a class called reference to make the real implementation almost behave like a real array of bool without using 1 byte for each value.
class BitVector {
public:
class reference {
reference &operator=(reference t);
reference& operator=(bool t);
operator bool() const;
};
reference operator[](unsigned Idx);
bool operator[](unsigned Idx) const;
};
this code here has the nice properties:
BitVector b(10, false); // size 10, default false
BitVector::reference &x = b[5]; // that's what really happens
bool y = b[5]; // implicitly converted to bool
assert(b[5] == false); // converted to bool
assert(b[6] == b[7]); // bool operator==(const reference &, const reference &);
b[5] = true; // assignment on reference
assert(b[5] == true); // and actually it does work.
This code actually has a flaw, try to run:
std::for_each(&b[5], &b[6], some_func); // address of reference not an iterator
will not work because assert( (&b[5] - &b[3]) == (5 - 3) ); will fail (within llvm::BitVector)
this is the very simple llvm version. std::vector<bool> has also working iterators in it.
thus the call for(auto i = b.begin(), e = b.end(); i != e; ++i) will work. and also std::vector<bool>::const_iterator.
However there are still limitations in std::vector<bool> that makes it behave differently in some cases.
This comes from http://www.cplusplus.com/reference/vector/vector-bool/
Vector of bool This is a specialized version of vector, which is used
for elements of type bool and optimizes for space.
It behaves like the unspecialized version of vector, with the
following changes:
The storage is not necessarily an array of bool values, but the library implementation may optimize storage so that each value is
stored in a single bit.
Elements are not constructed using the allocator object, but their value is directly set on the proper bit in the internal storage.
Member function flip and a new signature for member swap.
A special member type, reference, a class that accesses individual bits in the container's internal storage with an interface that
emulates a bool reference. Conversely, member type const_reference is
a plain bool.
The pointer and iterator types used by the container are not necessarily neither pointers nor conforming iterators, although they
shall simulate most of their expected behavior.
These changes provide a quirky interface to this specialization and
favor memory optimization over processing (which may or may not suit
your needs). In any case, it is not possible to instantiate the
unspecialized template of vector for bool directly. Workarounds to
avoid this range from using a different type (char, unsigned char) or
container (like deque) to use wrapper types or further specialize for
specific allocator types.
bitset is a class that provides a similar functionality for fixed-size
arrays of bits.
Item 18 of Scott Meyers's book Effective STL: 50 Specific Ways to Improve Your Use of the Standard Template Library says to avoid vector <bool> as it's not an STL container and it doesn't really hold bools.
The following code:
vector <bool> v;
bool *pb =&v[0];
will not compile, violating a requirement of STL containers.
Error:
cannot convert 'std::vector<bool>::reference* {aka std::_Bit_reference*}' to 'bool*' in initialization
vector<T>::operator [] return type is supposed to be T&, but why is it a special case for vector<bool>?
What does vector<bool> really consist of?
The Item further says:
deque<bool> v; // is a STL container and it really contains bools
Can this be used as an alternative to vector<bool>?
Can anyone please explain this?
For space-optimization reasons, the C++ standard (as far back as C++98) explicitly calls out vector<bool> as a special standard container where each bool uses only one bit of space rather than one byte as a normal bool would (implementing a kind of "dynamic bitset"). In exchange for this optimization it doesn't offer all the capabilities and interface of a normal standard container.
In this case, since you can't take the address of a bit within a byte, things such as operator[] can't return a bool& but instead return a proxy object that allows to manipulate the particular bit in question. Since this proxy object is not a bool&, you can't assign its address to a bool* like you could with the result of such an operator call on a "normal" container. In turn this means that bool *pb =&v[0]; isn't valid code.
On the other hand deque doesn't have any such specialization called out so each bool takes a byte and you can take the address of the value return from operator[].
Finally note that the MS standard library implementation is (arguably) suboptimal in that it uses a small chunk size for deques, which means that using deque as a substitute isn't always the right answer.
The problems is that vector<bool> returns a proxy reference object instead of a true reference, so that C++98 style code bool * p = &v[0]; won't compile. However, modern C++11 with auto p = &v[0]; can be made to compile if operator& also returns a proxy pointer object. Howard Hinnant has written a blog post detailing the algorithmic improvements when using such proxy references and pointers.
Scott Meyers has a long Item 30 in More Effective C++ about proxy classes. You can come a long way to almost mimic the builtin types: for any given type T, a pair of proxies (e.g. reference_proxy<T> and iterator_proxy<T>) can be made mutually consistent in the sense that reference_proxy<T>::operator&() and iterator_proxy<T>::operator*() are each other's inverse.
However, at some point one needs to map the proxy objects back to behave like T* or T&. For iterator proxies, one can overload operator->() and access the template T's interface without reimplementing all the functionality. However, for reference proxies, you would need to overload operator.(), and that is not allowed in current C++ (although Sebastian Redl presented such a proposal on BoostCon 2013). You can make a verbose work-around like a .get() member inside the reference proxy, or implement all of T's interface inside the reference (this is what is done for vector<bool>::bit_reference), but this will either lose the builtin syntax or introduce user-defined conversions that do not have builtin semantics for type conversions (you can have at most one user-defined conversion per argument).
TL;DR: no vector<bool> is not a container because the Standard requires a real reference, but it can be made to behave almost like a container, at least much closer with C++11 (auto) than in C++98.
vector<bool> contains boolean values in compressed form using only one bit for value (and not 8 how bool[] arrays do). It is not possible to return a reference to a bit in c++, so there is a special helper type, "bit reference", which provides you a interface to some bit in memory and allows you to use standard operators and casts.
Many consider the vector<bool> specialization to be a mistake.
In a paper "Deprecating Vestigial Library Parts in C++17"
There is a proposal to
Reconsider vector Partial Specialization.
There has been a long history of the bool partial specialization of
std::vector not satisfying the container requirements, and in
particular, its iterators not satisfying the requirements of a random
access iterator. A previous attempt to deprecate this container was
rejected for C++11, N2204.
One of the reasons for rejection is that it is not clear what it would
mean to deprecate a particular specialization of a template. That
could be addressed with careful wording. The larger issue is that the
(packed) specialization of vector offers an important
optimization that clients of the standard library genuinely seek, but
would not longer be available. It is unlikely that we would be able to
deprecate this part of the standard until a replacement facility is
proposed and accepted, such as N2050. Unfortunately, there are no such
revised proposals currently being offered to the Library Evolution
Working Group.
Look at how it is implemented. the STL builds vastly on templates and therefore the headers do contain the code they do.
for instance look at the stdc++ implementation here.
also interesting even though not an stl conforming bit vector is the llvm::BitVector from here.
the essence of the llvm::BitVector is a nested class called reference and suitable operator overloading to make the BitVector behaves similar to vector with some limitations. The code below is a simplified interface to show how BitVector hides a class called reference to make the real implementation almost behave like a real array of bool without using 1 byte for each value.
class BitVector {
public:
class reference {
reference &operator=(reference t);
reference& operator=(bool t);
operator bool() const;
};
reference operator[](unsigned Idx);
bool operator[](unsigned Idx) const;
};
this code here has the nice properties:
BitVector b(10, false); // size 10, default false
BitVector::reference &x = b[5]; // that's what really happens
bool y = b[5]; // implicitly converted to bool
assert(b[5] == false); // converted to bool
assert(b[6] == b[7]); // bool operator==(const reference &, const reference &);
b[5] = true; // assignment on reference
assert(b[5] == true); // and actually it does work.
This code actually has a flaw, try to run:
std::for_each(&b[5], &b[6], some_func); // address of reference not an iterator
will not work because assert( (&b[5] - &b[3]) == (5 - 3) ); will fail (within llvm::BitVector)
this is the very simple llvm version. std::vector<bool> has also working iterators in it.
thus the call for(auto i = b.begin(), e = b.end(); i != e; ++i) will work. and also std::vector<bool>::const_iterator.
However there are still limitations in std::vector<bool> that makes it behave differently in some cases.
This comes from http://www.cplusplus.com/reference/vector/vector-bool/
Vector of bool This is a specialized version of vector, which is used
for elements of type bool and optimizes for space.
It behaves like the unspecialized version of vector, with the
following changes:
The storage is not necessarily an array of bool values, but the library implementation may optimize storage so that each value is
stored in a single bit.
Elements are not constructed using the allocator object, but their value is directly set on the proper bit in the internal storage.
Member function flip and a new signature for member swap.
A special member type, reference, a class that accesses individual bits in the container's internal storage with an interface that
emulates a bool reference. Conversely, member type const_reference is
a plain bool.
The pointer and iterator types used by the container are not necessarily neither pointers nor conforming iterators, although they
shall simulate most of their expected behavior.
These changes provide a quirky interface to this specialization and
favor memory optimization over processing (which may or may not suit
your needs). In any case, it is not possible to instantiate the
unspecialized template of vector for bool directly. Workarounds to
avoid this range from using a different type (char, unsigned char) or
container (like deque) to use wrapper types or further specialize for
specific allocator types.
bitset is a class that provides a similar functionality for fixed-size
arrays of bits.
I have recently seen a colleague of mine using std::string as a buffer:
std::string receive_data(const Receiver& receiver) {
std::string buff;
int size = receiver.size();
if (size > 0) {
buff.resize(size);
const char* dst_ptr = buff.data();
const char* src_ptr = receiver.data();
memcpy((char*) dst_ptr, src_ptr, size);
}
return buff;
}
I guess this guy wants to take advantage of auto destruction of the returned string so he needs not worry about freeing of the allocated buffer.
This looks a bit strange to me since according to cplusplus.com the data() method returns a const char* pointing to a buffer internally managed by the string:
const char* data() const noexcept;
Memcpy-ing to a const char pointer? AFAIK this does no harm as long as we know what we do, but have I missed something? Is this dangerous?
Don't use std::string as a buffer.
It is bad practice to use std::string as a buffer, for several reasons (listed in no particular order):
std::string was not intended for use as a buffer; you would need to double-check the description of the class to make sure there are no "gotchas" which would prevent certain usage patterns (or make them trigger undefined behavior).
As a concrete example: Before C++17, you can't even write through the pointer you get with data() - it's const Tchar *; so your code would cause undefined behavior. (But &(str[0]), &(str.front()), or &(*(str.begin())) would work.)
Using std::strings for buffers is confusing to readers of your function's definition, who assume you would be using std::string for, well, strings. In other words, doing so breaks the Principle of Least Astonishment.
Worse yet, it's confusing for whoever might use your function - they too may think what you're returning is a string, i.e. valid human-readable text.
std::unique_ptr would be fine for your case, or even std::vector. In C++17, you can use std::byte for the element type, too. A more sophisticated option is a class with an SSO-like feature, e.g. Boost's small_vector (thank you, #gast128, for mentioning it).
(Minor point:) libstdc++ had to change its ABI for std::string to conform to the C++11 standard, so in some cases (which by now are rather unlikely), you might run into some linkage or runtime issues that you wouldn't with a different type for your buffer.
Also, your code may make two instead of one heap allocations (implementation dependent): Once upon string construction and another when resize()ing. But that in itself is not really a reason to avoid std::string, since you can avoid the double allocation using the construction in #Jarod42's answer.
You can completely avoid a manual memcpy by calling the appropriate constructor:
std::string receive_data(const Receiver& receiver) {
return {receiver.data(), receiver.size()};
}
That even handles \0 in a string.
BTW, unless content is actually text, I would prefer std::vector<std::byte> (or equivalent).
Memcpy-ing to a const char pointer? AFAIK this does no harm as long as we know what we do, but is this good behavior and why?
The current code may have undefined behavior, depending on the C++ version. To avoid undefined behavior in C++14 and below take the address of the first element. It yields a non-const pointer:
buff.resize(size);
memcpy(&buff[0], &receiver[0], size);
I have recently seen a colleague of mine using std::string as a buffer...
That was somewhat common in older code, especially circa C++03. There are several benefits and downsides to using a string like that. Depending on what you are doing with the code, std::vector can be a bit anemic, and you sometimes used a string instead and accepted the extra overhead of char_traits.
For example, std::string is usually a faster container than std::vector on append, and you can't return std::vector from a function. (Or you could not do so in practice in C++98 because C++98 required the vector to be constructed in the function and copied out). Additionally, std::string allowed you to search with a richer assortment of member functions, like find_first_of and find_first_not_of. That was convenient when searching though arrays of bytes.
I think what you really want/need is SGI's Rope class, but it never made it into the STL. It looks like GCC's libstdc++ may provide it.
There a lengthy discussion about this being legal in C++14 and below:
const char* dst_ptr = buff.data();
const char* src_ptr = receiver.data();
memcpy((char*) dst_ptr, src_ptr, size);
I know for certain it is not safe in GCC. I once did something like this in some self tests and it resulted in a segfault:
std::string buff("A");
...
char* ptr = (char*)buff.data();
size_t len = buff.size();
ptr[0] ^= 1; // tamper with byte
bool tampered = HMAC(key, ptr, len, mac);
GCC put the single byte 'A' in register AL. The high 3-bytes were garbage, so the 32-bit register was 0xXXXXXX41. When I dereferenced at ptr[0], GCC dereferenced a garbage address 0xXXXXXX41.
The two take-aways for me were, don't write half-ass self tests, and don't try to make data() a non-const pointer.
From C++17, data can return a non const char *.
Draft n4659 declares at [string.accessors]:
const charT* c_str() const noexcept;
const charT* data() const noexcept;
....
charT* data() noexcept;
The code is unnecessary, considering that
std::string receive_data(const Receiver& receiver) {
std::string buff;
int size = receiver.size();
if (size > 0) {
buff.assign(receiver.data(), size);
}
return buff;
}
will do exactly the same.
The big optimization opportunity I would investigate here is: Receiver appears to be some kind of container that supports .data() and .size(). If you can consume it, and pass it in as a rvalue reference Receiver&&, you might be able to use move semantics without making any copies at all! If it’s got an iterator interface, you could use those for range-based constructors or std::move() from <algorithm>.
In C++17 (as Serge Ballesta and others have mentioned), std::string::data() returns a pointer to non-const data. A std::string has been guaranteed to store all its data contiguously for years.
The code as written smells a bit, although it’s not really the programmer’s fault: those hacks were necessary at the time. Today, you should at least change the type of dst_ptr from const char* to char* and remove the cast in the first argument to memcpy(). You could also reserve() a number of bytes for the buffer and then use a STL function to move the data.
As others have mentioned, a std::vector or std::unique_ptr would be a more natural data structure to use here.
One downside is performance.
The .resize method will default-initialize all the new byte locations to 0.
That initialization is unnecessary if you're then going to overwrite the 0s with other data.
I do feel std::string is a legitimate contender for managing a "buffer"; whether or not it's the best choice depends on a few things...
Is your buffer content textual or binary in nature?
One major input to your decision should be whether the buffer content is textual in nature. It will be less potentially confusing to readers of your code if std::string is used for textual content.
char is not a good type for storing bytes. Keep in mind that the C++ Standard leaves it up to each implementation to decide whether char will be signed or unsigned, but for generic black-box handling of binary data (and sometimes even when passing characters to functions like std::toupper(int) that have undefined behaviour unless the argument is in range for unsigned char or is equal to EOF) you probably want unsigned data: why would you assume or imply that the first bit of each byte is a sign bit if it's opaque binary data?
Because of that, it's undeniably somewhat hackish to use std::string for "binary" data. You could use std::basic_string<std::byte>, but that's not what the question asks about, and you'd lose some inoperability benefits from using the ubiquitous std::string type.
Some potential benefits of using std::string
Firstly a few benefits:
it sports the RAII semantics we all know and love
most implementations feature short-string optimisation (SSO), which ensures that if the number of bytes is small enough to fit directly inside the string object, dynamic allocation/deallocation can be avoided (but there may be an extra branch each time the data is accessed)
this is perhaps more useful for passing around copies of data read or to be written, rather than for a buffer which should be pre-sized to accept a decent chunk of data if available (to improve throughput by handling more I/O at a time)
there's a wealth of std::string member functions, and non-member functions designed to work well with std::strings (including e.g. cout << my_string): if your client code would find them useful to parse/manipulate/process the buffer content, then you're off to a flying start
the API is very familiar to most C++ programmers
Mixed blessings
being a familiar, ubiquitous type, the code you interact with may have specialisations to for std::string that better suit your use for buffered data, or those specialisations might be worse: do evaluate that
Concerned
As Waxrat observed, what is lacking API wise is the ability to grow the buffer efficiently, as resize() writes NULs/'\0's into the characters added which is pointless if you're about to "receive" values into that memory. This isn't relevant in the OPs code where a copy of received data is being made, and the size is already known.
Discussion
Addressing einpoklum's concern:
std::string was not intended for use as a buffer; you would need to double-check the description of the class to make sure there are no "gotchas" which would prevent certain usage patterns (or make them trigger undefined behavior).
While it's true that std::string wasn't originally intended for this, the rest is mainly FUD. The Standard has made concessions to this kind of usage with C++17's non-const member function char* data(), and string has always supported embedded zero bytes. Most advanced programmers know what's safe to do.
Alternatives
static buffers (C char[N] array or std::array<char, N>) sized to some maximum message size, or ferrying slices of the data per call
a manually allocated buffer with std::unique_ptr to automate destruction: leaves you to do fiddly resizing, and track the allocated vs in-use sizes yourself; more error-prone overall
std::vector (possibly of std::byte for the element type; is widely understood to imply binary data, but the API is more restrictive and (for better or worse) it can't be expected to have anything equivalent to Short-String Optimisation.
Boost's small_vector: perhaps, if SSO was the only thing holding you back from std::vector, and you're happy using boost.
returning a functor that allows lazy access to the received data (providing you know it won't be deallocated or overwritten), deferring the choice of how it's stored by client code
use C++23's string::resize_and_overwrite
https://en.cppreference.com/w/cpp/string/basic_string/resize_and_overwrite
https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2021/p1072r10.html
[[nodiscard]] static inline string formaterr (DWORD errcode) {
string strerr;
strerr.resize_and_overwrite(2048, [errcode](char* buf, size_t buflen) {
// https://learn.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-formatmessage
return FormatMessageA(
FORMAT_MESSAGE_FROM_SYSTEM | FORMAT_MESSAGE_IGNORE_INSERTS,
nullptr,
errcode,
0,
buf,
static_cast<DWORD>(buflen),
nullptr
);
});
return strerr;
}
It's well known that std::vector<bool> does not satisfy the Standard's container requirements, mainly because the packed representation prevents T* x = &v[i] from returning a pointer to a bool.
My question is: can this be remedied/mitigated when the reference_proxy overloads the address-of operator& to return a pointer_proxy?
The pointer-proxy could contain the same data as the reference_proxy in most implementations, namely a pointer into the packed data and a mask to isolate the particular bit inside the block pointed to. Indirection of the pointer_proxy would then yield the reference_proxy. Essentially both proxies are "fat" pointers, which are, however, still rather light-weight compared to disk-based proxy containers.
Instead of T* x = &v[0] one could then do auto x = &v[0], and use x like if(*x) without problems. I would also like to be able to write for(auto b: v) { /* ... */ }
Questions: would such a multi-proxy approach work with the STL's algorithms? Or do some algorithms really rely on the requirement that x needs to be a real bool*? Or are there too many consecutive user-defined conversions required that prevent this to work? I'd like to know any of such obstructions before trying to fully complete the above implementation sketch.
UPDATE (based on #HowardHinnant 's answer and this ancient discussion on comp.std.c++)
You can come a long way to almost mimic the builtin types: for any given type T, a pair of proxies (e.g. reference_proxy and iterator_proxy) can be made mutually consistent in the sense that reference_proxy::operator&() and iterator_proxy::operator*() are each other's inverse.
However, at some point one needs to map the proxy objects back to behave like T* or T&. For iterator proxies, one can overload operator->() and access the template T's interface without reimplementing all the functionality. However, for reference proxies, you would need to overload operator.(), and that is not allowed in current C++ (although Sebastian Redl presented such a proposal on BoostCon 2013). You can make a verbose work-around like a .get() member inside the reference proxy, or implement all of T's interface inside the reference (this is what is done for vector::bit_reference), but this will either lose the builtin syntax or introduce user-defined conversions that do not have builtin semantics for type conversions (you can have at most one user-defined conversion per argument).
My question is: can this be remedied/mitigated when the
reference_proxy overloads the address-of operator& to return a
pointer_proxy?
libc++ actually does this.
#include <vector>
#include <cassert>
int main()
{
std::vector<bool> v(1);
std::vector<bool>::pointer pb = &v[0];
assert(*pb == false);
*pb = true;
assert(v[0] == true);
std::vector<bool>::const_pointer cbp = pb;
assert(*cbp == true);
v[0] = false;
assert(*cbp == false);
}
It even extends to const_pointer and const_reference in ways that mimic the same types for vector<int>. This is a non-conforming extension for libc++. But it makes writing generic code which might be instantiated on vector<bool> far more likely to compile and behave correctly.
Questions: would such a multi-proxy approach work with the STL's
algorithms? Or do some algorithms really rely on the requirement that
x needs to be a real bool*? Or are there too many consecutive
user-defined conversions required that prevent this to work?
All of libc++'s algorithms work with vector<bool>. Some of them with quite spectacular performance. One algorithm in particular must have special treatment which the standard unfortunately does not mandate:
#include <vector>
#include <cassert>
int main()
{
std::vector<bool> v(1);
bool b = true;
assert(v[0] == false);
assert(b == true);
std::swap(b, v[0]);
assert(v[0] == true);
assert(b == false);
}
This is very easy for the implementation to accomplish. One simply needs to make sure swap works for any combination of bool and vector<bool>::reference. But I don't know if any implementation besides libc++ does this, and it is not mandated by C++11.
An array of bits is a wonderful data structure. But unfortunately it is poorly specified in the C++ standard. libc++ has gone somewhat outlaw to demonstrate that this can be a very useful and high performance data structure. The hope is that a future C++ standard may migrate in this direction to the benefit of the C++ programmer.
Offhand I would say first, that it will actually depend more on the particulars of each individual STL implementation since it doesn't officially conform to the standard requirement that a *reference_type to be lvalue of T*. So regarding potential implementation issues:
The main reason any piece of code would be explicitly dependent on the container's pointer being a real bool* is if the algo was using pointer arithmetic, in which case the size of the pointer type becomes relevant. Pointer arithmetic though would bypass the iterator interface and thus defeat the main purpose of the whole STL container-by-iterator design. std::vector<> itself is guaranteed to be contiguous in C++11, which allows optimized specializations of both STL algos and compiler for(:), both of which may use pointer arithmetic internally. If your type isn't derived from std::vector then that shouldn't be an issue; everything should just assume the iterator method instead.
However! STL code may still take pointers not for the purpose of pointer arithmetic but rather for some other purpose. In this case the problem is C++ syntax. Eg, quoting your own question:
Instead of T* x = &v[0] one could then do auto x = &v[0]
Any templated code in the STL will also have to do the same thing... and that seems entirely unlikely at this point that STL implementations will be making wide use of auto. There may be other situations were the STL attempts to do clever r-value casting tricks that end up failing because it isn't expecting mismatched reference types.
Regarding for(auto b: v) { /* ... */ }: I see no reason that shouldn't work. I think it will generate code that will be far less efficient than the same version you could just roll yourself in 15 mins (or less). I only bring it up since you mention intrinsics in the OP, which imples some consideration for performance. You won't be able to help it out using intrinsics either. There's nothing an intrinsic can do that somehow surpasses a simple bitwise shift for sequentially traversing an array of bits. Most of the added overhead will be from the compiler generating code to update the iterator pointer and mask values, and then reload the mask value on the next iteration. It won't be able to magically deduce what you're trying to do and turn it into a sequential shift op for you. It may at least be able to optimize out the pointer update+writeback stage by caching it into a register outside the loop, though honestly I'd be very skeptical based on my experiences.
Here's one way for going through bits from start to end, just for sake of comparison (a version capable of starting at any arbitrary point in the bitstream would require a little extra setup logic):
uint64_t* pBitSet = &v[-1]; // gets incremented on first iteration through loop.
uint64_t curBitSet = v[0];
for (int i=0; i<v.length(); ++i) {
if ((i % 64) == 0) {
curBitSet = *(++pBitSet);
}
int bit = curBitSet & 1;
curBitSet >>= 1;
// do stuff based on 'bit' here.
}
I need a common rule for warnings in x64 mode. Which way is better?
Consider the following lines of some code
const int N = std::max_element(cont.begin(), cont.end()) - cont.begin();
or
const int ARR_SIZE = 1024;
char arr[ARR_SIZE];
//...
const int N = std::max_element(arr, arr + ARR_SIZE) - arr;
It is my usual code. I have no problems with x86.
But if I run compiler in x64 mode I have some warnings:
conversion from 'std::_Array_iterator<_Ty,_Size>::difference_type' to 'int', possible loss of data
conversion from '__int64' to 'int', possible loss of data
I want to solve these problems by common rule. Which way is better?
Making static_cast:
const int N = static_cast<int>(
std::max_element(cont.begin(), cont.end()) - cont.begin() );
I think this is not general-purpose. And too much letters.
Replace output type with ptrdiff_t:
const ptrdiff_t N = std::max_element(cont.begin(), cont.end()) - cont.begin();
What should I do then with this unknown type ptrdiff_t?
I'll get another dozen warnings then.
I want to make many operations with N: save, addition, multiplication, cycles and etc.
Important: but what if std::_Array_iterator<_Ty,_Size>::difference_type and ptrdiff_t are different types?
Replace output type with std::_Array_iterator<_Ty,_Size>::difference_type:
//h-file
struct S {
type mymember; // What is the type?
};
//cpp-file
typedef std::vector<int> cont_t;
const cont_t::difference_type N = std::max_element(cont.begin(), cont.end()) - cont.begin();
// Save N
S mystruct;
mystruct.mymember = N; // What type of mystruct.mymember?
How should I save N? What type of mystruct.mymember? I don't know it in h-file.
Your solution.
"what if std::_Array_iterator<_Ty,_Size>::difference_type and ptrdiff_t are different types?" Don't use such a compiler. Also, chances are that it can't formally be different. E.g. this is the case for a vector using the default standard allocator, since that's where it fetches its typedefs, but since the formal guarantee doesn't matter (he he, it really doesn't) I'm not going to look this up in the C++0x draft.
So, use ptrdiff_t.
But it can be a good idea to add a few typedefs, like
typedef ptrdiff_t Size;
typedef ptrdiff_t Index;
and then in your concrete case you'd use Index.
These typedefs are naturally accompanied by custom freestanding countOf, startOf and endOf functions, enabling you to treat raw arrays and standard library containers in exactly the same way.
When you see the name Index it's a bit more clear that it's an index, which can't very naturally get out of the Index or Size set of types almost no matter what you do. E.g., add something to it, it's still an Index. So mostly there will not be a "another dozen warnings".
But in some rare case you'll need to get from Index to just int, say. In and in those rare cases just do a static_cast to shut up the compiler and make your intent clear. Or even a custom static_cast-like narrowTo operation, for expressiveness...
Cheers & hth.,
To keep result of max_element() - cont.begin() you should use
struct Foo { std::vector<int>::difference_type n; };
or
template<typename T> struct Foo { std::vector<T>::difference_type n; };
or
template<T> struct Foo { T n; };
Because difference_type is difference_type, and when you cast it to int you get undefined behavior.
You can use &*c.begin() to convert iterator to pointer, and use ptrdiff_t for difference of this pointers.
I'd use std::ptrdiff_t.
I can't think of a reasonable implementation where std::vector<T>::iterator::difference_type would not be assignable to a std::ptrdiff_t. They're almost certainly going to be the same. If they are not the same, the difference_type would have to be smaller than ptrdiff_t.
Also, ptrdiff_t is a signed type, so if all your code is designed to work with ints, you'll be better off than if you tried to use an unsigned type, like std::vector<int>::size_type.
In visual-studio-2010 you can write:
const auto N = std::max_element( cont.begin(), cont.end() ) - cont.begin();
Use std::vector<int>::size_type:
It is guaranteed to represent any non-negative value of difference_type
It's what all of vector's indexing operations accept
If cont is non-empty, std::max_element(cont.begin(), cont.end()) - cont.begin(); will not evaluate to a negative value. If it is empty, then you shouldn't be doing any of this processing anyhow.
What should I do then with this 'unknown' type? I'll get another dozen warnings then. I want to make many operations with N: save, addition, multiplication, cycles and etc.
You won't get any warnings if you use the type consistently and limit the usage to where you actually need it. N is an index into a vector; that's all it's good for. Any meaningful operation you perform on it is going to result in another possible index into a vector.
My solution is to use a type that is known to be large enough, based on the domain knowledge that I have but which may not be available to the compiler.
If the compiler then complains about a possible loss of data, I add a cast (which is guaranteed safe, because I know beforehand that the target type must be large enough).