I'm looking for an efficient way to hash a 6-byte field so that it can be used for std::unordered_map .
I think this would be the conventional way of creating a hash:
struct Hash {
std::size_t operator()(const std::array<uint8_t, 6> & mac) const {
std::size_t key = 0;
boost::hash_combine(key, mac[0]);
boost::hash_combine(key, mac[1]);
boost::hash_combine(key, mac[2]);
boost::hash_combine(key, mac[3]);
boost::hash_combine(key, mac[4]);
boost::hash_combine(key, mac[5]);
return key;
}
};
However I noticed that I can make it a little faster (~20%) using this trick:
struct Hash {
std::size_t operator()(const std::array<uint8_t, 6> & mac) const {
std::size_t key = 0;
// Possibly UB?
boost::hash_combine(key, reinterpret_cast<const uint32_t&>(mac[0]));
boost::hash_combine(key, reinterpret_cast<const uint16_t&>(mac[4]));
return key;
}
};
And this was even faster:
struct Hash {
std::size_t operator()(const std::array<uint8_t, 6> & mac) const {
// Requires size_t to be 64-bit.
static_assert(sizeof(std::size_t) >= 6, "MAC address doesn't fit in std::size_t!");
std::size_t key = 0;
// Likely UB?
boost::hash_combine(key, 0x0000FFFFFFFFFFFF & reinterpret_cast<const uint64_t&>(mac[0]));
return key;
}
};
My question is two-fold:
Are these optimizations going to result in UB?
Is my first solution the way to go? Or is there a better way?
Your optimizations are breaking the strict aliasing rules, which leads (standardly speaking) to undefined behavior.
The last optimization worries me the most since you are essentially reading memory you ought not to, which may provoke traps if this memory happened to be protected.
Any reason you are not using boost::hash_range ?
Since boost::hash_range turns out not to be as fast as required, I would propose another solution, based on aliasing. Or rather, two solutions in one.
The first idea is that aliasing can be subdued using char* as a temporary type.
size_t key = 0;
char* k = &reinterpret_cast<char*>(&key);
std::copy(mac.begin(), mac.end(), k);
return key;
is therefore a valid implementation of the hash.
However, we can go one step further. Because of alignment and padding, storing a char[6] and char[8] are likely to use the same amount of memory within a map node. Therefore, we could enrich the type, by using union:
union MacType {
unsigned char value[8];
size_t hash;
};
Now, you can encapsulate this properly within a class (and make sure you always initialize the bytes 7 and 8 to 0), and implement the interface of std::array<unsigned char, 6> that you actually need.
I've used a similar trick for tiny strings (below 8 characters) for hashing and fast (non-alphabetic) comparisons and it's really sweet.
Related
I've taken over some code, and came across a weird reallocation of an array. This is a function from within an Array class (used by the JsonValue)
void reserve( uint32_t newCapacity ) {
if ( newCapacity > length + additionalCapacity ) {
newCapacity = std::min( newCapacity, length + std::numeric_limits<decltype( additionalCapacity )>::max() );
JsonValue *newPtr = new JsonValue[newCapacity];
if ( length > 0 ) {
memcpy( newPtr, values, length * sizeof( JsonValue ) );
memset( values, 0, length * sizeof( JsonValue ) );
}
delete[] values;
values = newPtr;
additionalCapacity = uint16_t( newCapacity - length );
}
}
I get the point of this; it is just allocating a new array, and doing a copy of the memory contents from the old array into the new array, then zero-ing out the old array's contents. I also know this was done in order to prevent calling destructors, and moves.
The JsonValue is a class with functions, and some data which is stored in a union (string, array, number, etc.).
My concern is whether this is actually defined behaviour or not. I know it works, and has not had a problem since we began using it a few months ago; but if its undefined then it doesn't mean it is going to keep working.
EDIT:
JsonValue looks something like this:
struct JsonValue {
// …
~JsonValue() {
switch ( details.type ) {
case Type::Array:
case Type::Object:
array.destroy();
break;
case Type::String:
delete[] string.buffer;
break;
default: break;
}
}
private:
struct Details {
Key key = Key::Unknown;
Type type = Type::Null; // (0)
};
union {
Array array;
String string;
EmbedString embedString;
Number number;
Details details;
};
};
Where Array is a wrapper around an array of JsonValues, String is a char*, EmbedString is char[14], Number is a union of int, unsigned int, and double, Details contains the type of value it holds. All values have 16-bits of unused data at the beginning, which is used for Details. Example:
struct EmbedString {
uint16_t : 16;
char buffer[14] = { 0 };
};
Whether this code has well-defined behavior basically depends on two things: 1) is JsonValue trivially-copyable and, 2) if so, are a bunch of all-zero Bytes a valid object representation for a JsonValue.
If JsonValue is trivially-copyable, then the memcpy from one array of JsonValues to another will indeed be equivalent to copying all the elements over [basic.types]/3. If all-zeroes is a valid object representation for a JsonValue, then the memset should be ok (I believe this actually falls into a bit of a grey-area with the current wording of the standard, but I believe at least the intention would be that this is fine).
I'm not sure why you'd need to "prevent calling destructors and moves", but overwriting objects with zeroes does not prevent destructors from running. delete[] values will call the destructurs of the array members. And moving the elements of an array of trivially-copyable type should compile down to just copying over the bytes anyways.
Furthermore, I would suggest to get rid of these String and EmbedString classes and simply use std::string. At least, it would seem to me that the sole purpose of EmbedString is to manually perform small string optimization. Any std::string implementation worth its salt is already going to do exactly that under the hood. Note that std::string is not guaranteed (and will often not be) trivially-copyable. Thus, you cannot simply replace String and EmbedString with std::string while keeping the rest of this current implementation.
If you can use C++17, I would suggest to simply use std::variant instead of or at least inside this custom JsonValue implementation as that seems to be exactly what it's trying to do. If you need some common information stored in front of whatever the variant value may be, just have a suitable member holding that information in front of the member that holds the variant value rather than relying on every member of the union starting with the same couple of members (which would only be well-defined if all union members are standard-layout types that keep this information in their common initial sequence [class.mem]/23).
The sole purpose of Array would seem to be to serve as a vector that zeroes memory before deallocating it for security reasons. If this is the case, I would suggest to just use an std::vector with an allocator that zeros memory before deallocating instead. For example:
template <typename T>
struct ZeroingAllocator
{
using value_type = T;
T* allocate(std::size_t N)
{
return reinterpret_cast<T*>(new unsigned char[N * sizeof(T)]);
}
void deallocate(T* buffer, std::size_t N) noexcept
{
auto ptr = reinterpret_cast<volatile unsigned char*>(buffer);
std::fill(ptr, ptr + N, 0);
delete[] reinterpret_cast<unsigned char*>(buffer);
}
};
template <typename A, typename B>
bool operator ==(const ZeroingAllocator<A>&, const ZeroingAllocator<B>&) noexcept { return true; }
template <typename A, typename B>
bool operator !=(const ZeroingAllocator<A>&, const ZeroingAllocator<B>&) noexcept { return false; }
and then
using Array = std::vector<JsonValue, ZeroingAllocator<JsonValue>>;
Note: I fill the memory via volatile unsigned char* to prevent the compiler from optimizing away the zeroing. If you need to support overaligned types, you can replace the new[] and delete[] with direct calls to ::operator new and ::operator delete (doing this will prevent the compiler from optimizing away allocations). Pre C++17, you will have to allocate a sufficiently large buffer and then manually align the pointer, e.g., using std::align…
I have such expressions in my code:
QByteArray idx0 = ...
unsigned short ushortIdx0;
if ( idx0.size() >= sizeof(ushortIdx0) ) {
// Do something
}
But I'm getting the warning:
warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
if ( idx0.size() >= sizeof(ushortIdx0) ) {
~~~~~~~~~~~~^~~~~~~~~~
Why size() of QByteArray is returned as int rather than unsigned int? How can I get rid of this warning safely?
Some folk feel that the introduction of unsigned types into C all those years ago was a bad idea. Such types found themselves introduced into C++, where they are deeply embedded in the C++ Standard Library and operator return types.
Yes, sizeof must, by the standard, return an unsigned type.
The Qt developers adopt the modern thinking that the unsigned types were a bad idea, and favour instead making the return type of size a signed type. Personally I find it idiosyncratic.
To solve, you could (i) live with the warning, (ii) switch it off for the duration of the function, or (iii) write something like
(std::size_t)idx0.size() >= sizeof(ushortIdx0)
at the expense of clarity.
Why size() of QByteArray is returned as int rather than unsigned int?
I literally have no idea why Qt chose a signed return for size(). However, there are good reasons to use a signed instead of an unsigned.
One infamous example where a unsigned size() fails miserably is this quite innocent looking loop:
for (int i = 0; i < some_container.size() - 1; ++i) {
do_somehting(some_container[i] , some_container[i+1] );
}
Its not too uncommon to make the loop body operate on two elements and in that case its seems to be a valid choice to iterate only till some_container.size() - 1.
However, if the container is empty some_container.size() - 1 will silently (unsigned overflow is well defined) turn into the biggest value for the unsigned type. Hence, instead of avoiding the out-of-bounds access it leads to the maximum out of bounds you can get.
Note that there are easy fixes for this problem, but if size() does return a signed value, then there is no issue that needs to be fixed in the first place.
Because in Qt containers (like: QByteArray, QVector, ...) there are functions which can return a negative number, like: indexOf, lastIndexOf, contains, ... and some can accept negative numbers, like: mid, ...; So to be class-compatible or even framework-compatibe, developers use a signed type (int).
You can use standard c++ casting:
if ( static_cast<size_t>(idx0.size()) >= sizeof(ushortIdx0) )
The reason why is a duplicate part of the question, but the solution to the type mismatch is a valid problem to solve. For the comparisons of the kind you're doing, it'd probably be useful to factor them out, as they have a certain reusable meaning:
template <typename T> bool fitsIn(const QByteArray &a) {
return static_cast<int>(sizeof(T)) <= a.size();
}
template <typename T> bool fitsIn(T, const QByteArray &a) {
return fitsIn<T>(a);
}
if (fitsIn(ushortIdx0, idx0)) ...
Hopefully you'll have just a few kinds of such comparisons, and it'd make most sense to DRY (do not repeat yourself) and instead of a copypasta of casts, use functions dedicated to the task - functions that also express the intent of the original comparison. It then becomes easy to centralize handling of any corner cases you might wish to handle, i.e. when sizeof(T) > INT_MAX.
Another approach would be to define a new type to wrap size_t and adapt it to the types you need to use it with:
class size_of {
size_t val;
template <typename T> static typename std::enable_if<std::is_signed<T>::value, size_t>::type fromSigned(T sVal) {
return (sVal > 0) ? static_cast<size_t>(sVal) : 0;
}
public:
template <typename T, typename U = std::enable_if<std::is_scalar<T>::value>::type>
size_of(const T&) : val(sizeof(T)) {}
size_of(const QByteArray &a) : val(fromSigned(a.size())) {}
...
bool operator>=(size_of o) const { return value >= o.value; }
};
if (size_of(idx0) >= size_of(ushortIdx0)) ...
This would conceptually extend sizeof and specialize it for comparison(s) and nothing else.
Here's what I need to do. I'm sure it's a routine and recognizable coding task for many C++ developers out there:
void processAsUint16(const char* memory, size_t size) {
auto uint16_ptr = (const uint16_t*)memory;
for (size_t i = 0, n = size/sizeof(uint16_t); i < n; ++i) {
std::cout << uint16_ptr[i]; // Some processing of the other unrelated type
}
}
Problem: I'm developing with an IDE that integrates clang static code analysis, and every way of casting I tried, short of memcpy (which I don't want to resort to) is either discouraged or strongly discouraged. For example, reinterpret_cast is simply banned by the CPP Core Guidelines. C-style cast is discouraged. static_cast cannot be used here.
What's the right way of doing this that avoids type aliasing problems and other kinds of undefined behavior?
What's the right way of doing this that avoids type aliasing problems and other kinds of undefined behavior?
You use memcpy:
void processAsUint16(const char* memory, size_t size) {
for (size_t i = 0; i < size; i += sizeof(uint16_t)) {
uint16_t x;
memcpy(&x, memory + i, sizeof(x));
// do something with x
}
}
uint16_t is trivially copyable, so this is fine.
Or, in C++20, with std::bit_cast (which awkwardly has to go through an array first):
void processAsUint16(const char* memory, size_t size) {
for (size_t i = 0; i < size; i += sizeof(uint16_t)) {
alignas(uint16_t) char buf[sizeof(uint16_t)];
memcpy(buf, memory + i, sizeof(buf));
auto x = std::bit_cast<uint16_t>(buf);
// do something with x
}
}
Practically speaking, compilers will just "do the right thing" if you just reinterpret_cast, even if it's undefined behavior. Perhaps something like std::bless will give us a more direct, non-copying, mechanism of doing this, but until then...
My preference would be to treat the array of char as a sequence of octets in a defined order. This obviously doesn't work if it actually can be either order depending on target architecture, but in practise, a memory buffer like this usually comes from a file or a network connection.
void processAsUint16(const char* memory, size_t size) {
for (size_t i = 0; i < size; i += 2) {
const unsigned char lo = memory[i];
const unsigned char hi = memory[i+1];
const uint16_t x = lo + hi*256; // or "lo | hi << 8"
// do something with x
}
}
Note that we do not use sizeof(uint16_t) here. memory is a sequence of octets, so even if CHAR_BITS is 16, there will be two chars needed to hold a uint16_t.
This can be a little bit cleaner if memory can be declared as unsigned char - no need for the definition of lo and hi.
For RAM optimization purpose, I need to store my data as a std::array<char, N> where N is the "templated size_t" size of the array.
I need to manage a huge amount of "Line" objects that contain any kind of data (numerical and char).
So my Line class is:
template<size_t NByte>
class Line {
public:
Line (std::array<char, NByte> data, std::vector<size_t> offset) :
_data(data), _offset(offset) {}
template<typename T>
T getValue (const size_t& index) const {
return *reinterpret_cast<const T*>(_data.data() + _offset[index]);
}
template<typename T>
void setValue (const size_t& index, const T value) const {
char * new_value = const_cast<char *>(reinterpret_cast<const char *>(&value));
std::move(new_value, new_value + sizeof(T), const_cast<char *>(_data.data() + _offset[index]));
}
private:
std::array<char, NByte> _data;
std::vector<size_t> _offset;
};
My questions are:
Is there a better way to do the setter and getter function?
Is this robust against memory leak?
Is there any problem to use this code in production/release?
Edit: The question behind those is: Is there any other way to work with binary data in memory and provide "human understandable" interface for final user through setter and getter?
Is there any problem in using this code in production/release?
Yes, this code is platform-dependent.
The data will be stored differently in Big-Endian platforms and in Little-Endian platforms.
If your're counting on two systems communicating with each other (transmitting and receiving this data), then you will have to make sure that both sides use platforms of the same Endianness.
I have a lump of binary data in the form of const std::vector<unsigned char>, and want to be able to extract individual fields from that, such as 4 bytes for an integer, 1 for a boolean, etc. This needs to be, as far as possible, both efficient and simple. eg. It should be able to read the data in place without needing to copy it (eg. into a string or array). And it should be able to read one field at a time, like a parser, since the lump of data does not have a fixed format. I already know how to determine what type of field to read in each case - the problem is getting a usable interface on top of an std::vector for doing this.
However I can't find a simple way to get this data into an easily usable form that gives me useful read functionality. eg. std::basic_istringstream<unsigned char> gives me a reading interface, but it seems like I need to copy the data into a temporary std::basic_string<unsigned char> first, which is not idea for bigger blocks of data.
Maybe there is some way I can use a streambuf in this situation to read the data in place, but it would appear that I'd need to derive my own streambuf class to do that.
It occurs to me that I can probably just use sscanf on the vector's data(), and that would seem to be both more succinct and more efficient than the C++ standard library alternatives. EDIT: Having been reminded that sscanf doesn't do what I wrongly thought it did, I actually don't know a clean way to do this in C or C++. But am I missing something, and if so, what?
You have access to the data in a vector through its operator[]. A vector's data is guranteed to be stored in a single contiguous array, and [] returns a reference to a member of that array. You may use that reference directly, or through a memcpy.
std::vector<unsigned char> v;
...
byteField = v[12];
memcpy(&intField, &v[13], sizeof intField);
memcpy(charArray, &v[20], lengthOfCharArray);
EDIT 1:
If you want something "more convenient" that that, you could try:
template <class T>
ReadFromVector(T& t, std::size_t offset,
const std::vector<unsigned char>& v) {
memcpy(&t, &v[offset], sizeof(T));
}
Usage would be:
std::vector<unsigned char> v;
...
char c;
int i;
uint64_t ull;
ReadFromVector(c, 17, v);
ReadFromVector(i, 99, v);
ReadFromVector(ull, 43, v);
EDIT 2:
struct Reader {
const std::vector<unsigned char>& v;
std::size_t offset;
Reader(const std::vector<unsigned char>& v) : v(v), offset() {}
template <class T>
Reader& operator>>(T&t) {
memcpy(&t, &v[offset], sizeof t);
offset += sizeof t;
return *this;
}
void operator+=(int i) { offset += i };
char *getStringPointer() { return &v[offset]; }
};
Usage:
std::vector<unsigned char> v;
Reader r(v);
int i; uint64_t ull;
r >> i >> ull;
char *companyName = r.getStringPointer();
r += strlen(companyName);
If your vector stores binary data, you can't use sscanf or similar, they work on text.
For converting a byte for a bool is simple enough
bool b = my_vec[10];
For extracting an unsigned int that's stored in big endian order (assuming your ints are 32 bits):
unsigned int i = my_vec[10] << 24 | my_vec[11] << 16 | my_vec[12] << 8 | my_vec[13];
A 16 bit unsigned short would be similar:
unsigned short s = my_vec[10] << 8 | my_vec[11];¨
If you can afford the Qt dependency, QByteArray has the fromRawData() named constructor, which wraps existing data buffers in a QByteArray without copying the data. With that byte array, you can the feed a QTextStream.
I'm not aware of any such function in the standard streams library (short of implementing your own streambuf, of course), but I'd love to be proved wrong :)
You can use a struct that describes the data you are trying to extract. You can move data from your vector into the struct like this:
struct MyData {
int intVal;
bool boolVal;
char[15] stringVal;
} __attribute__((__packed__));
// assuming all extracted types are prefixed with a one byte indicator.
// Also assumes "vec" is your populated vector
int pos = 0;
while (pos < vec.size()-1) {
switch(vec[pos++]) {
case 0: { // handle int
int intValue;
memcpy(&vec[pos], &intValue, sizeof(int));
pos += sizeof(int);
// do something with handled value
break;
}
case 1: { // handle double
double doubleValue;
memcpy(&vec[pos], &doubleValue, sizeof(double));
pos += sizeof(double);
// do something with handled value
break;
}
case 2: { // handle MyData
struct MyData data;
memcpy(&vec[pos], &data, sizeof(struct MyData));
pos += sizeof(struct MyData);
// do something with handled value
break;
}
default: {
// ERROR: unknown type indicator
break;
}
}
}
Use a for loop to iterate over the vector and use bitwise operators to access each bit group. For example, to access the upper four bits of the first usigned char in your vector:
int myInt = vec[0] & 0xF0;
To read the fifth bit from the right, right after the chunk we just read:
bool myBool = vec[0] & 0x08;
The three least significant (lowest) bits can be accesed like so:
int myInt2 = vec[0] & 0x07;
You can then repeat this process (using a for loop) for every element in your vector.