Turning a string literal into a CRC32 constexpr inside a function - c++

I have an API that needs to be as simple as possible. At the same time, strings to certain functions are "well known", and so are always string literals, and can therefore be turned into CRC32 values at compile time, like so:
function(crc32("text"), ...);
The checksumming happens at compile-time using constexpr.
I want to simplify this API so that the people I work with don't have to know this pointless details. I want to make the compile-time checksumming happen inside the function, like so:
function("text", ...);
Doing the checksumming inside the inline function does not work because the function argument is no longer constexpr. The following code will fail to compile:
inline void function (const char* text, ...) {
constexpr uint32_t hash = crc32(text); // does not work
}
This is for an emulated environment so its crucial that the checksumming happens at compile-time. There are several functions that work like this, and they will all benefit from a simplification of the API.
What options do I have to hide the fact that you have to remember to use the call to crc32?
Using rustyx idea, I made this:
struct AutoCRC {
constexpr AutoCRC(const char* str) : hash{crc32(str)} {}
constexpr AutoCRC(const uint32_t h) : hash{h} {}
const uint32_t hash;
};
But it didn't work. Compared to using a constexpr CRC32 hash directly this almost doubled the binary size: 1488 -> 2696.
I have some limited C++20 access with GCC 9.2.

Note that even in case of function(crc32("text"), ...) the context of the crc32("text") call isn't constexpr and so it isn't guaranteed to happen at compile time. You need consteval for such a guarantee.
Similar to the macro solution, you could wrap the string literal in a helper "checksummer" class that stores the string and its checksum, and accept that as an argument. This way the checksumming happens before calling the function and a good compiler will do it at compile time for strings that are known at compile time.
For example:
#include <string_view>
constexpr uint32_t checksum(std::string_view sv) noexcept {
uint32_t sum = 0;
for (unsigned char c : sv)
sum += c;
return sum;
}
struct StrWithSum {
const char* str;
const uint32_t sum;
constexpr StrWithSum(const char* p) : str(p), sum(checksum(p)) {}
};
uint32_t do_something(StrWithSum str) {
return str.sum + 2;
}
int main() {
return do_something("ABC");
static_assert(StrWithSum("ABC").sum == 198, "sum");
}
We can see that the do_something function compiles down to the + 2 operation in both GCC and clang:
_Z12do_something10StrWithSum: # do_something
lea eax, [rsi + 2]
ret
And the entire program to
main: # #main
mov eax, 200
ret
(not the case in VC++, unfortunately).

Related

Why this __METHOD__NAME__ requires a memory copy?

The idea is from https://stackoverflow.com/a/15775519/1277762
I added a '\0' at end of string_view so we can use data() for printf, spdlog, et al.
I use this macro to print function name with class name.
However, I find that the compiler is not smart enough to inline the string, but requires a memory copy to stack first:
https://godbolt.org/z/bqob37G3z
See the difference between CALL and CALLFUNC in main function.
Is it possible to tell compiler just put the string in some RO section, like const char *?
template<std::size_t N>
consteval const std::array<char, N> __get_function_name(const char * arr)
{
std::array<char, N> data {};
std::string_view prettyFunction(arr);
size_t bracket = prettyFunction.rfind("(");
size_t space = prettyFunction.rfind(" ", bracket) + 1;
size_t i;
for (i = 0; i < bracket - space; i += 1) {
data[i] = arr[space + i];
}
data[i] = '\0';
return data;
}
#define __METHOD_NAME__ __get_function_name<strlen(__PRETTY_FUNCTION__)>(__PRETTY_FUNCTION__).data()
Thanks #n.m. and user17732522. I finally get a workable version:
https://godbolt.org/z/sof1j3Md4
Still not perfect as this solution needs search '(' and ' ' twice, which might increase compile time.
Updated: only call rfind(" ") once: https://godbolt.org/z/zYcajqaje
#include <array>
#include <string_view>
constexpr size_t my_func_end(const char * arr)
{
std::string_view prettyFunction(arr);
return prettyFunction.rfind("(");
}
constexpr size_t my_func_start(const char * arr, const size_t end)
{
std::string_view prettyFunction(arr);
return prettyFunction.rfind(" ", end) + 1;
}
template<std::size_t S, std::size_t E>
constexpr const std::array<char, E - S + 1> my_get_function_name(const char * arr)
{
std::array<char, E - S + 1> data {};
size_t i;
for ( i = 0; i < E - S; i += 1) {
data[i] = arr[S + i];
}
data[i] = '\0';
return data;
}
template<auto tofix>
struct fixme
{
static constexpr decltype(tofix) fixed = tofix;
};
#define __METHOD_NAME_ARRAY__(x) my_get_function_name<my_func_start(x, my_func_end(x)), my_func_end(x)>(x)
#define __METHOD_NAME__ (fixme<__METHOD_NAME_ARRAY__(__PRETTY_FUNCTION__)>::fixed.data())
Not sure if compiler can cache the calculation. If yes, it is good enough.
The compiler doesn't realize or take into account the specific behavior of puts, namely that it doesn't store the pointer it is passed and doesn't call its caller again.
The problem is that without this knowledge the compiler has to take into account the possibility that puts will store the pointer it is passed in e.g. a global variable, then calls its caller again, and then compares the new pointer argument with the old one stored in the global variable. These must compare unequal because they are pointers into different temporary objects, both in their lifetime. So the compiler can't reuse the same read-only static memory location as the argument to the puts call.
So you need to tell the compiler explicitly to use a static memory location:
#define METHOD_NAME get_function_name<my_strlen(__PRETTY_FUNCTION__)>(__PRETTY_FUNCTION__)
#define CALL() { static constexpr auto v = METHOD_NAME; puts(v.data()); }
Technically the constexpr on v is redundant with consteval on the function. Just constexpr on both would also be sufficient.
If you don't add constexpr on v you might want to add const though to make sure that the compiler won't need to consider the possibility that puts will modify the contents of the string, although it seems that GCC in your example is aware of that. (That puts takes a const char* as argument is not sufficient to establish this.)
You can't use strlen there by the way (assuming you want this to be portable to some other compiler beyond GCC that is supporting __PRETTY_FUNCTION__ in the way you are using it, e.g. Clang with libc++). That GCC is allowing it without diagnostic is not standard-conforming and it is not guaranteed to work on other compilers. std::strlen is not marked constexpr per standard. You can use std::char_traits<char>::length instead, which is constexpr since C++17.

What is the proper definition for a constexpr function that take a character array?

I'm writing a hashing function to help speed up string comparisons.
My codebase compares strings against a lot of const char[] constants, and it would be ideal if I could work with hashes instead. I went ahead and translated xxHash to modern C++, and I have a working prototype that does work at compile time, but I'm not sure what the function definition should be for the main hashing function.
At the moment, I have this:
template <size_t arr_size>
constexpr uint64_t xxHash64(const char(data)[arr_size])
{...}
This does work, and I am able to do a compile time call like this
constexpr char myString[] = "foobar";
constexpr uint64_t hashedString = xxHash64<sizeof myString>(myString);
[Find a minimal example here]
All good so far, but I would like to add a user-defined literal wrapper function for some eye candy, and this is where the problem lies.
UDLs come with a fixed prototype, as specified here
The Microsoft doc stipulates "Also, any of these operators can be defined as constexpr".
But when I try to call my hashing function from a constexpr UDL:
constexpr uint64_t operator "" _hashed(const char *arr, size_t size) {
return xxHash64<size>(arr);
}
function "xxHash64" cannot be called with the given argument list
argument types are: (const char*)
And the error does make sense. My function expects a character array, and instead it gets a pointer.
But if I were to modify the definition of my xxHash64 function to take a const char *, I can no longer work in a constexpr context because the compiler needs to resolve the pointer first, which happens at runtime.
So am I doing anything wrong here, or is this a limitation of UDLs or constexpr functions as a whole?
Again, I'm not 100% sure the templated definition at the top is the way to go, but I'm not sure how else I could read characters from a string at compile time.
I'm not limited by any compiler version or library. If there is a better way to do this, feel free to suggest.
there is no problem to call constexpr function with constexpr pointer as constant expression
constexpr uint64_t xxHash64(const char* s){return s[0];}
constexpr uint64_t operator "" _g(const char *arr,std::size_t){
return xxHash64(arr);
}
int main()
{
xxHash64("foo");
constexpr auto c = "foobar"_g;
return c;
}
would just work fine.
with c++20, you can also get the size as constant expression with string literal operator template.
#include <cstdint>
template <std::size_t arr_size>
constexpr std::uint64_t xxHash64(const char(&data)[arr_size]){
return data[0];
}
// template <std::size_t N> // can also be full class template (with CTAD)
struct hash_value{
std::uint64_t value;
template <std::size_t N>
constexpr hash_value(const char(&p)[N]):value(xxHash64(p)){}
};
template < hash_value v >
constexpr std::uint64_t operator ""_hashed() { return v.value; }
int main()
{
constexpr auto v = "foobar"_hashed;
return v;
}

Constexpr constants inside a constexpr function?

Suppose I have a static function which takes an enum and returns a cstring ptr for debugging.
The function can be constexpr but no guarantee is made that it can always be evaluated at compile time. Say it is running on a microcontroller and this is signalling events from a Bluetooth stack; e.g. device connected, disconnected, data in, etc, for context. So the parameter is not necessarily known at compile time.
Is there any value, meaning or difference in having the cstrings also defined as constexpr vs not?
A condensed example of what I mean:
#include <cstdio>
enum myEnum
{
EVENT1,
EVENT2,
EVENT3
};
static constexpr const char* foo(const myEnum i)
{
// Does having these as constexpr change anything?
/*constexpr*/ const char* text1 = "Text1";
/*constexpr*/ const char* text2 = "Text2";
/*constexpr*/ const char* text3 = "Text3";
/*constexpr*/ const char* textUndef = "TextUndef";
switch (i)
{
case EVENT1:
return text1;
case EVENT2:
return text2;
case EVENT3:
return text3;
default:
return textUndef;
}
}
int main()
{
const char* x = foo(EVENT1);
const char* y = foo(/*some value only known at runtime*/);
printf(x);
printf(y);
return 1;
}
I am compiling with gcc for cpp17 for an embedded microcontroller. Typically with either -Og or -Os.
Having the variables const char* declared constexpr in foo() doesn't add any value because you're not exploiting their constexpr-ness inside the function.
In main, you could declare x as constexpr in order to be sure that x's value is assigned at compile time. This could be useful because calling a constexpr function (even when passing literal values) doesn't guarantee that it will be evaluated at compile time. There's a catch though: you will not be able to modify the pointer anymore (x will become a const char *const, that is a const pointer to const char).
y is always evaluated at run time, so you cannot declare it constexpr.
My suggestion is to try your snippets in godbolt.org so that you can see the results with different optimizations options

Using CRC32 algorithm to hash string at compile-time

Basically I want in my code to be able to do this:
Engine.getById(WSID('some-id'));
Which should get transformed by
Engine.getById('1a61bc96');
just before being compiled into asm. So at compile-time.
This is my try
constexpr int WSID(const char* str) {
boost::crc_32_type result;
result.process_bytes(str,sizeof(str));
return result.checksum();
}
But I get this when trying to compile with MSVC 18 (CTP November 2013)
error C3249: illegal statement or sub-expression for 'constexpr' function
How can I get the WSID function, using this way or any, as long as it is done during compile time?
Tried this: Compile time string hashing
warning C4592: 'crc32': 'constexpr' call evaluation failed; function will be called at run-time
EDIT:
I first heard about this technique in Game Engine Architecture by Jason Gregory. I contacted the author who obligingly answer to me this :
What we do is to pass our source code through a custom little pre-processor that searches for text of the form SID('xxxxxx') and converts whatever is between the single quotes into its hashed equivalent as a hex literal (0xNNNNNNNN). [...]
You could conceivably do it via a macro and/or some template metaprogramming, too, although as you say it's tricky to get the compiler to do this kind of work for you. It's not impossible, but writing a custom tool is easier and much more flexible. [...]
Note also that we chose single quotes for SID('xxxx') literals. This was done so that we'd get some reasonable syntax highlighting in our code editors, yet if something went wrong and some un-preprocessed code ever made it thru to the compiler, it would throw a syntax error because single quotes are normally reserved for single-character literals.
Note also that it's crucial to have your little pre-processing tool cache the strings in a database of some sort, so that the original strings can be looked up given the hash code. When you are debugging your code and you inspect a StringId variable, the debugger will normally show you the rather unintelligible hash code. But with a SID database, you can write a plug-in that converts these hash codes back to their string equivalents. That way, you'll see SID('foo') in your watch window, not 0x75AE3080 [...]. Also, the game should be able to load this same database, so that it can print strings instead of hex hash codes on the screen for debugging purposes [...].
But while preprocess has some main advantages, it means that I have to prepare some kind of output system of modified files (those will be stored elsewhere, and then we need to tell MSVC). So it might complicate the compiling task. Is there a way to preprocess file with python for instance without headaches? But this is not the question, and I'm still interested about using compile-time function (about cache I could use an ID index)
Here is a solution that works entirely at compile time, but may also be used at runtime. It is a mix of constexpr, templates and macros. You may want to change some of the names or put them in a separate file since they are quite short.
Note that I reused code from this answer for the CRC table generation and I based myself off of code from this page for the implementation.
I have not tested it on MSVC since I don't currently have it installed in my Windows VM, but I believe it should work, or at least be made to work with trivial changes.
Here is the code, you may use the crc32 function directly, or the WSID function that more closely matches your question :
#include <cstring>
#include <cstdint>
#include <iostream>
// Generate CRC lookup table
template <unsigned c, int k = 8>
struct f : f<((c & 1) ? 0xedb88320 : 0) ^ (c >> 1), k - 1> {};
template <unsigned c> struct f<c, 0>{enum {value = c};};
#define A(x) B(x) B(x + 128)
#define B(x) C(x) C(x + 64)
#define C(x) D(x) D(x + 32)
#define D(x) E(x) E(x + 16)
#define E(x) F(x) F(x + 8)
#define F(x) G(x) G(x + 4)
#define G(x) H(x) H(x + 2)
#define H(x) I(x) I(x + 1)
#define I(x) f<x>::value ,
constexpr unsigned crc_table[] = { A(0) };
// Constexpr implementation and helpers
constexpr uint32_t crc32_impl(const uint8_t* p, size_t len, uint32_t crc) {
return len ?
crc32_impl(p+1,len-1,(crc>>8)^crc_table[(crc&0xFF)^*p])
: crc;
}
constexpr uint32_t crc32(const uint8_t* data, size_t length) {
return ~crc32_impl(data, length, ~0);
}
constexpr size_t strlen_c(const char* str) {
return *str ? 1+strlen_c(str+1) : 0;
}
constexpr int WSID(const char* str) {
return crc32((uint8_t*)str, strlen_c(str));
}
// Example usage
using namespace std;
int main() {
cout << "The CRC32 is: " << hex << WSID("some-id") << endl;
}
The first part takes care of generating the table of constants, while crc32_impl is a standard CRC32 implementation converted to a recursive style that works with a C++11 constexpr.
Then crc32 and WSID are just simple wrappers for convenience.
If anyone is interested, I coded up a CRC-32 table generator function and code generator function using C++14 style constexpr functions. The result is, in my opinion, much more maintainable code than many other attempts I have seen on the internet and it stays far, far away from the preprocessor.
Now, it does use a custom std::array 'clone' called cexp::array, because G++ seems to not have not added the constexpr keyword to their non-const reference index access/write operator.
However, it is quite light-weight, and hopefully the keyword will be added to std::array in the close future. But for now, the very simple array implementation is as follows:
namespace cexp
{
// Small implementation of std::array, needed until constexpr
// is added to the function 'reference operator[](size_type)'
template <typename T, std::size_t N>
struct array {
T m_data[N];
using value_type = T;
using reference = value_type &;
using const_reference = const value_type &;
using size_type = std::size_t;
// This is NOT constexpr in std::array until C++17
constexpr reference operator[](size_type i) noexcept {
return m_data[i];
}
constexpr const_reference operator[](size_type i) const noexcept {
return m_data[i];
}
constexpr size_type size() const noexcept {
return N;
}
};
}
Now, we need to generate the CRC-32 table. I based the algorithm off some Hacker's Delight code, and it can probably be extended to support the many other CRC algorithms out there. But alas, I only required the standard implementation, so here it is:
// Generates CRC-32 table, algorithm based from this link:
// http://www.hackersdelight.org/hdcodetxt/crc.c.txt
constexpr auto gen_crc32_table() {
constexpr auto num_bytes = 256;
constexpr auto num_iterations = 8;
constexpr auto polynomial = 0xEDB88320;
auto crc32_table = cexp::array<uint32_t, num_bytes>{};
for (auto byte = 0u; byte < num_bytes; ++byte) {
auto crc = byte;
for (auto i = 0; i < num_iterations; ++i) {
auto mask = -(crc & 1);
crc = (crc >> 1) ^ (polynomial & mask);
}
crc32_table[byte] = crc;
}
return crc32_table;
}
Next, we store the table in a global and perform rudimentary static checking on it. This checking could most likely be improved, and it is not necessary to store it in a global.
// Stores CRC-32 table and softly validates it.
static constexpr auto crc32_table = gen_crc32_table();
static_assert(
crc32_table.size() == 256 &&
crc32_table[1] == 0x77073096 &&
crc32_table[255] == 0x2D02EF8D,
"gen_crc32_table generated unexpected result."
);
Now that the table is generated, it's time to generate the CRC-32 codes. I again based the algorithm off the Hacker's Delight link, and at the moment it only supports input from a c-string.
// Generates CRC-32 code from null-terminated, c-string,
// algorithm based from this link:
// http://www.hackersdelight.org/hdcodetxt/crc.c.txt
constexpr auto crc32(const char *in) {
auto crc = 0xFFFFFFFFu;
for (auto i = 0u; auto c = in[i]; ++i) {
crc = crc32_table[(crc ^ c) & 0xFF] ^ (crc >> 8);
}
return ~crc;
}
For sake of completion, I generate one CRC-32 code below and statically check if it has the expected output, and then print it to the output stream.
int main() {
constexpr auto crc_code = crc32("some-id");
static_assert(crc_code == 0x1A61BC96, "crc32 generated unexpected result.");
std::cout << std::hex << crc_code << std::endl;
}
Hopefully this helps anyone else that was looking to achieve compile time generation of CRC-32, or even in general.
#tux3's answer is pretty slick! Hard to maintain, though, because you are basically writing your own implementation of CRC32 in preprocessor commands.
Another way to solve your question is to go back and understand the need for the requirement first. If I understand you right, the concern seems to be performance. In that case, there is a second point of time you can call your function without performance impact: at program load time. In that case, you would be accessing a global variable instead of passing a constant. Performance-wise, after initialization both should be identical (a const fetches 32 bits from your code, a global variable fetches 32 bits from a regular memory location).
You could do something like this:
static int myWSID = 0;
// don't call this directly
static int WSID(const char* str) {
boost::crc_32_type result;
result.process_bytes(str,sizeof(str));
return result.checksum();
}
// Put this early into your program into the
// initialization code.
...
myWSID = WSID('some-id');
Depending on your overall program, you may want to have an inline accessor to retrieve the value.
If a minor performance impact is acceptable, you would also write your function like this, basically using the singleton pattern.
// don't call this directly
int WSID(const char* str) {
boost::crc_32_type result;
result.process_bytes(str,sizeof(str));
return result.checksum();
}
// call this instead. Note the hard-coded ID string.
// Create one such function for each ID you need to
// have available.
static int myWSID() {
// Note: not thread safe!
static int computedId = 0;
if (computedId == 0)
computedId = WSID('some-id');
return computedId;
}
Of course, if the reason for asking for compile-time evaluation is something different (such as, not wanting some-id to appear in the compiled code), these techniques won't help.
The other option is to use Jason Gregory's suggestion of a custom preprocessor. It can be done fairly cleanly if you collect all the IDS into a separate file. This file doesn't need to have C syntax. I'd give it an extension such as .wsid. The custom preprocessor generates a .H file from it.
Here is how this could look:
idcollection.wsid (before custom preprocessor):
some_id1
some_id2
some_id3
Your preprocessor would generate the following idcollection.h:
#define WSID_some_id1 0xabcdef12
#define WSID_some_id2 0xbcdef123
#define WSID_some_id3 0xcdef1234
And in your code, you'd call
Engine.getById(WSID_some_id1);
A few notes about this:
This assumes that all the original IDs can be converted into valid identifiers. If they contain special characters, your preprocessor may need to do additional munging.
I notice a mismatch in your original question. Your function returns an int, but Engine.getById seems to take a string. My proposed code would always use int (easy to change if you want always string).

How to have "constexpr and runtime" alias

Constexpr can be awsome and useful for compilation optimisation. For example...
strlen(char*)
Can be precompiled using....
constexpr inline size_t strlen_constexpr(char* baseChar) {
return (
( baseChar[0] == 0 )
?(// if {
0
)// }
:(// else {
strlen_constexpr( baseChar+1 ) + 1
)// }
);
}
Which gives it a runtime cost of "0" when optimised... But is more than 10+x slower on runtime
// Test results ran on a 2010 macbook air
--------- strlen ---------
Time took for 100,000 runs:1054us.
Avg Time took for 1 run: 0.01054us.
--------- strlen_constexpr ---------
Time took for 100,000 runs:19098us.
Avg Time took for 1 run: 0.19098us.
Are there any existing macro / template hack where a single unified function can be used instead. ie.
constexpr size_t strlen_smart(char* baseChar) {
#if constexpr
... constexpr function
#else its runtime
... runtime function
}
Or some overloading hack that would allow the following
constexpr size_t strlen_smart(char* baseChar) {
... constexpr function
}
inline size_t strlen_smart(char* baseChar) {
... runtime function
}
Note: This question applies to the concept in general. Of having 2 separate functions for runtime and constexpr instead of the example functions given.
Disclaimer: Setting the compiler to -O3 (optimization level) is more than enough to fix 99.9% of static char optimizations making all the examples above "pointless". But that's beside the point of this question, as it applies to other "examples", and not just strlen.
I don't know any generic way, but I know two specific cases where it is possible.
Specific case of some compilers
Also gcc, and clang which copies all features of gcc, have a built-in function __builtin_constant_p. I am not sure whether gcc will correctly see argument to inline function as constant, but I fear you'd have to use it from a macro:
#define strlen_smart(s) \
(__builtin_constant_p(s) && __builtin_constant_p(*s) ? \
strlen_constexpr(s) : \
strlen(s))
Might be of use. Note that I am testing both s and *s for constexpr, because pointer to static buffer is a compile time constant while it's length is not.
Bonus: Specific case of literals (not an actual answer)
For the specific cast of strlen you can use the fact that string literals are not of type const char * but of type const char[N] that implicitly converts to const char *. But it also converts to const char (&)[N] as well while const char * does not.
So you can define:
template <size_t N>
constexpr size_t strlen_smart(const char (&array)[N])
(plus obviously strlen_smart on const char * forwards to strlen)
I've sometimes used function with this type of argument even in C++98 with definition corresponding to (I didn't try to overload strlen itself, but the overloads were so I could avoid calling it):
template <size_t N>
size_t strlen_smart(const char (&)[N]) { return N - 1; }
This has the problem that for
char buffer[10] = { 0 };
strlen_smart(buffer);
should say 0, but that optimized variant just says 9. The functions don't make sense to be called on buffers like that so I didn't care.