Basically I want in my code to be able to do this:
Engine.getById(WSID('some-id'));
Which should get transformed by
Engine.getById('1a61bc96');
just before being compiled into asm. So at compile-time.
This is my try
constexpr int WSID(const char* str) {
boost::crc_32_type result;
result.process_bytes(str,sizeof(str));
return result.checksum();
}
But I get this when trying to compile with MSVC 18 (CTP November 2013)
error C3249: illegal statement or sub-expression for 'constexpr' function
How can I get the WSID function, using this way or any, as long as it is done during compile time?
Tried this: Compile time string hashing
warning C4592: 'crc32': 'constexpr' call evaluation failed; function will be called at run-time
EDIT:
I first heard about this technique in Game Engine Architecture by Jason Gregory. I contacted the author who obligingly answer to me this :
What we do is to pass our source code through a custom little pre-processor that searches for text of the form SID('xxxxxx') and converts whatever is between the single quotes into its hashed equivalent as a hex literal (0xNNNNNNNN). [...]
You could conceivably do it via a macro and/or some template metaprogramming, too, although as you say it's tricky to get the compiler to do this kind of work for you. It's not impossible, but writing a custom tool is easier and much more flexible. [...]
Note also that we chose single quotes for SID('xxxx') literals. This was done so that we'd get some reasonable syntax highlighting in our code editors, yet if something went wrong and some un-preprocessed code ever made it thru to the compiler, it would throw a syntax error because single quotes are normally reserved for single-character literals.
Note also that it's crucial to have your little pre-processing tool cache the strings in a database of some sort, so that the original strings can be looked up given the hash code. When you are debugging your code and you inspect a StringId variable, the debugger will normally show you the rather unintelligible hash code. But with a SID database, you can write a plug-in that converts these hash codes back to their string equivalents. That way, you'll see SID('foo') in your watch window, not 0x75AE3080 [...]. Also, the game should be able to load this same database, so that it can print strings instead of hex hash codes on the screen for debugging purposes [...].
But while preprocess has some main advantages, it means that I have to prepare some kind of output system of modified files (those will be stored elsewhere, and then we need to tell MSVC). So it might complicate the compiling task. Is there a way to preprocess file with python for instance without headaches? But this is not the question, and I'm still interested about using compile-time function (about cache I could use an ID index)
Here is a solution that works entirely at compile time, but may also be used at runtime. It is a mix of constexpr, templates and macros. You may want to change some of the names or put them in a separate file since they are quite short.
Note that I reused code from this answer for the CRC table generation and I based myself off of code from this page for the implementation.
I have not tested it on MSVC since I don't currently have it installed in my Windows VM, but I believe it should work, or at least be made to work with trivial changes.
Here is the code, you may use the crc32 function directly, or the WSID function that more closely matches your question :
#include <cstring>
#include <cstdint>
#include <iostream>
// Generate CRC lookup table
template <unsigned c, int k = 8>
struct f : f<((c & 1) ? 0xedb88320 : 0) ^ (c >> 1), k - 1> {};
template <unsigned c> struct f<c, 0>{enum {value = c};};
#define A(x) B(x) B(x + 128)
#define B(x) C(x) C(x + 64)
#define C(x) D(x) D(x + 32)
#define D(x) E(x) E(x + 16)
#define E(x) F(x) F(x + 8)
#define F(x) G(x) G(x + 4)
#define G(x) H(x) H(x + 2)
#define H(x) I(x) I(x + 1)
#define I(x) f<x>::value ,
constexpr unsigned crc_table[] = { A(0) };
// Constexpr implementation and helpers
constexpr uint32_t crc32_impl(const uint8_t* p, size_t len, uint32_t crc) {
return len ?
crc32_impl(p+1,len-1,(crc>>8)^crc_table[(crc&0xFF)^*p])
: crc;
}
constexpr uint32_t crc32(const uint8_t* data, size_t length) {
return ~crc32_impl(data, length, ~0);
}
constexpr size_t strlen_c(const char* str) {
return *str ? 1+strlen_c(str+1) : 0;
}
constexpr int WSID(const char* str) {
return crc32((uint8_t*)str, strlen_c(str));
}
// Example usage
using namespace std;
int main() {
cout << "The CRC32 is: " << hex << WSID("some-id") << endl;
}
The first part takes care of generating the table of constants, while crc32_impl is a standard CRC32 implementation converted to a recursive style that works with a C++11 constexpr.
Then crc32 and WSID are just simple wrappers for convenience.
If anyone is interested, I coded up a CRC-32 table generator function and code generator function using C++14 style constexpr functions. The result is, in my opinion, much more maintainable code than many other attempts I have seen on the internet and it stays far, far away from the preprocessor.
Now, it does use a custom std::array 'clone' called cexp::array, because G++ seems to not have not added the constexpr keyword to their non-const reference index access/write operator.
However, it is quite light-weight, and hopefully the keyword will be added to std::array in the close future. But for now, the very simple array implementation is as follows:
namespace cexp
{
// Small implementation of std::array, needed until constexpr
// is added to the function 'reference operator[](size_type)'
template <typename T, std::size_t N>
struct array {
T m_data[N];
using value_type = T;
using reference = value_type &;
using const_reference = const value_type &;
using size_type = std::size_t;
// This is NOT constexpr in std::array until C++17
constexpr reference operator[](size_type i) noexcept {
return m_data[i];
}
constexpr const_reference operator[](size_type i) const noexcept {
return m_data[i];
}
constexpr size_type size() const noexcept {
return N;
}
};
}
Now, we need to generate the CRC-32 table. I based the algorithm off some Hacker's Delight code, and it can probably be extended to support the many other CRC algorithms out there. But alas, I only required the standard implementation, so here it is:
// Generates CRC-32 table, algorithm based from this link:
// http://www.hackersdelight.org/hdcodetxt/crc.c.txt
constexpr auto gen_crc32_table() {
constexpr auto num_bytes = 256;
constexpr auto num_iterations = 8;
constexpr auto polynomial = 0xEDB88320;
auto crc32_table = cexp::array<uint32_t, num_bytes>{};
for (auto byte = 0u; byte < num_bytes; ++byte) {
auto crc = byte;
for (auto i = 0; i < num_iterations; ++i) {
auto mask = -(crc & 1);
crc = (crc >> 1) ^ (polynomial & mask);
}
crc32_table[byte] = crc;
}
return crc32_table;
}
Next, we store the table in a global and perform rudimentary static checking on it. This checking could most likely be improved, and it is not necessary to store it in a global.
// Stores CRC-32 table and softly validates it.
static constexpr auto crc32_table = gen_crc32_table();
static_assert(
crc32_table.size() == 256 &&
crc32_table[1] == 0x77073096 &&
crc32_table[255] == 0x2D02EF8D,
"gen_crc32_table generated unexpected result."
);
Now that the table is generated, it's time to generate the CRC-32 codes. I again based the algorithm off the Hacker's Delight link, and at the moment it only supports input from a c-string.
// Generates CRC-32 code from null-terminated, c-string,
// algorithm based from this link:
// http://www.hackersdelight.org/hdcodetxt/crc.c.txt
constexpr auto crc32(const char *in) {
auto crc = 0xFFFFFFFFu;
for (auto i = 0u; auto c = in[i]; ++i) {
crc = crc32_table[(crc ^ c) & 0xFF] ^ (crc >> 8);
}
return ~crc;
}
For sake of completion, I generate one CRC-32 code below and statically check if it has the expected output, and then print it to the output stream.
int main() {
constexpr auto crc_code = crc32("some-id");
static_assert(crc_code == 0x1A61BC96, "crc32 generated unexpected result.");
std::cout << std::hex << crc_code << std::endl;
}
Hopefully this helps anyone else that was looking to achieve compile time generation of CRC-32, or even in general.
#tux3's answer is pretty slick! Hard to maintain, though, because you are basically writing your own implementation of CRC32 in preprocessor commands.
Another way to solve your question is to go back and understand the need for the requirement first. If I understand you right, the concern seems to be performance. In that case, there is a second point of time you can call your function without performance impact: at program load time. In that case, you would be accessing a global variable instead of passing a constant. Performance-wise, after initialization both should be identical (a const fetches 32 bits from your code, a global variable fetches 32 bits from a regular memory location).
You could do something like this:
static int myWSID = 0;
// don't call this directly
static int WSID(const char* str) {
boost::crc_32_type result;
result.process_bytes(str,sizeof(str));
return result.checksum();
}
// Put this early into your program into the
// initialization code.
...
myWSID = WSID('some-id');
Depending on your overall program, you may want to have an inline accessor to retrieve the value.
If a minor performance impact is acceptable, you would also write your function like this, basically using the singleton pattern.
// don't call this directly
int WSID(const char* str) {
boost::crc_32_type result;
result.process_bytes(str,sizeof(str));
return result.checksum();
}
// call this instead. Note the hard-coded ID string.
// Create one such function for each ID you need to
// have available.
static int myWSID() {
// Note: not thread safe!
static int computedId = 0;
if (computedId == 0)
computedId = WSID('some-id');
return computedId;
}
Of course, if the reason for asking for compile-time evaluation is something different (such as, not wanting some-id to appear in the compiled code), these techniques won't help.
The other option is to use Jason Gregory's suggestion of a custom preprocessor. It can be done fairly cleanly if you collect all the IDS into a separate file. This file doesn't need to have C syntax. I'd give it an extension such as .wsid. The custom preprocessor generates a .H file from it.
Here is how this could look:
idcollection.wsid (before custom preprocessor):
some_id1
some_id2
some_id3
Your preprocessor would generate the following idcollection.h:
#define WSID_some_id1 0xabcdef12
#define WSID_some_id2 0xbcdef123
#define WSID_some_id3 0xcdef1234
And in your code, you'd call
Engine.getById(WSID_some_id1);
A few notes about this:
This assumes that all the original IDs can be converted into valid identifiers. If they contain special characters, your preprocessor may need to do additional munging.
I notice a mismatch in your original question. Your function returns an int, but Engine.getById seems to take a string. My proposed code would always use int (easy to change if you want always string).
Related
I have an API that needs to be as simple as possible. At the same time, strings to certain functions are "well known", and so are always string literals, and can therefore be turned into CRC32 values at compile time, like so:
function(crc32("text"), ...);
The checksumming happens at compile-time using constexpr.
I want to simplify this API so that the people I work with don't have to know this pointless details. I want to make the compile-time checksumming happen inside the function, like so:
function("text", ...);
Doing the checksumming inside the inline function does not work because the function argument is no longer constexpr. The following code will fail to compile:
inline void function (const char* text, ...) {
constexpr uint32_t hash = crc32(text); // does not work
}
This is for an emulated environment so its crucial that the checksumming happens at compile-time. There are several functions that work like this, and they will all benefit from a simplification of the API.
What options do I have to hide the fact that you have to remember to use the call to crc32?
Using rustyx idea, I made this:
struct AutoCRC {
constexpr AutoCRC(const char* str) : hash{crc32(str)} {}
constexpr AutoCRC(const uint32_t h) : hash{h} {}
const uint32_t hash;
};
But it didn't work. Compared to using a constexpr CRC32 hash directly this almost doubled the binary size: 1488 -> 2696.
I have some limited C++20 access with GCC 9.2.
Note that even in case of function(crc32("text"), ...) the context of the crc32("text") call isn't constexpr and so it isn't guaranteed to happen at compile time. You need consteval for such a guarantee.
Similar to the macro solution, you could wrap the string literal in a helper "checksummer" class that stores the string and its checksum, and accept that as an argument. This way the checksumming happens before calling the function and a good compiler will do it at compile time for strings that are known at compile time.
For example:
#include <string_view>
constexpr uint32_t checksum(std::string_view sv) noexcept {
uint32_t sum = 0;
for (unsigned char c : sv)
sum += c;
return sum;
}
struct StrWithSum {
const char* str;
const uint32_t sum;
constexpr StrWithSum(const char* p) : str(p), sum(checksum(p)) {}
};
uint32_t do_something(StrWithSum str) {
return str.sum + 2;
}
int main() {
return do_something("ABC");
static_assert(StrWithSum("ABC").sum == 198, "sum");
}
We can see that the do_something function compiles down to the + 2 operation in both GCC and clang:
_Z12do_something10StrWithSum: # do_something
lea eax, [rsi + 2]
ret
And the entire program to
main: # #main
mov eax, 200
ret
(not the case in VC++, unfortunately).
I have a function which executes a bunch of tests. Whenever a new test is created, the function gets one or two more lines. And - the result is pushed back into an array. So it goes something like this (simplified):
void foo(int *results) {
auto index { 0 };
results[i++] = test_1(some, args, here);
results[i++] = test_1(some, other_args, here);
results[i++] = test_2(some, args, here);
results[i++] = test_3(some, args, here);
// etc. etc.
}
void bar() {
auto results = new int/* magic */];
foo(results);
}
I want to use the number of statements in this function to allocate space for the results (the line in bar()). I cannot use a dynamically-reallocated structure like an std::vector or a list etc. - since I am precluded from allocating any memory due to hardware restrictions.
Now, I could just manually count the lines - and this would work. But then whenever I add another test I would have to remember to update the magical constant.
Is there some way to do the counting with the result usable for the "magic" expression?
Note: Since I'm a scrupulous man with no dignity, I am willing to stoop to the use of macros.
Speaking of macro hackery:
#include <iostream>
#define ADD_TEST(X) do { results[i++] = (X); (void)__COUNTER__; } while (0)
const int foo_start = __COUNTER__;
void foo(int *results) {
int i = 0;
ADD_TEST(100);
ADD_TEST(200);
ADD_TEST(300);
}
const int foo_end = __COUNTER__;
int main() {
int results[foo_end - foo_start - 1];
foo(results);
for (int i : results) {
std::cout << i << '\n';
}
}
It's slightly awful and __COUNTER__ is a non-standard extension in GCC and other compilers, but hey, it works.
The advantage is that it doesn't use any fancy C++ features, so in principle it should be compatible with older compilers and even C.
As you haven't specified any language version, though, did tag it with constexpr, I've solved this making use of C++17. This without any dirty macros. Instead, I'm relying on CTAD (Constructor template argument deduction).
First of all, I've assumed your functions are constexpr. That way, everything can be done at compile-time. (In the resulting code, you don't even see memory being used for the array.
constexpr int test_1(int a, int b, int c)
{
return a + b + c;
}
constexpr int test_2(int a, int b, int c)
{
return a * b * c;
}
This isn't strictly needed, however, it can move unneeded calculations to compile time. It also allows propagating constexpr upto the final variable. That way, you could guarantee that none of the calculations will happen at run-time.
static constexpr auto myArr = createFilledArray();
However, the most important part is CTAD. A new C++17 feature that allows deducing the template arguments of your class based on the values that are passed at the constructor.
Instead of first creating an array, I create the array directly with all the different values that you pass to it. Since you haven't provided any arguments in your example, I assume they are known at compile time, which is again required for the constexpr waterfall. However, more importantly, I assume the number of elements is known at compile time.
By constructing all arguments when calling the constructor of std::array, there is no need for specifying its template arguments (note also the auto as return type). This gets deduced as std::array<int, 3> for this example.
constexpr auto createFilledArray(){
std::array a
{
test_1(1, 2, 3),
test_1(4, 5, 6),
test_2(7, 8, 9),
};
return a;
}
int main(int, char**)
{
return myArr.size(); // Returns 3
}
Code at compiler explorer
From what I'm aware, there is a proposal for C++20 that is intended to make std::vector constexpr. However, none of the compilers I've tested at compiler explorer support this. This will most likely allow you to write code based on std::vector and use that at compile time. In other words, the allocated memory that represents your data, will be part of your executable.
A quick attempt of what your code could look like can be found here at compiler explorer. (However, it ain't compiling at this point)
I'm working on a huge project which uses a system that, when running make, generates some header files that contain some constants that are used everywhere in the code. Due to the size of this project if one of those headers is changed (a constant is removed or another is added) almost the whole project must be recompiled (which sometimes takes several hours).
My initial solution was to write some sort of ConstantManager class that has a map with each code-value pair and a getter that returns that given the code as a string returns its value (pretty straight forward) and also change the header generator so it would define the constants as #defines that expand to an instance of the ConstantManager and a call to the getter. The problem with this is that it will not work with switch statements (well... case statements actually) because the return values are not constant expressions.
My question is: are there any alternative solutions to this problem or some trick to make mine work with switches?
You could split the huge header into smaller ones and include those. This might be a lot of initial work but is the most straight forward and probably compatible with your current solution.
Another option is to create to make your ConstManager class have constexpr members. For this you would not need much but you cannot use map.
ConstManager.h
namespace ConstManager {
namespace detail {
struct Constant {
char const * const name;
char const * const value;
};
Constant const * const constants;
unsigned int total_variables;
}
inline char const * const ConstManager::get(char const * needle) constexpr {
using namespace ConstManager::detail;
/* i was not able to quickly find a constexpr compatible binary search function*/
if(int i = 0; i < total_variables; ++i){
Constant & const c = constants[i];
if(strcmp(c.name, needle) == 0){
return c.value;
}
}
return nullptr;
}
}
ConstManager.c should be generated
ConstManager::detail::Constant ConstManager::detail::constants [] = {
{"first var", "first value"},
{"second var", "second value"}
};
unsigned int ConstManager::detail::total_variables = 2;
First solution proposed (this was before the first edit):
I think you should replace all your defines by regular external variables.
Although two caveats:
It will not work if those defines are used to concatenate strings
It might make inlining harder if not impossible for the compiler but if this matters only a profiler can tell.
constmgr.h
// make all your needed constansts externs
namespace Constants {
extern int const theFirstConstant;
extern char const * const someStringConstant;
}
And the source file is then generated at build time.
constexpr.cpp
int const Constants::theFirstConstant = 1000;
char const * const Constants::someStringConstant = "hihihoho";
Let's say that I need to create a LUT containing precomputed bit count values (count of 1 bits in a number) for 0...255 values:
int CB_LUT[256] = {0, 1, 1, 2, ... 7, 8};
If I don't want to use hard-coded values, I can use nice template solution How to count the number of set bits in a 32-bit integer?
template <int BITS>
int CountBits(int val)
{
return (val & 0x1) + CountBits<BITS-1>(val >> 1);
}
template<>
int CountBits<1>(int val)
{
return val & 0x1;
}
int CB_LUT[256] = {CountBits<8>(0), CountBits<8>(1) ... CountBits<8>(255)};
This array is computed completely at compile time. Is there any way to avoid a long list, and generate such array using some kind of templates or even macros (sorry!), something like:
Generate(CB_LUT, 0, 255); // array declaration
...
cout << CB_LUT[255]; // should print 8
Notes. This question is not about counting 1 bits in an number, it is used just as example. I want to generate such array completely in the code, without using external code generators. Array must be generated at compile time.
Edit.
To overcome compiler limits, I found the following solution, based on
Bartek Banachewicz` code:
#define MACRO(z,n,text) CountBits<8>(n)
int CB_LUT[] = {
BOOST_PP_ENUM(128, MACRO, _)
};
#undef MACRO
#define MACRO(z,n,text) CountBits<8>(n+128)
int CB_LUT2[] = {
BOOST_PP_ENUM(128, MACRO, _)
};
#undef MACRO
for(int i = 0; i < 256; ++i) // use only CB_LUT
{
cout << CB_LUT[i] << endl;
}
I know that this is possibly UB...
It would be fairly easy with macros using (recently re-discovered by me for my code) Boost.Preprocessor - I am not sure if it falls under "without using external code generators".
PP_ENUM version
Thanks to #TemplateRex for BOOST_PP_ENUM, as I said, I am not very experienced at PP yet :)
#include <boost/preprocessor/repetition/enum.hpp>
// with ENUM we don't need a comma at the end
#define MACRO(z,n,text) CountBits<8>(n)
int CB_LUT[256] = {
BOOST_PP_ENUM(256, MACRO, _)
};
#undef MACRO
The main difference with PP_ENUM is that it automatically adds the comma after each element and strips the last one.
PP_REPEAT version
#include <boost/preprocessor/repetition/repeat.hpp>
#define MACRO(z,n,data) CountBits<8>(n),
int CB_LUT[256] = {
BOOST_PP_REPEAT(256, MACRO, _)
};
#undef MACRO
Remarks
It's actually very straightforward and easy to use, though it's up to you to decide if you will accept macros. I've personally struggled a lot with Boost.MPL and template techniques, to find PP solutions easy to read, short and powerful, especially for enumerations like those. Additional important advantage of PP over TMP is the compilation time.
As for the comma at the end, all reasonable compilers should support it, but in case yours doesn't, simply change the number of repetitions to 255 and add last case by hand.
You might also want to rename MACRO to something meaningful to avoid possible redefinitions.
I like to do it like this:
#define MYLIB_PP_COUNT_BITS(z, i, data) \
CountBits< 8 >(i)
int CB_LUT[] = {
BOOST_PP_ENUM(256, MYLIB_PP_COUNT_BITS, ~)
};
#undef MYLIB_PP_COUNT_BITS
The difference with BOOST_PP_REPEAT is that BOOST_PP_ENUM generates a comma-separated sequence of values, so no need to worry about comma's and last-case behavior.
Furthermore, it is recommended to make your macros really loud and obnoixous by using a NAMESPACE_PP_FUNCTION naming scheme.
a small configuration thing is to omit the [256] in favor of [] in the array size so that you can more easily modify it later.
Finally, I would recommend making your CountBit function template constexpr so that you also can initialize const arrays.
I'm trying to create a compile-time bit mask using metaprograming techniques, my idea is to create something like this:
unsigned int Mask3 = Mask<2>(); // value = 0x03 = b00000000000000000000000000000011
unsigned int Mask3 = Mask<3>(); // value = 0x07 = b00000000000000000000000000000111
unsigned int Mask3 = Mask<7>(); // value = 0x7F = b00000000000000000000000001111111
The code that I'm trying is this:
template <const unsigned int N> const unsigned int Mask()
{
if (N <= 1)
{
return 1;
}
else
{
return ((1 << N) | Mask<N - 1>());
}
}
return 1;
But it result in tons pairs of warnings:
warning C4554: '<<' : check operator precedence for possible error
warning C4293: '<<' : shift count negative or too big
And in the end, the compile error:
error C1202: recursive type or function dependency context too complex.
So, I deduce that the recursivity never ends and falls into a compiler infinite loop but I'm don't understanding WHY.
As has already been pointed out, you're depending on a runtime check to
stop a compile time recursion, which can't work. More importantly,
perhaps, for what you want to do, is that you're defining a function,
which has no value until you call it. So even after you stop the
recursion with a specialization, you still have a nested sequence of
functions, which will be called at runtime.
If you want full compile time evaluation, you must define a static data
member of a class template, since that's the only way a compile time
constant can appear in a template. Something like:
template <unsigned int N>
struct Mask
{
static unsigned int const value = (1 << (N - 1)) | Mask<N - 1>::value;
};
template <>
struct Mask<0>
{
static unsigned int const value = 0;
};
(I've also corrected the numerical values you got wrong.)
Of course, you don't need anything this complicated. The following
should do the trick:
template <unsigned int N>
struct Mask
{
static unsigned int const value = (1 << (N + 1)) - 1;
};
template <>
struct Mask<0>
{
static unsigned int const value = 0;
};
(You still need the specialization for 0. Otherwise, 0 means all bits
set.)
Finally, of course: to access the value, you need to write something
like Mask<3>::value. (You might want to wrap this in a macro.)
It doesn't need to be recursive. This should work just fine :
template <const unsigned int N> const unsigned int Mask()
{
return ((1 << N) - 1);
}
It doesn't even need to be a template really. An (inlined) function is ok.
Note that if you want to support any value of N, specifically N >= sizeof(unsigned int) * CHAR_BIT, you probably want to treat those as a special case.
A template is created at compile time, but you are relying on run time behavior to stop the recursion.
For example, if you instantiate Mask<2>, it is going to use Mask<1>, which is going to use Mask<0>, which is going to use Mask<-1>, etc.
You have a runtime check for N being <= 1, but this doesn't help when it's compiling. It still creates an infinite sequence of functions.
To blunt template instantiation recursion you need to introduce one explicit specialization:
template <0> const unsigned int Mask()
{
return 1;
}
Your recursion never ends, because compiler tries to generate template implementation for both if-branches. So, when it generates Mask<0> it also generates Mask<0xffffffff> and so on
C++11 -- no recursion or templates:
constexpr unsigned mask(unsigned N) { return unsigned(~(-1<<N)); }
So far the answers only addressed the second error (C1202), but you asked more than that.
Warning C4554 is caused by a Microsoft compiler bug involving template parameters and the << operator. So, (1 << N) generates a warning. If N were an ordinary parameter, there would be no warning of course.
The very simple workaround is to use (1 << (N)) instead of (1 << N), and C4554 goes away!