Recursive template for compile-time bit mask - c++

I'm trying to create a compile-time bit mask using metaprograming techniques, my idea is to create something like this:
unsigned int Mask3 = Mask<2>(); // value = 0x03 = b00000000000000000000000000000011
unsigned int Mask3 = Mask<3>(); // value = 0x07 = b00000000000000000000000000000111
unsigned int Mask3 = Mask<7>(); // value = 0x7F = b00000000000000000000000001111111
The code that I'm trying is this:
template <const unsigned int N> const unsigned int Mask()
{
if (N <= 1)
{
return 1;
}
else
{
return ((1 << N) | Mask<N - 1>());
}
}
return 1;
But it result in tons pairs of warnings:
warning C4554: '<<' : check operator precedence for possible error
warning C4293: '<<' : shift count negative or too big
And in the end, the compile error:
error C1202: recursive type or function dependency context too complex.
So, I deduce that the recursivity never ends and falls into a compiler infinite loop but I'm don't understanding WHY.

As has already been pointed out, you're depending on a runtime check to
stop a compile time recursion, which can't work. More importantly,
perhaps, for what you want to do, is that you're defining a function,
which has no value until you call it. So even after you stop the
recursion with a specialization, you still have a nested sequence of
functions, which will be called at runtime.
If you want full compile time evaluation, you must define a static data
member of a class template, since that's the only way a compile time
constant can appear in a template. Something like:
template <unsigned int N>
struct Mask
{
static unsigned int const value = (1 << (N - 1)) | Mask<N - 1>::value;
};
template <>
struct Mask<0>
{
static unsigned int const value = 0;
};
(I've also corrected the numerical values you got wrong.)
Of course, you don't need anything this complicated. The following
should do the trick:
template <unsigned int N>
struct Mask
{
static unsigned int const value = (1 << (N + 1)) - 1;
};
template <>
struct Mask<0>
{
static unsigned int const value = 0;
};
(You still need the specialization for 0. Otherwise, 0 means all bits
set.)
Finally, of course: to access the value, you need to write something
like Mask<3>::value. (You might want to wrap this in a macro.)

It doesn't need to be recursive. This should work just fine :
template <const unsigned int N> const unsigned int Mask()
{
return ((1 << N) - 1);
}
It doesn't even need to be a template really. An (inlined) function is ok.
Note that if you want to support any value of N, specifically N >= sizeof(unsigned int) * CHAR_BIT, you probably want to treat those as a special case.

A template is created at compile time, but you are relying on run time behavior to stop the recursion.
For example, if you instantiate Mask<2>, it is going to use Mask<1>, which is going to use Mask<0>, which is going to use Mask<-1>, etc.
You have a runtime check for N being <= 1, but this doesn't help when it's compiling. It still creates an infinite sequence of functions.

To blunt template instantiation recursion you need to introduce one explicit specialization:
template <0> const unsigned int Mask()
{
return 1;
}
Your recursion never ends, because compiler tries to generate template implementation for both if-branches. So, when it generates Mask<0> it also generates Mask<0xffffffff> and so on

C++11 -- no recursion or templates:
constexpr unsigned mask(unsigned N) { return unsigned(~(-1<<N)); }

So far the answers only addressed the second error (C1202), but you asked more than that.
Warning C4554 is caused by a Microsoft compiler bug involving template parameters and the << operator. So, (1 << N) generates a warning. If N were an ordinary parameter, there would be no warning of course.
The very simple workaround is to use (1 << (N)) instead of (1 << N), and C4554 goes away!

Related

Why size of QByteArray is `int` rather than `unsigned int`

I have such expressions in my code:
QByteArray idx0 = ...
unsigned short ushortIdx0;
if ( idx0.size() >= sizeof(ushortIdx0) ) {
// Do something
}
But I'm getting the warning:
warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
if ( idx0.size() >= sizeof(ushortIdx0) ) {
~~~~~~~~~~~~^~~~~~~~~~
Why size() of QByteArray is returned as int rather than unsigned int? How can I get rid of this warning safely?
Some folk feel that the introduction of unsigned types into C all those years ago was a bad idea. Such types found themselves introduced into C++, where they are deeply embedded in the C++ Standard Library and operator return types.
Yes, sizeof must, by the standard, return an unsigned type.
The Qt developers adopt the modern thinking that the unsigned types were a bad idea, and favour instead making the return type of size a signed type. Personally I find it idiosyncratic.
To solve, you could (i) live with the warning, (ii) switch it off for the duration of the function, or (iii) write something like
(std::size_t)idx0.size() >= sizeof(ushortIdx0)
at the expense of clarity.
Why size() of QByteArray is returned as int rather than unsigned int?
I literally have no idea why Qt chose a signed return for size(). However, there are good reasons to use a signed instead of an unsigned.
One infamous example where a unsigned size() fails miserably is this quite innocent looking loop:
for (int i = 0; i < some_container.size() - 1; ++i) {
do_somehting(some_container[i] , some_container[i+1] );
}
Its not too uncommon to make the loop body operate on two elements and in that case its seems to be a valid choice to iterate only till some_container.size() - 1.
However, if the container is empty some_container.size() - 1 will silently (unsigned overflow is well defined) turn into the biggest value for the unsigned type. Hence, instead of avoiding the out-of-bounds access it leads to the maximum out of bounds you can get.
Note that there are easy fixes for this problem, but if size() does return a signed value, then there is no issue that needs to be fixed in the first place.
Because in Qt containers (like: QByteArray, QVector, ...) there are functions which can return a negative number, like: indexOf, lastIndexOf, contains, ... and some can accept negative numbers, like: mid, ...; So to be class-compatible or even framework-compatibe, developers use a signed type (int).
You can use standard c++ casting:
if ( static_cast<size_t>(idx0.size()) >= sizeof(ushortIdx0) )
The reason why is a duplicate part of the question, but the solution to the type mismatch is a valid problem to solve. For the comparisons of the kind you're doing, it'd probably be useful to factor them out, as they have a certain reusable meaning:
template <typename T> bool fitsIn(const QByteArray &a) {
return static_cast<int>(sizeof(T)) <= a.size();
}
template <typename T> bool fitsIn(T, const QByteArray &a) {
return fitsIn<T>(a);
}
if (fitsIn(ushortIdx0, idx0)) ...
Hopefully you'll have just a few kinds of such comparisons, and it'd make most sense to DRY (do not repeat yourself) and instead of a copypasta of casts, use functions dedicated to the task - functions that also express the intent of the original comparison. It then becomes easy to centralize handling of any corner cases you might wish to handle, i.e. when sizeof(T) > INT_MAX.
Another approach would be to define a new type to wrap size_t and adapt it to the types you need to use it with:
class size_of {
size_t val;
template <typename T> static typename std::enable_if<std::is_signed<T>::value, size_t>::type fromSigned(T sVal) {
return (sVal > 0) ? static_cast<size_t>(sVal) : 0;
}
public:
template <typename T, typename U = std::enable_if<std::is_scalar<T>::value>::type>
size_of(const T&) : val(sizeof(T)) {}
size_of(const QByteArray &a) : val(fromSigned(a.size())) {}
...
bool operator>=(size_of o) const { return value >= o.value; }
};
if (size_of(idx0) >= size_of(ushortIdx0)) ...
This would conceptually extend sizeof and specialize it for comparison(s) and nothing else.

C++ template specialization not working when comparing sizeof(type) == constant

#include <cstdint>
#include <iostream>
uint32_t ReadInt() { return 0; }
uint64_t ReadLong() { return 0; }
template<class SizeType>
class MyType
{
public:
SizeType Field;
MyType()
{
Field = sizeof(SizeType) == 8 ? ReadLong() : ReadInt();
}
};
int main()
{
MyType<uint32_t> m;
std::cout << m.Field;
}
I get a compiler warning because it looks like the the condition sizeof(MyType) == 4 is not being evaluated at compile time.
If it were we would have specialization and this wouldn't be a problem.
Anyway I could achieve this?
EDIT: What I really want to achieve is this:
Field = ReadLong();
OR
Field = ReadInt();
through template meta-programming. Can I do this without specializing the Read* functions? Given how powerful c++ templates I really feel like I'm missing some aha moment, because if I have to specialize Read* functions the rabbit hole continues to go deeper.
warning C4244: '=': conversion from 'int64_t' to 'uint32_t', possible loss of data for the int specialization
I get a compiler warning because it looks like the the condition sizeof(MyType) == 4 is not being evaluated at compile time.
No, it doesn't look like that at all. Whether the comparison is evaluated at compile time or not has no effect on the return type of the conditional expression.
If we simplify the example by removing the conditional and expand the template using the type that you used, we get this minimal reproduction:
uint32_t Field;
int64_t temp = ReadInt(); // temporary variable for exposition
Field = temp;
Clearly, the int64_t may contain a value that is much greater than can be represented by int32_t. If it does, then the correct value will be lost in conversion. The compiler warns about this possibility.
Why is the type of the conditional int64_t specifically you might ask. Well, the types of each subexpression is int64_t and uint32_t respectively, so the type must be one of those. The type of Field or the result of the comparison - even if evaluated at compile time - has no effect on which type is chosen. Instead, integral promotion rules are used in this case.
Anyway I could achieve this?
A template function should work:
template<class T>
T Read() {
static_assert(false, "not implemented");
}
template<>
uint64_t Read<uint64_t>() {
return ReadLong();
}
template<>
uint32_t Read<uint32_t>() {
return ReadInt();
}
// ...
Field = Read<SizeType>();
I get a compiler warning because it looks like the the condition sizeof(MyType) == 4 is not being evaluated at compile time
Wrong, it is evaluated at compile-time, but the conditional operator still has the bigger result-type of the arguments.
To fix it, just add a cast:
Field = sizeof(SizeType) == 8 ? (SizeType)ReadLong() : ReadInt();
Another approach using overload-resolution:
long ReadData(long) { return ReadLong(); }
int ReadData(int) { return ReadInt(); }
Field = ReadData(SizeType());
A conditional operator ?: expression evaluates to a single type, which accommodates both possible internal-to-the-construct result types.
This happens even when it's evaluated at compile time.
The common type in your case is the largest of the two possible integer types.

How to make a template with customized bit shift values

Suppose that I need a function template that does different amounts of bit shifts on different integer types. For example, if the input value n is of type char, the function does some calculations on n>>2 and n<<3. If it is short int, then the function uses n>>1 and n<<8. For int type, there would be n>>11 and n<<9, and so on. Of course, the values mentioned are just for example and there is no correlation between them and the size of int_type.
My suggestion for this problem was something like this:
template <typename Num_Type = char, int s1 = 2, int s2 = 3> void test1(Num_Type &n)
{
// this is just an example code:
int a = n >> s1,
b = n << s2;
// do some calculations on a and b
}
So I can use this template for different types of input value. But when I compile it, it gives me several warnings about undefined behavior in shift operations, because the value may be too big.
So the question can be asked in two forms:
How can I customize bit-shift operations without getting "undefined behavior" warnings? or
Can I have a limited data type, say intlength which can only have a specified range of numbers, say 0, 1, 2, ... , 31? I know it may sound stupid, but in this case, I can define the template as
template <typename Num_Type = char, intlength s1 = 2, intlength s2 = 3> void test1(Num_Type &n)
and so the compiler should not complain about shift values.
You can disable the warning for this function with a #pragma command, which depends on the compiler you are using. You will have to google that yourself.
Edit: since you mention you are using VisualStudio, here is a link to disable warnings. :)
Once you disable the warning you can add a static_assert to check if s1 and s2 are in range of the Num_Type. Although I find it strange that your compiler does not do that itself since everything is known at compile time.
template <typename Num_Type, int s1 = 2, int s2 = 3> void test1(Num_Type &n)
{
constexpr auto max = sizeof(Num_Type) * 8;
static_assert(s1 < max && s2 < max, "integer overflow");
// this is just an example code:
int a = n >> s1,
b = n << s2;
}
Live demo
The healthy way to change template function behavior depending on input type is to use template specialization. For example:
template <typename num_t> test(num_t n);
template <> test<int> (num_t n){
int shift_amount = 4;
// do shifts
}
template <> test<char> (num_t n){
int shift_amount = 1;
// do shifts
}
Also, make sure to surround your shifts with if checks to make sure there is no overflow.
I see no need for templates. All you need is simple overloading:
auto test(char n) { ... }
auto test(uint16_t n) { .. }
auto test(uint32_t n) { .. }

C++11 constexpr function compiler error with ternary conditional operator (?:)

What is wrong with this piece of code?
#include <iostream>
template<unsigned int N, unsigned int P=0>
constexpr unsigned int Log2() {
return (N <= 1) ? P : Log2<N/2,P+1>();
}
int main()
{
std::cout << "Log2(8) = " << Log2<8>() << std::endl;
return 0;
}
When compiling with gcc version 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5), I get the following error:
log2.cpp: In function ‘constexpr unsigned int Log2() [with unsigned int N = 0u, unsigned int P = 1023u]’:
log2.cpp:5:38: error: template instantiation depth exceeds maximum of 1024 (use -ftemplate-depth= to increase the maximum) instantiating ‘constexpr unsigned int Log2() [with unsigned int N = 0u, unsigned int P = 1024u]’
log2.cpp:5:38: recursively instantiated from ‘constexpr unsigned int Log2() [with unsigned int N = 4u, unsigned int P = 1u]’
log2.cpp:5:38: instantiated from ‘constexpr unsigned int Log2() [with unsigned int N = 8u, unsigned int P = 0u]’
log2.cpp:10:37: instantiated from here
Constexpr doesn't work that way.
Simply put, constexpr functions must be available as runtime functions too. Imagine you took the constexpr off the function. Then think about why it cannot possibly work.
The reason is that the compiler has to instantiate the body of the function completely; it cannot decide based on the condition in the ?: not to instantiate one side. So it always has to instantiate the recursive call, leading to infinite recursion.
In any case, you're using constexpr wrong. You're using the old template metaprogramming calculation technique (passing stuff as template parameters) when constexpr was intended to replace this. Just use normal parameters.
constexpr unsigned Log2(unsigned n, unsigned p = 0) {
return (n <= 1) ? p : Log2(n / 2, p + 1);
}
std::cout << "Log2(8) = " << Log2(8) << std::endl;
Edit: I'll try to elaborate on how this works.
When the compiler encounters your code, it parses the template function and stores it in template form. (How this works differs between compilers.) So far, all is fine.
Next, in main, the compiler sees the call Log2<8>(). It sees that it has to instantiate the template, so it goes ahead and does exactly that: it instantiates Log2<8, 0>. The body of the function template is this:
return (N <= 1) ? P : Log2<N/2,P+1>();
OK, the compiler sees this, but it doesn't try to evaluate it. Why would it? It's currently instantiating a template, not calculating a value. It just substitutes the values supplied:
return (8 <= 1) ? 0 : Log2<8/2,0+1>();
Huh, there's another template instantiation here. It doesn't matter that it's in a conditional expression, or that the left hand side could be known. Template instantiation must be complete. So it goes ahead and calculates the values for the new instantiation and then instantiates Log2<4, 1>:
return (4 <= 1) ? 1 : Log2<4/2,1+1>();
And the game begins again. There's a template instantiation in there, and it's Log2<2, 2>:
return (2 <= 1) ? 2 : Log2<2/2,2+1>();
And again, Log2<1,3>():
return (1 <= 1) ? 3 : Log2<1/2,3+1>();
Did I mention that the compiler doesn't care about the semantic meaning of this stuff? It's just yet another template to instantiate: Log2<0,4>:
return (0 <= 1) ? 4 : Log2<0/2,4+1>();
And then Log2<0,5>:
return (0 <= 1) ? 5 : Log2<0/2,5+1>();
And so on, and so on. At some point the compiler realizes that it never stops, and gives up. But at no point does it say, "Wait, the condition of that ternary operator is false, I don't need to instantiate the right-hand side." That's because the C++ standard doesn't allow it to. The function body must be instantiated completely.
Now look at my solution. There's no template. There's just a function. The compiler sees it and goes, "Hey, here's a function. Awesome, let me put in a call to that function here." And then at some point (it might be immediately, it might be a lot later, depending on the compiler), it might (but is not forced to, in this case) say, "Hey, wait, this function is constexpr and I know the parameter values, let me evaluate that." Now it goes ahead and evaluates Log2(8, 0). Remember the body:
return (n <= 1) ? p : Log2(n / 2, p + 1);
"OK", the compiler says, "I just want to know what this function returns. Let's see, 8 <= 1 is false, so look at the right side. Log2(4, 1), huh? Let me look at that. OK, 4 <= 1 is also false, so it must be Log2(2, 2). What's that, 2 <= 1? Also false, so it's Log2(1, 3). Hey, 1 <= 1 is true, so let me take that 3 and return it. All the way up the call stack."
So it comes up with the answer 3. It doesn't go into endless recursion because it's evaluating the function with full knowledge of values and semantics, not just stupidly building up ASTs.
I hope that helps.
As others already said: The compiler won't evaluate a conditional expression such as if/else or a ternary operator ?:. However, there is still a way to make this conditional expression compile time:
#include <cstddef> // size_t is shorter than unsigned int, it's a matter of taste in this case
#include <iostream> // to show our results
#include <type_traits> // needed for the mighty std::enable_if
template<size_t N, size_t P = 0>
constexpr typename std::enable_if<(N <= 1), size_t>::type Log2()
{
return P;
}
template<size_t N, size_t P = 0>
constexpr typename std::enable_if<!(N <= 1), size_t>::type Log2()
{
return Log2<N / 2, P + 1>();
}
int main()
{
std::cout << Log2<1>() << "\n";
std::cout << Log2<2>() << "\n";
std::cout << Log2<4>() << "\n";
std::cout << Log2<8>() << "\n";
std::cout << Log2<16>() << "\n";
}
What this does is rather obvious: If N <= 1, the first branch should be evaluated, thus Log2<0, P>() and Log2<1, P>() should evaluate to P. If N <= 1, the upper method is enabled as this function header is valid. For everything else, i.e. N >= 2 or !(N <= 1), we need to recurse, which is done by the second method.

Compile time type conversion check (constexpr and user defined literals)

Update: I posted my own answer below
And there's a longer version of this matter here: http://scrupulousabstractions.tumblr.com/post/38460349771/c-11-type-safe-use-of-integer-user-defined-literals
Question:
I've made a simple constexpr user defined literal _X that gets the value as an unsigned long long (that's how numeral user defined literals work: http://en.cppreference.com/w/cpp/language/user_literal), and then I make sure that the value fits in a signed long long.
It all works well (too big values cause compilation error), but only when I explicitly create a variable such as
constexpr auto a= 150_X;
If instead I write something typical like
cout << 150_X << endl;;
the tests are not performed at compile time.
Are constexpr functions only executed at compile time if they are assigned to a constexpr variable? (I could not find that in the standard)
Is it possible to achieve the safe behaviour of _X that I'm looking for?
Full example:
#include<iostream>
#include<stdexcept>
inline constexpr long long testConv(unsigned long long v) {
return (v > 100 ) ? throw std::exception() : v;
} // will eventually use actual limit from numeric_limits
inline constexpr long long operator "" _X(unsigned long long f) {
return testConv(f) ;
}
int main(){
constexpr auto a= 5_X;
std::cout << a << std::endl;
std::cout << 200_X << std::endl; // This bad literal is accepted at compile time
constexpr auto c=250_X; // This bad literal is not accepted at compile time
std::cout << c << std::endl;
}
oh, for reference: I used gcc4.7.2.
Self answer:
I found a complete solution, inspired by the comments and other answers to my question, and other questions such as https://stackoverflow.com/a/13384317/1149664.
The solution is to use the template form of user defined literals and sum up the number manually, multiplying the sum based on the already parsed digits by 10.
I wrote a detailed version of this self-answer here: http://scrupulousabstractions.tumblr.com/post/38460349771/c-11-type-safe-use-of-integer-user-defined-literals
template<char... Chars>
int operator"" _steps(){
return {litparser<0,Chars...>::value};
}
Litparser is a little template meta-program which takes a list of characters as arguments expanded from the input characters held by the Chars parameter pack.
typedef unsigned long long ULL;
// Delcare the litparser
template<ULL Sum, char... Chars> struct litparser;
// Specialize on the case where there's at least one character left:
template<ULL Sum, char Head, char... Rest>
struct litparser<Sum, Head, Rest...> {
// parse a digit. recurse with new sum and ramaining digits
static const ULL value = litparser<
(Head <'0' || Head >'9') ? throw std::exception() :
Sum*10 + Head-'0' , Rest...>::value;
};
// When 'Rest' finally is empty, we reach this terminating case
template<ULL Sum> struct litparser<Sum> {
static const ULL value = Sum;
};
constexpr functions may be executed at compile time; that is, they are eligible to be used in constant expressions. If they're not used in a constant expression, there's no point executing them at compile time, although I think it's allowed.
Since you're not allowed to declare a parameter as constexpr (section 7.1.5/1) [1], I don't think there is any way to force evaluation of operator "" _X(unsigned long long) at compile time, but you can probably do it with template<char...> operator "" _X()
If the constexpr function is called within a constant expression, the argument will be a constant expression (and if it is not, then the call is not a constant expression). However, you cannot force the call to be a constant expression by declaring the parameter to be constexpr, because you're not allowed to declare parameters as constexpr, see reference to standard.
[Note 1]: Thanks to #LightnessRacesInOrbit for searching the standard to justify the claim in paragraph two.
Constexpr functions need not be executed at compile-time. But your goal can be achieved. For better understanding of the problem, and an example how to create a iteral that is always evaluated at compile-time, I recommend this post.