Can a preprocessor macro expand just some pasted parameters? - c++

I know that in expanding a function-like preprocessor macro, the # and ## tokens in the top-level substitution list essentially act "before" any macro expansions on the argument. For example, given
#define CONCAT_NO_EXPAND(x,y,z) x ## y ## z
#define EXPAND_AND_CONCAT(x,y,z) CONCAT_NO_EXPAND(x,y,z)
#define A X
#define B Y
#define C Z
then CONCAT_NO_EXPAND(A,B,C) is the pp-token ABC, and EXPAND_AND_CONCAT(A,B,C) is the pp-token XYZ.
But what if I want to define a macro that expands just some of its arguments before pasting? For example, I would like a macro that allows only the middle of three arguments to expand, then pastes it together with an exact unexpanded prefix and an exact unexpanded suffix, even if the prefix or suffix is the identifier of an object-like macro. That is, if again we have
#define MAGIC(x,y,z) /* What here? */
#define A X
#define B Y
#define C Z
then MAGIC(A,B,C) is AYC.
A simple attempt like
#define EXPAND(x) x
#define MAGIC(x,y,z) x ## EXPAND(y) ## z
results in an error 'pasting ")" and "C" does not give a valid preprocessing token". This makes sense (and I assume it's also producing the unwanted token AEXPAND).
Is there any way to get that sort of result using just standard, portable preprocessor rules? (No extra code-generating or -modifying tools.)
If not, maybe a way that works on most common implementations? Here Boost.PP would be fair game, even if it involves some compiler-specific tricks or workarounds under the hood.
If it makes any difference, I'm most interested in the preprocessor steps as defined in C++11 and C++17.

Here's a solution:
#define A X
#define B Y
#define C Z
#define PASTE3(q,r,s) q##r##s
#define MAGIC(x,y,z,...) PASTE3(x##__VA_ARGS__,y,__VA_ARGS__##z)
MACRO(A,B,C,)
Note that the invocation "requires" another argument (see below for why); but:
MACRO(A,B,C) here is compliant for C++20
MACRO(A,B,C) will "work" in many C++11/C++17 preprocessors (e.g., gnu/clang), but that is an extension not a C++11/C++17 compliant behavior
I know that in expanding a function-like preprocessor macro, the # and ## tokens in the top-level substitution list essentially act "before" any macro expansions on the argument.
To be more precise, there are four steps to macro expansion:
argument identification
argument substitution
stringification and pasting (in an unspecified order)
rescan and further replacement
Argument identification associates parameters in the macro definition with arguments in an invocation. In this case, x associates with A, y with B, z with C, and ... with a "placemarker" (abstract empty value associated with a parameter whose argument has no tokens). For C++ preprocessors up to C++20, use of a ... requires at least one parameter; since C++20's addition of the __VA_OPT__ feature, use of the ... in an invocation is optional.
Argument substitution is the step where arguments are expanded. Specifically, what happens here is that for each parameter in the macro's replacement list (here, PASTE3(x##__VA_ARGS__,y,__VA_ARGS__##z)), where said parameter does not participate in a paste or stringification, the associated argument is fully expanded as if it appeared outside of an invocation; then, all mentions of that parameter in the replacement list that do not participate in stringification and paste are replaced with the expanded result. For example, at this step for the MAGIC(A,B,C,) invocation, y is the only mentioned qualifying parameter, so B is expanded producing Y; at that point we get PASTE3(x##__VA_ARGS__,Y,__VA_ARGS__##z).
The next step applies pastes and stringification operators in no particular order. Placemarker's are needed here specifically because you want to expand the middle and not the end, and you don't want extra stuff; i.e., to get A to not expand to X, and to stay A (as opposed to changing to "A"), you need to avoid argument substitution specifically. a.s. is avoided in only two ways; pasting or stringifying, so if stringification doesn't work we have to paste. And since you want that token to stay the same as what you had, you need to paste to a placemarker (which means you need one to paste to, which is why there's another parameter).
Once this macro applies the pastes to the "placemarkers", you wind up with PASTE3(A,Y,C); then there is the rescan and further replacement step, during which PASTE3 is identified as a macro invocation. Fast forwarding, since PASTE3 pastes its arguments, a.s. doesn't apply to any of them, we do the pastes in "some order" and we wind up with AYC.
As a final note, in this solution I'm using a varying argument to produce the placemarker token precisely because it allows invocations of the form MACRO(A,B,C) in at least C++20. I'm left-pasting that to z because that makes the addition at least potentially useful for something else (MAGIC(A,B,C,_) would use _ as a "delimiter" to produce A_Y_C).

Related

Why won't my variadic macro accept no arguments correctly?

Overloading Macro on Number of Arguments
https://codecraft.co/2014/11/25/variadic-macros-tricks/
I've been looking at the two links above, trying to get the following code to work:
#define _GET_NUMBER(_0, _1, _2, _3, _4, _5, NAME, ...) NAME
#define OUTPUT_ARGS_COUNT(...) _GET_NUMBER(_0, ##__VA_ARGS__, 5, 4, 3, 2, 1, 0)
...
cout << OUTPUT_ARGS_COUNT("HelloWorld", 1.2) << endl;
cout << OUTPUT_ARGS_COUNT("HelloWorld") << endl;
cout << OUTPUT_ARGS_COUNT() << endl;
This compiles, runs, and gives the following output:
2
1
1
I can not for the life of me figure out why the call OUTPUT_ARGS_COUNT() is giving me 1 instead of 0. I have an ok understanding of the code I'm trying to use, but it's a tad greek to me still so I guess it's possible that I'm not applying something correctly despite the fact I literally copied and pasted the example code from the link on stack overflow.
I'm compiling using g++ 5.4.0 20160609.
Any ideas or additional resources you can point me to would be greatly appreciated.
You can see at http://gcc.gnu.org/onlinedocs/cpp/Variadic-Macros.html:
Second, the ‘##’ token paste operator has a special meaning when placed between a comma and a variable argument. If you write
#define eprintf(format, ...) fprintf (stderr, format, ##__VA_ARGS__)
and the variable argument is left out when the eprintf macro is used, then the comma before the ‘##’ will be deleted. This does not happen if you pass an empty argument, nor does it happen if the token preceding ‘##’ is anything other than a comma.
eprintf ("success!\n")
→ fprintf(stderr, "success!\n");
The above explanation is ambiguous about the case where the only macro parameter is a variable arguments parameter, as it is meaningless to try to distinguish whether no argument at all is an empty argument or a missing argument. CPP retains the comma when conforming to a specific C standard. Otherwise the comma is dropped as an extension to the standard.
So, (unless appropriate extension used) OUTPUT_ARGS_COUNT() is counted as 1 empty argument (comma kept with ##__VA_ARGS__).
The C standard specifies
If the identifier-list in the macro definition does not end with an ellipsis, [...]. Otherwise, there shall be more arguments in the invocation than there are parameters in the macro definition (excluding the ...)
(C2011 6.10.3/4; emphasis added)
C++11 contains language to the same effect in paragraph 16.3/4.
In both cases, then, if your macro invocation were interpreted to have zero arguments then your program would be non-conforming. On the other hand, the preprocessor does recognize and support empty macro arguments -- that is, arguments consisting of zero preprocessing tokens. In principle, then, there is an ambiguity here between no argument and a single empty argument, but in practice, only the latter interpretation results in a conforming program.
That g++ opts for the latter interpretation (the other answer quotes its documentation to that effect) is thus reasonable and appropriate, but it is not safe to rely upon it if you want your code to be portable. A compiler that takes the alternative interpretation would behave differently, possibly by providing the behavior you expected, but also possibly by rejecting the code.

Macro expansion order confusion between compilers

This piece of code compiles in Visual Studio 2015, but not in Clang:
#define COMMA ,
#define MC(a) a
#define MA(a,b,c) MC(a b c)
map <MA(int,COMMA,int)> FF;
It appears that Clang expands the COMMA macro before submitting it to the MC() macro. "Who is right" according to the C++ standard? Also, how can I make Clang behave like Visual Studio?
EDIT: Simplified the example, and changed some macro names.
Clang conforms to the standard; Visual Studio doesn't. I think you will have a lot of trouble getting Clang to not conform to the standard, so I won't attempt to answer "how do I get Clang to act like Visual Studio?". Maybe that wasn't really what you wanted to know.
When the compiler identifies the invocation of a function-like macro (that is, a macro with parameters) it expands the macro using the procedure explained in detail in §16.3 [cpp.replace] of the C++ standard. In the following, I've simplified the procedure by not considering the # and ## operators, because they do not appear in your example and the full procedure is more complicated.
We'll examine the invocation MC(int, COMMA, int). Here's what happens after the compiler sees the tokens MC and (, which indicate an invocation of the macro.
The compiler identifies what the arguments are, which involves finding the closing parenthesis. There are three arguments, which corresponds to the number of parameters, so that's OK. The arguments have not yet been expanded, so the compiler only sees the punctuation actually in the source file. It identifies the arguments as int, COMMA and int.
Every argument (except the ones whose corresponding parameter participates in token concatenation or stringification -- but, as I said, I'm not going to go into that scenario here) are then fully expanded. This happens before they are substituted into the macro body, so that the names of the macro's parameters don't leak out of the macro. So now the three arguments are int, , and int.
A copy of the macro body is made, in which each parameter is substituted with the corresponding (fully expanded) argument. The macro body ("replacement list", in standardese) was MC(A B C); after substituting the arguments, that becomes MC(A , C).
The sequence of tokens created in step 3 is inserted into the input in place of the macro invocation, and preprocessing continues.
At this point, the compiler will see the invocation of the function-like macro MC(A, B), and will proceed as above. However, this time the first step fails because two arguments are identified but the macro MC only has one parameter.

C++ Preprocessor Standard Behaviour

I'm studying the C++ standard on the exact behaviour the preprocessor (I need to implement some sort of C++ preprocessor). From what I understand, the example I made up (to aid my understanding) below should be valid:
#define dds(x) f(x,
#define f(a,b) a+b
dds(eoe)
su)
I expect the first function like macro invocation dds(eoe) be replaced by f(eoe, (note the comma within the replacement string) which then considered as f(eoe,su) when the input is rescanned.
But a test with VC++2010 gave me this (I told the VC++ to output the preprocessed file):
eoe+et_leoe+et_l
su)
This is counter-intuitive and is obviously incorrect. Is it a bug with VC++2010 or my misunderstanding of the C++ standard? In particular, is it incorrect to put a comma at the end of the replacement string like I did? My understanding of the C++ standard grammar is that any preprocessing-token's are allowed there.
EDIT:
I don't have GCC or other versions of VC++. Could someone help me to verify with these compilers.
My answer is valid for the C preprocessor, but according to Is a C++ preprocessor identical to a C preprocessor?, the differences are not relevant for this case.
From C, A Reference Manual, 5th edition:
When a functionlike macro call is encoutered, the entire macro call is
replaced, after parameter processing, by a copy of the body. Parameter
processing proceeds as follows. Actual argument token strings are
associated with the corresponding formal parameter names. A copy of
the body is then made in which every occurrence of a formal parameter
name is replace by a copy of the actual parameter token sequence
associated with it. This copy of the body then replaces the macro
call.
[...] Once a macro call has been expanded, the scan for macro calls
resumes at the beginning of the expansion so that names of macros may
be recognized within the expansion for the purpose of further macro
replacement.
Note the words within the expansion. That's what makes your example invalid. Now, combine it with this: UPDATE: read comments below.
[...] The macro is invoked by writing its name, a left parenthesis,
then once actual argument token sequence for each formal parameter,
then a right parenthesis. The actual argument token sequences are
separated by commas.
Basically, it all boils down to whether the preprocessor will rescan for further macro invocations only within the previous expansion, or if it will keep reading tokens that show up even after the expansion.
This may be hard to think about, but I believe that what should happen with your example is that the macro name f is recognized during rescanning, and since subsequent token processing reveals a macro invocation for f(), your example is correct and should output what you expect. GCC and clang give the correct output, and according to this reasoning, this would also be valid (and yield equivalent outputs):
#define dds f
#define f(a,b) a+b
dds(eoe,su)
And indeed, the preprocessing output is the same in both examples. As for the output you get with VC++, I'd say you found a bug.
This is consistent with C99 section 6.10.3.4, as well as C++ standard section 16.3.4, Rescanning and further replacement:
After all parameters in the replacement list have been substituted and # and ##
processing has taken place, all placemarker preprocessing tokens are removed. Then, the
resulting preprocessing token sequence is rescanned, along with all subsequent
preprocessing tokens of the source file, for more macro names to replace.
To the best of my understanding there is nothing in the [cpp.subst/rescan] portions of the standard that makes what you do illegal, and clang and gcc are right in expanding it as eoe+su, and the MSC (Visual C++) behaviour has to be reported as a bug.
I failed to make it work but I managed to find an ugly MSC workaround for you, using variadics - you may find it helpful, or you may not, but in any event it is:
#define f(a,b) (a+b
#define dds(...) f(__VA_ARGS__)
It is expanded as:
(eoe+
su)
Of course, this won't work with gcc and clang.
Well, the problem i see is that the preprocessor does the following
ddx(x) becomes f(x,
However, f(x, is defined as well (even thou it's defined as f(a,b) ), so f(x, expands to x+ garbage.
So ddx(x) finally transforms into x + garbage (because you defined f(smthing, ).
Your dds(eoe) actually expands into a+b where a is eoe and b is et_l .
And it does that twice for whatever reason :).
This scenario you made is compiler specific, depends how the preprocessor chooses to handle the defines expansion.

What does the tilde (~) in macros mean?

Seen on this site, the code shows macro invocations using a tilde in parentheses:
HAS_COMMA(_TRIGGER_PARENTHESIS_ __VA_ARGS__ (~))
// ^^^
What does it mean / do? I suspect it to just be an empty argument, but I'm not sure. Is it maybe specific to C(99) like the __VA_ARGS__ is specific to C99 and existent in C++?
On the introduction page of Boost.Preprocessor, an example is given in A.4.1.1 Horizontal Repetition
#define TINY_print(z, n, data) data
#define TINY_size(z, n, unused) \
template <BOOST_PP_ENUM_PARAMS(n, class T)> \
struct tiny_size< \
BOOST_PP_ENUM_PARAMS(n,T) \
BOOST_PP_COMMA_IF(n) \
BOOST_PP_ENUM( \
BOOST_PP_SUB(TINY_MAX_SIZE,n), TINY_print, none) \
> \
: mpl::int_<n> {};
BOOST_PP_REPEAT(TINY_MAX_SIZE, TINY_size, ~) // Oh! a tilde!
#undef TINY_size
#undef TINY_print
An explanation is provided below:
The code generation process is kicked off by calling BOOST_PP_REPEAT, a higher-order macro that repeatedly invokes the macro named by its second argument (TINY_size). The first argument specifies the number of repeated invocations, and the third one can be any data; it is passed on unchanged to the macro being invoked. In this case, TINY_size doesn't use that data, so the choice to pass ~ was arbitrary. [5]
(emphasis mine)
And there is the note:
[5] ~ is not an entirely arbitrary choice. Both # and $ might have been good choices, except that they are technically not part of the basic character set that C++ implementations are required to support. An identifier like ignored might be subject to macro expansion, leading to unexpected results.
The tilde, therefore, is simply a place holder because an argument is required, but none is necessary. Since any user-defined identifier wannabe could be expanded, you need to use something else.
It turns out that ~ is pretty much unused (binary negation is not that often called) in comparison to + or - for example, so there is little chance of confusion. Once you've settled on this, using it consistently gives it a new meaning to the tilde; like using operator<< and operator>> for streaming data has become a C++ idiom.
The ~ does nothing. Almost any other content inside those parentheses would work the same.
The lynchpin of this trick is to test whether _TRIGGER_PARENTHESIS_ is next to (~) in the expansion of _TRIGGER_PARENTHESIS_ __VA_ARGS__ (~). Either way, HAS_COMMA(...) expands its arguments to either 0 or 1.
The arguments to be tested is placed between the macro and its parenthesis, the macro only triggers if the arguments are empty:
_TRIGGER_PARENTHESIS_ __VA_ARGS__ (~)
NOTE: Actually the very link you posted states it. I will check for a reference to this in the standard.

Argument Preceded by a # Token in a Macro

#define LINK_ENTITY_TO_CLASS(mapClassName,DLLClassName) \
static CEntityFactory<DLLClassName> mapClassName( #mapClassName );
This is a macro from the Alien Swarm mod for Half-Life 2, meant to be compiled with MSVC.
I've never seen an argument preceded by a # in a macro before, and I'm not sure if this is a MSVC specific thing or just uncommon. What does it mean?
This is part of both standard C and C++ and is not implementation-specific. The # preprocessing operator stringizes its argument. It takes whatever tokens were passed into the macro for the parameter designated by its operand (in this case, the parameter mapClassName) and makes a string literal out of them. So, for a simple example,
#define STRINGIZE(x) # x
STRINGIZE(Hello World)
// gets replaced with
"Hello World"
Note that the argument tokens are not macro replaced before they are stringized, so if Hello or World were defined as a macro, the result would still be the same. You need to use an extra level of indirection to get the arguments macro replaced (that linked answer discusses the concatenation operator, ##, but applies equally to the stringization operator.