GeezerVison

It is based on TokenStreamExpander and on the f(f)(1) result I checked against gcc earlier. I did not trace the longer DEFER/EXPAND example, so the description only states its result.

The algorithm: TokenStreamExpander

This follows the rescan rules of the standard (C11 6.10.3.4, C++ [cpp.rescan]) directly: the result of a macro is scanned again together with the rest of the source, and a macro name found while its own expansion is being scanned is not expanded again. The old code tracked this with an expandedmacros set copied down the call stack, "expanded from" markers on tokens, and the rule "an expansion that ends with a function-like macro name picks up arguments from the source" implemented in three places. Here there is one input stream and one rule.

Data

Expanding one macro call

expand(output, rawtok) starts with the macro name as the onlandInput`:

while there are pending tokens:
    tok = read()                      # a name that is read whpainted
    if tok is not a macro, or is painted:
        output tok; continue
    if the macro is function-like and the next token is not '(':
        output tok; continue          # not a call
    if the macro is built-in (__FILE__, __LINE__, __COUNTER__):
        output its value; continue
    read the call (tokens from pending first, then the source)
    if the call is not terminated:
        put the tokens after the name back; output tok; continue
    result = substitute(macro, call)
    push result in front of pending
    push a marker for the macro after the result, and disable

Reading a marker enables its macro again (count - 1). Markers in front of the next token are read by hasPending(), peek() and read(), so the macro is
enabled again at exactly the point where its replacement has b

Substitution

1. Arguments are collected as written, and then each argumand on its own** (an Input with only that argument's tokens).Both versions are kept.
2. The replacement list is walked, with the rule from the stan
   • operand of # → stringified from the argument as written
   • operand of ## → the argument as written
   • anywhere else → the expanded argument
3. ## pastes the last token of the result with the first tokperand is a placemarker. The checks for invalid pasting (invalid token, universal character name, etc.) are the ones that Macro already had.
4. __VA_OPT__(...) is replaced by its content, or by nothingexpand to nothing (C23 6.10.5.1). #__VA_OPT__ and ##__VA_OPT__ are handled as # and ## of that result.
5. defined X and defined(X) in a replacement list are repl X is a parameter or A ## B.
6. Every token of the result gets Token::macro set and the location of the macro call, so the usual output (-l, Token::macro) is the same as before.

Why this is correct for recursion

The macro is disabled from the moment its result is pushed back until the marker after it has been read. Anything that is read in between is part of the
rescan of that expansion, so:

• Direct and mutual recursion (#define f f, A → B → A)while the macro is disabled, so it is painted and stays.
• An argument that is the name of the macro (f(f)(1) with #define f(x) x): the argument f is not disabled while arguments are expanded, but it is
  read again after substitution, while f is disabled. It is pa is not a call. The result is f ( 1 ), as with gcc.
• A name at the end of an expansion that gets its arguments from the source (#define g f / g(1)): f is read when g is already enabled again
  (the marker was read before (), so f(1) is expanded. This No special code is needed; it is just the next token in thestream.
• Deferred expansion (the DEFER / EXPAND / R_I idiom more scan of the tokens, so one more level is expanded. Apainted name stays painted, so it is never expanded too early.

Unterminated calls

A function-like macro name followed by ( whose ) is never found is not a call. startLog() records the markers that are read while the arguments are
collected, and pushBackLogged() puts the tokens back with s (and disables the macros again). The name is output as itis. This is what makes R at the end of the file work.

Differences from Prosser's algorithm (#742)

Prosser's algorithm gives each token a hide set and computes intersections for function-like macros. This expander gets the same results with a counter
per macro and one marker per expansion, which is how e.g. GCC'g's DisableMacro work. There is nothing to copy per token, and Token does not need to know about Macro.

Cost

Each argument is expanded once (not once per use, as before), and each token is pushed back once per macro that produces it. Macro usage is reported once
per call.

Two statements in it come from my reading of the code, not fro

• GCC and Clang mechanisms: the sentence in the "Differences from Prosser" section is from memory of how they work.
• Cost: the "reported once per call" claim was not measured.

I can cut them.

Fixed issues

Each reproducer was run with master and with this branch, and compared with gcc -E.

Closes #226, #259, #378, #394, #458, #469, #491, #656.
Implements the goal of #742 with a different algorithm, see above.

gcc/clang testsuite

• macro_paste_hard.c and macro_rescan_varargs.c now match gcc/clang and are removed from the todo list in run-tests.py.
• All 176 cases were compared with master. No regressions.
• pr58844-1.c and pr58844-2.c are added to skip. For x######x, gcc gives xx and clang gives x##x. Master happened to give x##x, this PR gives xx. On macOS gcc is clang, so only the clang result is accepted there.

Not fixed

• defined() not recognizing ## token paste in its operand (GNU extension) #653, token pasting with "defined" works with gcc (cpp) but not simplecpp #218, Invalid ## usage when expanding  #300, handling of defined() conditions nested in function like macros #75: defined(a ## b) inside a macro used
• Do not abort when there is #ifdef in macro expansion list #572: a preprocessor directive inside macro arguments still gives an error.
• Token pasting/concatenation with ## leads to invalid result #373: __attribute()##8; still pastes )8.

Not confirmed

• fuzzing crash (stack-overflow) in simplecpp::Macro::expandToken() #662, fuzzing crash in Macro::expandToken() #414: the fuzzing inputs do not crash on master, with ken` code they pointed to is removed, but I cannot claim a fix.
• failed to expand BOOST_MPL_AUX_NA_SPEC, Wrong number of parameters for macro BOOST_PP_TUPLE_ELEM #159, simplecpp chokes on clang/master/INPUTS/stpcpy-test.c #96: not tested, they need the Boost headers or the full input from the issue.

Behaviour changes

• A # that comes from a macro at the start of a line gets a space before it, so the output is not read as a directive (as gcc and clang do). See # in macro left in original location #491.
• A function-like macro name at the end of the file, or withouo it is no longer an error.
• The tokens of an expansion get the location of the macro call.