GeezerVison

If user's __repr__ contains color, it would break the color.

import os
import pprint

os.environ["FORCE_COLOR"] = "1"
os.environ["NO_COLOR"] = ""


class Colorful:
    """something with a colorful repr"""

    def __repr__(self):
        return "\x1b[31mred\x1b[0m"

pprint.pprint({"a": Colorful(), "b": 42, "c": "hello"}, color=True)

What should we do for this case?

1. Document and keep as is?
2. Pre check escape sequence before colorize?
3. Other techniques for this?

2. Pre check escape sequence before colorize?

I think if there's any colour escapes, we bail and print it as-is without extra colour from us. So we get something like this:

And I expect colour is rare in users' __repr__s? If so, this won't happen often, and when it does, we don't override any of their own colour or need more complex handling. And they don't end up with just a green "a" like in your example.

Documentation build overview

📚 cpython-previews | 🛠️ Build #32544585 | 📁 Comparing f6c4dd9 against main (4b33308)

  🔍 Preview build  

I think if there's any colour escapes, we bail and print it as-is without extra colour from us.

I agree with this.

But thinking further, our current highlighting logic relies on tokenize, yet pprint output has never made any guarantee that it must be valid Python source. __repr__ can return anything — it doesn't have to be tokenizer-friendly.

So, instead of special-casing \x1b[, how about

Probe the text with tokenize first,
-- if it can process the whole thing, highlight it;
-- if it chokes halfway through for any reason (ANSI escapes, illegal control chars, etc.), just print the raw text with no added colors.

No blacklist needed, no half-highlighted output.

What do you think?

 def _colorize_output(text):
     """Apply syntax highlighting."""
-    if "\x1b[" in text:
-        # If the text already contains ANSI escape sequences
-        # (for example, from a custom __repr__),
-        # return as-is to avoid breaking their color.
+    # Probe: if the text cannot be fully tokenized, fall back to plain text
+    # to avoid the "half-highlighted" problem.
+    try:
+        list(tokenize.generate_tokens(_StringIO(text).readline))
+    except (SyntaxError, tokenize.TokenError):
         return text
     colors = list(gen_colors(text))
     chars, _ = disp_str(text, colors=colors, force_color=True, escape=False)

Could do, although that'd mean we're potentially doing tokenize.generate_tokens() both here and in gen_colors(). Maybe that doesn't matter.

Maybe that doesn't matter.

If that matters, we could modify the latter to raise errors (rather than catch them)

I think I'll leave it like this, unless others also want to suggest something. We can always fix it later if needed. Thanks for the review!

Should this tokenize reprs that start with <? In the following, highlighting the in and from doesn't make much sense.

Updated to skip those.

That's inconsistent when nested:

Would it be possible to highlight as things are being formatted, while the structure is still available? I doubt post-processing can be done reliably.

Hmm, the root issue is that gen_colors tokenises as if it were Python source, so I think it would quite a big refactor to do that, at least to make it into 3.15.

I'm personally fine with the current version or the previous, but also fine in skipping this for 3.15 and doing it properly for 3.16 (or just closing :)

Hmm, the root issue is that gen_colors tokenises as if it were Python source

Yeah. That guess might not be right; let's resist the temptation.
Colorization is much more helpful when you can trust it.

In 3.16 a pprint refactor might be needed anyway, for PEP-813.

This PR is stale because it has been open for 90 days with no activity.