Description

decorator-factory

opened on Dec 8, 2025

Issue body actions

Bug report

Bug description:

Since Python 3.13, \uXXXX escapes that create "surrogates" (values from \uD800 to \uDFFF which cannot be encoded into UTF-8) are not allowed in docstrings when compiling source code. I believe this is due to a change in #106411 where docstrings are first converted to UTF-8 and then dedented:

$ ./python
Python 3.15.0a2+ (heads/main:3db7bf2d18, Dec  8 2025, 09:38:46) [GCC 15.2.1 20251111 (Red Hat 15.2.1-4)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> def is_ex_parrot(parrot: Parrot) -> bool:
...     """Checks if the parrot is \udead"""
...
UnicodeEncodeError: 'utf-8' codec can't encode character '\udead' in position 24: surrogates not allowed
>>>
$ ./python -OO
Python 3.15.0a2+ (heads/main:3db7bf2d18, Dec  8 2025, 09:38:46) [GCC 15.2.1 20251111 (Red Hat 15.2.1-4)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> def is_ex_parrot(parrot: Parrot) -> bool:
...     """Checks if the parrot is \udead"""
...
>>>  # no error because -OO turns off docstrings

A admit that this is extremely fringe, but it did break something seemingly unrelated to docstrings in IPython: ipython/ipython#15098

The compile function is documented to only raise SyntaxError when the syntax is invalid and ValueError when the source contains \x00. Perhaps this is expected behaviour, and it should just be additionally documented? (UnicodeDecodeError is a subclass off ValueError already, so maybe IPython should just catch ValueError when calling compile? Or maybe there's simply some better solution for what IPython is doing here).

CPython versions tested on:

3.13, CPython main branch

Operating systems tested on:

Linux

Linked PRs

• gh-142411: Change documentation to reflect the new docstring adjustments in 3.13 #142413
• [3.14] gh-142411: Change documentation to reflect the new docstring adjustments in 3.13 (GH-142413) #142689
• [3.13] gh-142411: Change documentation to reflect the new docstring adjustments in 3.13 (GH-142413) #142690

Activity

1. decorator-factory

   added

   type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

   on Dec 8, 2025
2. decorator-factory

   changed the title [-]Surrogate literals in docstrings[/-] [+]Surrogate literals not allowed in docstrings since Python 3.13[/+] on Dec 8, 2025
3. gvanrossum commented on Dec 8, 2025

   

   gvanrossum

   on Dec 8, 2025

   Member

   More actions

   Looks like this should just be a doc update right? Since a docstring should be printable, lone surrogates don't make sense in that context.

   Reacted by decorator-factory
4. picnixz

   added

   docsDocumentation in the Doc dirDocumentation in the Doc dir

   and removed

   type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

   on Dec 8, 2025
5. github-project-automation

   added this to docs issueson Dec 8, 2025
6. github-project-automation

   moved this to Todo in docs issueson Dec 8, 2025
7. picnixz

   changed the title [-]Surrogate literals not allowed in docstrings since Python 3.13[/-] [+]Indicate that surrogate literals are not allowed in docstrings since Python 3.13[/+] on Dec 8, 2025
8. hugovk

   added a commit that references this issue on Dec 13, 2025

   gh-142411: Change documentation to reflect the new docstring adjustme…

   f6b6a99
9. decorator-factory

   added a commit that references this issue on Dec 14, 2025

   [3.14] pythongh-142411: Change documentation to reflect the new docst…

   1080708
10. hugovk

    added 2 commits that reference this issue on Dec 14, 2025

    [3.14] gh-142411: Change documentation to reflect the new docstring a…

    78dd2d5

    [3.13] gh-142411: Change documentation to reflect the new docstring a…

    d2ba756
11. hugovk

    closed this as completedon Dec 14, 2025
12. StanFromIreland

    marked compile() raises UnicodeEncodeError for docstrings with lone surrogates #149303 as a duplicate of this issue on May 3, 2026

Issue actions