GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-119180: PEP 649 compiler changes by JelleZijlstra · Pull Request #119361 · python/cpython

Conversation

@JelleZijlstra

@JelleZijlstra JelleZijlstra commented May 21, 2024 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

This implements the compiler changes in PEP-649, creating a compiler-generated __annotate__ function that holds the function, class, and module annotations.

I commented on Discuss on some of the CPython tests I needed to change: https://discuss.python.org/t/pep-649-deferred-evaluation-of-annotations-tentatively-accepted/21331/60

I tried this branch on a few prominent runtime typing projects; my notes are in the issue description (#119180). Mostly there are few issues, except a few places that need adjustment because they were checking for annotations in the __dict__ directly.

  • Issue: Implement PEP 649 and PEP 749 #119180

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
JelleZijlstra added 15 commits May 20, 2024 09:59
@JelleZijlstra
Add __annotate__ descriptors
985f8df
@JelleZijlstra
fix refleaks
c822ffa
@JelleZijlstra
blurb
e80095e
@JelleZijlstra
Fix some tests
90ff2c4
@JelleZijlstra
regen globals
026c0ff
@JelleZijlstra
Some initial work
7968744
@JelleZijlstra
Add bytecode for adding annotate
4e54197
@JelleZijlstra
compiler changes
4469b32
@JelleZijlstra
Functions work
47d672e
@JelleZijlstra
Raise AssertionError on the wrong format
ab9359c
@JelleZijlstra
Merge remote-tracking branch 'upstream/main' into pep649-compile
e65d55a
@JelleZijlstra
Modules and classes
e50cd62
@JelleZijlstra
attempts
afae5c0
@JelleZijlstra
Fixes
e5a7b1a
@JelleZijlstra
Merge remote-tracking branch 'upstream/main' into pep649-compile
3f26d44
@bedevere-app bedevere-app Bot mentioned this pull request May 21, 2024
Implement PEP 649 and PEP 749 #119180
Closed
21 of 30 tasks
JelleZijlstra added 14 commits May 21, 2024 17:00
@JelleZijlstra
Fix code object name
31a4471
@JelleZijlstra
Start fixing test_type_annotations
fbb1d88
@JelleZijlstra
Fix class scoping
f452eb2
@JelleZijlstra
unyielding
8c4b4e3
@JelleZijlstra
Fix test_typing
cbf9a3d
@JelleZijlstra
Fix test_type_parmas
5d182fc
@JelleZijlstra
Add test, add to inspect
ce98c19
@JelleZijlstra
Use inspect constants
e0578fc
@JelleZijlstra
test_grammar tweaks
ed16167
@JelleZijlstra
fix test_positional_only_arg
f38de20
@JelleZijlstra
Fix test_module
87baca2
@JelleZijlstra
Fix symtable tests
355d3df
@JelleZijlstra
fix test_pydoc
f9d81b6
@JelleZijlstra
fix test_traceback
dd1f64a
JelleZijlstra added 2 commits May 31, 2024 07:30
@JelleZijlstra
Merge remote-tracking branch 'upstream/main' into pep649-compile
e748feb
@JelleZijlstra
Merge remote-tracking branch 'upstream/main' into pep649-compile
a2b4f9e
@JelleZijlstra JelleZijlstra mentioned this pull request Jun 1, 2024
gh-119180: Add annotationlib module to support PEP 649 #119891
Merged
JelleZijlstra added 2 commits May 31, 2024 18:19
@JelleZijlstra
Still generate __annotate__ if "from __future__ import annotations" i…
bd469ab
…s on

This makes it so functools.update_wrapper can always copy __annotate__ without having
to worry about whether or not the future is enabled.
@JelleZijlstra
Regen globals
278de22
@JelleZijlstra JelleZijlstra mentioned this pull request Jun 3, 2024
Type parameters: Incorrect text in SyntaxError for disallowed expression #119933
Closed
@JelleZijlstra
Merge branch 'main' into pep649-compile
487ea34
@JelleZijlstra

JelleZijlstra commented Jun 7, 2024

Copy link
Copy Markdown
Member Author

Any further feedback on this PR? Getting this in will make it possible to start working on the Python-level parts of the PEP, which can be done in smaller chunks.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

AlexWaygood
AlexWaygood reviewed Jun 7, 2024
View reviewed changes

@AlexWaygood AlexWaygood left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The bits I feel qualified to comment on all look good to me!

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Lib/test/test_pydoc/test_pydoc.py
class B(builtins.object)
| Methods defined here:
|
| __annotate__(...)

@AlexWaygood AlexWaygood Jun 7, 2024 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(It would be great if we could generate a nice __text_signature__ for these generated methods, but that definitely doesn't need to be done in this PR)

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@JelleZijlstra JelleZijlstra Jun 7, 2024

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They're actually Python function, so I wonder why pydoc doesn't pick them up. Possibly because the parameter is called .format.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@JelleZijlstra JelleZijlstra Jun 8, 2024

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed:

>>> inspect.signature(f.__annotate__)
Traceback (most recent call last):
  File "<python-input-2>", line 1, in <module>
    inspect.signature(f.__annotate__)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
  File "/Users/jelle/py/cpython/Lib/inspect.py", line 3329, in signature
    return Signature.from_callable(obj, follow_wrapped=follow_wrapped,
           ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                                   globals=globals, locals=locals, eval_str=eval_str)
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jelle/py/cpython/Lib/inspect.py", line 3055, in from_callable
    return _signature_from_callable(obj, sigcls=cls,
                                    follow_wrapper_chains=follow_wrapped,
                                    globals=globals, locals=locals, eval_str=eval_str)
  File "/Users/jelle/py/cpython/Lib/inspect.py", line 2558, in _signature_from_callable
    return _signature_from_function(sigcls, obj,
                                    skip_bound_arg=skip_bound_arg,
                                    globals=globals, locals=locals, eval_str=eval_str)
  File "/Users/jelle/py/cpython/Lib/inspect.py", line 2403, in _signature_from_function
    parameters.append(Parameter(name, annotation=annotation,
                      ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                                kind=kind))
                                ^^^^^^^^^^
  File "/Users/jelle/py/cpython/Lib/inspect.py", line 2750, in __init__
    raise ValueError('{!r} is not a valid parameter name'.format(name))
ValueError: '.format' is not a valid parameter name

Maybe in a followup PR we can figure out a way to make this work. The parameter has to have an illegal name so it doesn't shadow any symbol name that the user might have used.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@AlexWaygood AlexWaygood Jun 8, 2024 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The parameter has to have an illegal name so it doesn't shadow any symbol name that the user might have used.

I'm not sure I 100% understand this point, and no tests seem to fail if I make this change locally to your PR branch and recompile (and I verified that it means that __annotate__ methods have valid signatures as per inspect.signature):

Details
diff --git a/Include/internal/pycore_global_strings.h b/Include/internal/pycore_global_strings.h
index 009802c4416..899b19ae941 100644
--- a/Include/internal/pycore_global_strings.h
+++ b/Include/internal/pycore_global_strings.h
@@ -45,7 +45,7 @@ struct _Py_global_strings {
         STRUCT_FOR_STR(dot, ".")
         STRUCT_FOR_STR(dot_locals, ".<locals>")
         STRUCT_FOR_STR(empty, "")
-        STRUCT_FOR_STR(format, ".format")
+        STRUCT_FOR_STR(format, "format")
         STRUCT_FOR_STR(generic_base, ".generic_base")
         STRUCT_FOR_STR(json_decoder, "json.decoder")
         STRUCT_FOR_STR(kwdefaults, ".kwdefaults")
diff --git a/Include/internal/pycore_runtime_init_generated.h b/Include/internal/pycore_runtime_init_generated.h
index ff5b6ee8e0f..e8b5173a457 100644
--- a/Include/internal/pycore_runtime_init_generated.h
+++ b/Include/internal/pycore_runtime_init_generated.h
@@ -554,7 +554,7 @@ extern "C" {
     INIT_STR(dot, "."), \
     INIT_STR(dot_locals, ".<locals>"), \
     INIT_STR(empty, ""), \
-    INIT_STR(format, ".format"), \
+    INIT_STR(format, "format"), \
     INIT_STR(generic_base, ".generic_base"), \
     INIT_STR(json_decoder, "json.decoder"), \
     INIT_STR(kwdefaults, ".kwdefaults"), \
diff --git a/Python/compile.c b/Python/compile.c
index c3372766d0b..7535067ded0 100644
--- a/Python/compile.c
+++ b/Python/compile.c
@@ -1447,7 +1447,7 @@ compiler_setup_annotations_scope(struct compiler *c, location loc,
     }
     c->u->u_metadata.u_posonlyargcount = 1;
     // if .format != 1: raise NotImplementedError
-    _Py_DECLARE_STR(format, ".format");
+    _Py_DECLARE_STR(format, "format");
     ADDOP_I(c, loc, LOAD_FAST, 0);
     ADDOP_LOAD_CONST(c, loc, _PyLong_GetOne());
     ADDOP_I(c, loc, COMPARE_OP, (Py_NE << 5) | compare_masks[Py_NE]);
diff --git a/Python/symtable.c b/Python/symtable.c
index 23fc4a0ec03..0154092a905 100644
--- a/Python/symtable.c
+++ b/Python/symtable.c
@@ -2511,7 +2511,7 @@ symtable_visit_annotation(struct symtable *st, expr_ty annotation, void *key)
             }
         }
 
-        _Py_DECLARE_STR(format, ".format");
+        _Py_DECLARE_STR(format, "format");
         // The generated __annotate__ function takes a single parameter with the
         // internal name ".format".
         if (!symtable_add_def(st, &_Py_STR(format), DEF_PARAM,
@@ -2570,7 +2570,7 @@ symtable_visit_annotations(struct symtable *st, stmt_ty o, arguments_ty a, expr_
             return 0;
         }
     }
-    _Py_DECLARE_STR(format, ".format");
+    _Py_DECLARE_STR(format, "format");
     // We need to insert code that reads this "parameter" to the function.
     if (!symtable_add_def(st, &_Py_STR(format), DEF_PARAM, LOCATION(o))) {
         return 0;

This also might be worth a brief mention in PEP-749, since PEP-649 seems to specify that the signature of __annotate__ should be __annotate__(format: int).

In any event, this can definitely be tackled in a followup; I don't want to block this PR!

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@JelleZijlstra JelleZijlstra Jun 8, 2024

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Try something like this:

class format: pass

def f(x: format): pass

print(f.__annotations__)

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@AlexWaygood AlexWaygood Jun 8, 2024

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see, makes sense! Worth adding a test like that?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@JelleZijlstra JelleZijlstra Jun 9, 2024

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, the problem is that any such test can only tell us about one specific name.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@AlexWaygood AlexWaygood Jun 9, 2024 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, it doesn't verify any fundamental invariants about __annotate__ — but I still think it's useful to have such a test, so that the CI goes obviously red if someone else comes along in the future and tries to do the "obvious fix" to get inspect.signature() working for these methods.

Thanks for adding it!

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
picnixz
picnixz reviewed Jun 8, 2024
View reviewed changes
Comment thread Lib/symtable.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Lib/test/test_grammar.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Misc/NEWS.d/next/Core and Builtins/2024-05-22-06-22-47.gh-issue-119180.vZMiXm.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Python/compile.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

JelleZijlstra added 4 commits June 8, 2024 06:15
@JelleZijlstra
Merge remote-tracking branch 'upstream/main' into pep649-compile
8f486ba
@JelleZijlstra
Feedback from Bénédikt Tran
6b563db
@JelleZijlstra
add test
0058b82
@JelleZijlstra
Merge remote-tracking branch 'upstream/main' into pep649-compile
517fb56
carljm
carljm approved these changes Jun 10, 2024
View reviewed changes
Comment thread Lib/inspect.py Outdated


@enum.global_enum
class AnnotationsFormat(enum.IntEnum):

@carljm carljm Jun 10, 2024

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How does this PR relate to PEP 749? Are you intending for this PR to implement PEP 649 as written, or the updated proposals in PEP 749?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@JelleZijlstra JelleZijlstra Jun 11, 2024

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will just take out this part; I only use the enum values in one test. The companion PR #119891 will implement PEP 749's proposed new module.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

carljm reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

@JelleZijlstra JelleZijlstra Jun 11, 2024

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And thanks for reviewing!

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

carljm reacted with thumbs up emoji
All reactions
  • 👍 1 reaction
Comment thread Lib/test/test_pyrepl/test_interact.py
def test_no_active_future(self):
console = InteractiveColoredConsole()
source = "x: int = 1; print(__annotations__)"
source = "x: int = 1; print(__annotate__(1))"

@carljm carljm Jun 10, 2024

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why does printing __annotations__ not work here? Shouldn't it call __annotate__ under the hood anyway?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@JelleZijlstra JelleZijlstra Jun 11, 2024

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Printing a global doesn't go through the descriptor protocol, so printing __annotations__ will just throw a NameError; it is added to the namespace only when the __annotate__ descriptor is called.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

carljm reacted with thumbs up emoji
All reactions
  • 👍 1 reaction
@bedevere-app bedevere-app Bot added awaiting merge and removed awaiting core review labels Jun 10, 2024
JelleZijlstra added 2 commits June 10, 2024 23:05
@JelleZijlstra
Merge remote-tracking branch 'upstream/main' into pep649-compile
21f93b6
@JelleZijlstra
Undo changes to inspect that we do not want
f72dbca
@JelleZijlstra
JelleZijlstra enabled auto-merge (squash) June 11, 2024 05:07
JelleZijlstra added 2 commits June 10, 2024 23:17
@JelleZijlstra
fix
8674eab
@JelleZijlstra
Fix more tests
ee11fd9
@JelleZijlstra
JelleZijlstra merged commit 9b8611e into python:main Jun 11, 2024
@bedevere-app bedevere-app Bot removed the awaiting merge label Jun 11, 2024
@JelleZijlstra
JelleZijlstra deleted the pep649-compile branch June 11, 2024 13:08
mrahtz pushed a commit to mrahtz/cpython that referenced this pull request Jun 30, 2024
@JelleZijlstra @mrahtz
pythongh-119180: PEP 649 compiler changes (python#119361)
39ab1f0
estyxx pushed a commit to estyxx/cpython that referenced this pull request Jul 17, 2024
@JelleZijlstra @estyxx
pythongh-119180: PEP 649 compiler changes (python#119361)
6042ef2
@tacaswell tacaswell mentioned this pull request Dec 12, 2024
Possible regression against CPython main (future 3.14) pydantic/pydantic#11097
Closed
1 task done
@lemenkov lemenkov mentioned this pull request Aug 2, 2025
Doesn't work with Python 3.14 (after PEP-649) biqqles/dataclassy#64
Open
@coderabbitai coderabbitai Bot mentioned this pull request Aug 18, 2026
Promote dev to main abhiksark/pythonlings#70
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@iritkatriel iritkatriel iritkatriel left review comments
@picnixz picnixz picnixz left review comments
@AlexWaygood AlexWaygood AlexWaygood left review comments
@carljm carljm carljm approved these changes
@ericsnowcurrently ericsnowcurrently Awaiting requested review from ericsnowcurrently ericsnowcurrently is a code owner
@markshannon markshannon Awaiting requested review from markshannon markshannon is a code owner
@gvanrossum gvanrossum Awaiting requested review from gvanrossum
@pablogsal pablogsal Awaiting requested review from pablogsal pablogsal is a code owner
@lysnikolaou lysnikolaou Awaiting requested review from lysnikolaou lysnikolaou is a code owner
@ambv ambv Awaiting requested review from ambv ambv is a code owner

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

5 participants

@JelleZijlstra @carljm @iritkatriel @picnixz @AlexWaygood
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.