GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-114053: Fix another edge case involving `get_type_hints`, PEP 695 and PEP 563 by AlexWaygood · Pull Request #120272 · python/cpython

Conversation

@AlexWaygood

@AlexWaygood AlexWaygood commented Jun 8, 2024 •
edited by bedevere-app Bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

This PR fixes a bug in #118009 that can be seen in the following REPL demo, where get_type_hints under from __future__ import annotations semantics won't detect if a variable defined in a type-parameter namespace has been overidden in a local namespace:

Python 3.14.0a0 (heads/getannotations-nameerror:ec4848f2c8b, Jun  8 2024, 15:28:33) [Clang 15.0.0 (clang-1500.3.9.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> class Foo[T]:
...     T = int
...     x: T
...     
>>> Foo.__annotations__
{'x': <class 'int'>}
>>> from __future__ import annotations
>>> class Foo[T]:
...     T = int
...     x: T
...     
>>> from typing import get_type_hints
>>> get_type_hints(Foo)
{'x': T}

The cause of the bug is that get_type_hints does this weird swapping of globals and locals, and the logic added in #118009 doesn't account for that -- it injects the type parameters into the dictionary called localns (which is actually the globals) when it should be injecting the type parameters into globalns (which is actually the locals).

I actually added a test for this exact edge case in #118009, which should have caught the bug. But there was a bug in the test, because the test was trying to be too clever, and I got mixed up between ^ and & when you're using sets. Had I not tried to write such a clever test, I would have caught the bug.

  • Issue: typing.get_type_hints fails when passed a class with PEP 695 type parameters and PEP 563 is enabled #114053

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@AlexWaygood
pythongh-114053: Fix another edge case involving get_type_hints, PE…
879abeb 
…P 695 and PEP 563
@AlexWaygood AlexWaygood added needs backport to 3.12 only security fixes needs backport to 3.13 only security fixes labels Jun 8, 2024
@AlexWaygood
AlexWaygood requested a review from JelleZijlstra as a code owner June 8, 2024 14:52
@bedevere-app bedevere-app Bot mentioned this pull request Jun 8, 2024
typing.get_type_hints fails when passed a class with PEP 695 type parameters and PEP 563 is enabled #114053
Closed
@bedevere-app bedevere-app Bot added the awaiting core review label Jun 8, 2024
@AlexWaygood AlexWaygood added the topic-typing label Jun 8, 2024
@AlexWaygood AlexWaygood mentioned this pull request Jun 8, 2024
gh-114053: Fix bad interaction of PEP 695, PEP 563 and inspect.get_annotations #120270
Merged
JelleZijlstra
JelleZijlstra reviewed Jun 9, 2024
View reviewed changes

@JelleZijlstra JelleZijlstra left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add a test where T also exists in the global scope, ensuring that the type param is picked up over the global?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
AlexWaygood reacted with thumbs up emoji
All reactions
  • 👍 1 reaction
AlexWaygood and others added 4 commits June 9, 2024 11:29
@AlexWaygood
Merge branch 'main' into gettypehints-bug
065914e
@AlexWaygood
Add failing test
373b23c
@AlexWaygood
Fix failing test
4f95b1c
@AlexWaygood
remove redundant assertions in test
ffdba91
@AlexWaygood

AlexWaygood commented Jun 9, 2024

Copy link
Copy Markdown
Member Author

Can you add a test where T also exists in the global scope, ensuring that the type param is picked up over the global?

Thanks, good catch -- that test failed. I added it and fixed it.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@AlexWaygood
AlexWaygood requested a review from JelleZijlstra June 9, 2024 11:39
JelleZijlstra
JelleZijlstra reviewed Jun 9, 2024
View reviewed changes
Comment thread Lib/typing.py Resolved

Uh oh!

There was an error while loading. Please reload this page.
AlexWaygood added 2 commits June 9, 2024 13:27
@AlexWaygood
Don't mutate globals!
5505d97
@AlexWaygood
Merge branch 'gettypehints-bug' of https://github.com/AlexWaygood/cpy…
b05ca1a 
…thon into gettypehints-bug
@AlexWaygood
AlexWaygood requested a review from JelleZijlstra June 9, 2024 12:29
@AlexWaygood
simplify
bccaea2
picnixz
picnixz reviewed Jun 9, 2024
View reviewed changes
Comment thread Lib/typing.py Resolved

Uh oh!

There was an error while loading. Please reload this page.
@AlexWaygood

AlexWaygood commented Jun 9, 2024 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member Author

Okay, this still doesn't get things right in all cases :(

>>> from __future__ import annotations
>>> from typing import get_type_hints
>>> Eggs, Spam = int, str
>>> def f[Eggs, **Spam](x: Eggs, y: Spam): pass
... 
>>> get_type_hints(f)
{'x': <class 'int'>, 'y': <class 'str'>}
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@AlexWaygood
Fix yet another bug and add many more tests
2fbcd7b
@AlexWaygood

AlexWaygood commented Jun 9, 2024

Copy link
Copy Markdown
Member Author

Okay, this still doesn't get things right in all cases :(

>>> from __future__ import annotations
>>> from typing import get_type_hints
>>> Eggs, Spam = int, str
>>> def f[Eggs, **Spam](x: Eggs, y: Spam): pass
... 
>>> get_type_hints(f)
{'x': <class 'int'>, 'y': <class 'str'>}

Should have been fixed in 2fbcd7b.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
JelleZijlstra
JelleZijlstra reviewed Jun 13, 2024
View reviewed changes
Comment thread Lib/typing.py
globalns, localns = dict(globalns), dict(localns)
for param in type_params:
param_name = param.__name__
if not self.__forward_is_class__ or param_name not in globalns:

@JelleZijlstra JelleZijlstra Jun 13, 2024

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need the __forward_is_class__ condition? I don't understand why class-scoped forwardrefs are special here.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions

@AlexWaygood AlexWaygood Jun 13, 2024 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For functions, names in type_params should always take highest priority, and this is implemented by inserting the type params into the local namespace (which is here called globalns) and popping them from the global namespace (which is here called localns). For classes, we only want to give type_params highest priority if they haven't been overridden in the local scope, however.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
Comment thread Lib/typing.py
param_name = param.__name__
if not self.__forward_is_class__ or param_name not in globalns:
globalns[param_name] = param
localns.pop(param_name, None)

@JelleZijlstra JelleZijlstra Jun 13, 2024

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need to do this? Shouldn't the local namespace override the type params?

Test case:

class X[T]:
    T = int
    x: T

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions

@AlexWaygood AlexWaygood Jun 13, 2024

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I already have that test case included in this PR, and it fails unless we have this block of code here. The local namespace shoudl indeed override the type param, but the locals here are found in the globalns, and the globals here are found in the localns variable. This highly confusing situation is because the globals and locals are swapped higher up in the call stack here, for backwards compatibility reasons if you believe the comment:

cpython/Lib/typing.py

Lines 2419 to 2426 in a3711af

if localns is None and globalns is None:
# This is surprising, but required. Before Python 3.10,
# get_type_hints only evaluated the globalns of
# a class. To maintain backwards compatibility, we reverse
# the globalns and localns order so that eval() looks into
# *base_globals* first rather than *base_locals*.
# This only affects ForwardRefs.
base_globals, base_locals = base_locals, base_globals

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions

@JelleZijlstra JelleZijlstra Jun 13, 2024

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I saw that comment, maybe we should find a way to get rid of that compatibility hack.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions

@AlexWaygood AlexWaygood Jun 13, 2024

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't really see a way round it, short of deprecating get_type_hints altogether. Which... I'm not 100% against, but it would be quite disruptive.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions

@JelleZijlstra JelleZijlstra Jun 13, 2024

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There could be other approaches, like adding new keyword arguments. Will have to think about it more.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
AlexWaygood reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

@picnixz picnixz Jun 14, 2024 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you decide to add keywords arguments or deprecate things, please tell me in advance so that I can be reactive on Sphinx' side if needed.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@AlexWaygood
Merge branch 'main' into gettypehints-bug
432e749
@AlexWaygood

AlexWaygood commented Jun 25, 2024

Copy link
Copy Markdown
Member Author

@JelleZijlstra, are you okay with this going in? I'd love to get this into the next 3.13 beta if possible (but at the same time, the bug also exists in Python 3.12, so it wouldn't be a disaster if it waited until beta 4).

I agree that the implementation is kinda weird but I'm not sure there's a better way of doing it until we've sorted out the strange globals/locals swapping (which I think would probably be harder to backport to 3.12).
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
JelleZijlstra
JelleZijlstra approved these changes Jun 25, 2024
View reviewed changes
@bedevere-app bedevere-app Bot added awaiting merge and removed awaiting core review labels Jun 25, 2024
@AlexWaygood
AlexWaygood merged commit 2d3187b into python:main Jun 25, 2024
@AlexWaygood
AlexWaygood deleted the gettypehints-bug branch June 25, 2024 15:53
@miss-islington-app

miss-islington-app Bot commented Jun 25, 2024

Copy link
Copy Markdown

Thanks @AlexWaygood for the PR 🌮🎉.. I'm working now to backport this PR to: 3.12, 3.13.
🐍🍒⛏🤖
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-app bedevere-app Bot removed the awaiting merge label Jun 25, 2024
miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Jun 25, 2024
@AlexWaygood @miss-islington
pythongh-114053: Fix another edge case involving get_type_hints, PE…
91ed17c 
…P 695 and PEP 563 (pythonGH-120272)

(cherry picked from commit 2d3187b)

Co-authored-by: Alex Waygood <[email protected]>
@bedevere-app

bedevere-app Bot commented Jun 25, 2024

Copy link
Copy Markdown

GH-121003 is a backport of this pull request to the 3.13 branch.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Jun 25, 2024
@AlexWaygood @miss-islington
pythongh-114053: Fix another edge case involving get_type_hints, PE…
62f3c3b 
…P 695 and PEP 563 (pythonGH-120272)

(cherry picked from commit 2d3187b)

Co-authored-by: Alex Waygood <[email protected]>
@bedevere-app bedevere-app Bot removed the needs backport to 3.13 only security fixes label Jun 25, 2024
@bedevere-app

bedevere-app Bot commented Jun 25, 2024

Copy link
Copy Markdown

GH-121004 is a backport of this pull request to the 3.12 branch.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-app bedevere-app Bot removed the needs backport to 3.12 only security fixes label Jun 25, 2024
AlexWaygood added a commit that referenced this pull request Jun 25, 2024
@miss-islington @AlexWaygood
[3.13] gh-114053: Fix another edge case involving get_type_hints, P…
f4f8a71 
…EP 695 and PEP 563 (GH-120272) (#121003)

Co-authored-by: Alex Waygood <[email protected]>
AlexWaygood added a commit that referenced this pull request Jun 25, 2024
@miss-islington @AlexWaygood
[3.12] gh-114053: Fix another edge case involving get_type_hints, P…
a85241d 
…EP 695 and PEP 563 (GH-120272) (#121004)

Co-authored-by: Alex Waygood <[email protected]>
mrahtz pushed a commit to mrahtz/cpython that referenced this pull request Jun 30, 2024
@AlexWaygood @mrahtz
pythongh-114053: Fix another edge case involving get_type_hints, PE…
7357059 
…P 695 and PEP 563 (python#120272)
estyxx pushed a commit to estyxx/cpython that referenced this pull request Jul 17, 2024
@AlexWaygood @estyxx
pythongh-114053: Fix another edge case involving get_type_hints, PE…
f62761c 
…P 695 and PEP 563 (python#120272)
JelleZijlstra added a commit to JelleZijlstra/cpython that referenced this pull request Jul 21, 2024
@JelleZijlstra
Reapply python#120272
1f3a63c
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@picnixz picnixz picnixz left review comments
@JelleZijlstra JelleZijlstra JelleZijlstra approved these changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

topic-typing

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

3 participants

@AlexWaygood @JelleZijlstra @picnixz
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.