GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-89263: Add typing.get_overloads by JelleZijlstra · Pull Request #31716 · python/cpython

Conversation

@JelleZijlstra

@JelleZijlstra JelleZijlstra commented Mar 7, 2022 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

Credit to Spencer Brown for the idea to use overloads' firstlineno to decide whether to clear the registry.

#89263

https://bugs.python.org/issue45100

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
TeamSpen210 reacted with eyes emoji
All reactions
  • 👀 1 reaction
JelleZijlstra added 2 commits March 5, 2022 21:24
@JelleZijlstra
initial
2ee377d
@JelleZijlstra
Implementation, tests, and docs
831b565
@JelleZijlstra
JelleZijlstra requested review from Fidget-Spinner, gvanrossum and rhettinger as code owners March 7, 2022 02:16
@bedevere-bot bedevere-bot added the awaiting core review label Mar 7, 2022
@the-knights-who-say-ni the-knights-who-say-ni added the CLA signed label Mar 7, 2022
JelleZijlstra
JelleZijlstra commented Mar 7, 2022
View reviewed changes
Comment thread Doc/library/functools.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.
@JelleZijlstra
fix versionadded
f03f8a9
AlexWaygood
AlexWaygood reviewed Mar 7, 2022
View reviewed changes

@AlexWaygood AlexWaygood left a comment •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! I have a few thoughts.

Overall, I guess I'm a little concerned about "dumping" so many new functions in the global functools namespace. functools is already kind of a hodgepodge of unrelated things, and I worry that this will make the problem worse.

Normally I'd say that having a class with 0 instance methods would be a code smell, but what about something like this?

class VariantRegistry:
    _registry = defaultdict(list)

    @staticmethod
    def _get_key_for_callable(func):
        func = getattr(func, "__func__", func)
        try:
            return f"{func.__module__}.{func.__qualname__}"
        except AttributeError:
            return None

    @classmethod
    def register_variant(func, variant):
        key = cls._get_key_for_callable(func)
        if key is not None:
            cls._variant_registry[key].append(variant)

    @classmethod
    def get_variants(cls, func):
        return cls._registry[cls._get_key_for_callable(func)]

    @classmethod
    def clear_variants(cls, func=None):
        if func is None:
           cls._variant_registry.clear()
        else:
            cls._variant_registry.pop(cls._get_key_for_callable(func), None)

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
Comment thread Lib/functools.py Resolved

Uh oh!

There was an error while loading. Please reload this page.
Comment thread Lib/functools.py Resolved

Uh oh!

There was an error while loading. Please reload this page.
Comment thread Lib/functools.py Resolved

Uh oh!

There was an error while loading. Please reload this page.
@JelleZijlstra
Merge branch 'main' into funcregistry
404668a
@JelleZijlstra

JelleZijlstra commented Mar 8, 2022

Copy link
Copy Markdown
Member Author

Thanks for the review!

I see the point about dumping, but I don't think using a class just as a bag of methods is a good solution. This group of functions serves a similar purpose to update_wrapper and wraps (enabling introspection for advanced usage of functions) and functools now feels like a good home for them.
AlexWaygood reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@JelleZijlstra
make get_key_for_callable private
7a5b0d1
AlexWaygood
AlexWaygood reviewed Mar 8, 2022
View reviewed changes
Comment thread Doc/library/functools.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.
AlexWaygood
AlexWaygood reviewed Mar 8, 2022
View reviewed changes
Comment thread Lib/test/test_typing.py Resolved

Uh oh!

There was an error while loading. Please reload this page.
AlexWaygood
AlexWaygood reviewed Mar 8, 2022
View reviewed changes
Comment thread Doc/library/functools.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.
AlexWaygood
AlexWaygood reviewed Mar 8, 2022
View reviewed changes

@AlexWaygood AlexWaygood left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What happens if a user does something weird (and probably misguided?) like this?

@overload
def complex_func(arg: str) -> int: ...

@overload
def complex_func(arg: int) -> str: ...

@singledispatch
def complex_func(arg: object):
    raise NotImplementedError

@complex_func.register
def _(arg: str) -> int: 
    return int(arg)

@complex_func.register
def _(arg: int) -> str:
    return str(arg)

What's the runtime behaviour in this case? Do we care? Do we warn users not to do this in the docs? Do we add a test case to make sure the behaviour is "correct"?

I think we need to account for this possible (mis)usage somehow, even if type checkers would probably balk at it.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@JelleZijlstra
doc updates; remove unnecessary try-except
6998255
@JelleZijlstra
JelleZijlstra marked this pull request as draft March 9, 2022 04:56
JelleZijlstra added 6 commits March 27, 2022 15:42
@JelleZijlstra
Merge remote-tracking branch 'upstream/main' into funcregistry
26bb908
@JelleZijlstra
rename method
f52b757
@JelleZijlstra
Don't store singledispatch in the registry
fc6a925
@JelleZijlstra
more tests
b524244
@JelleZijlstra
and another
e95558e
@JelleZijlstra
fix line length in new tests
31fd72d
@JelleZijlstra

JelleZijlstra commented Mar 27, 2022

Copy link
Copy Markdown
Member Author

What's the runtime behaviour in this case? Do we care? Do we warn users not to do this in the docs? Do we add a test case to make sure the behaviour is "correct"?

I think we need to account for this possible (mis)usage somehow, even if type checkers would probably balk at it.

We simply get variants both from overload and singledispatch. I added a test case based on this example.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
JelleZijlstra
JelleZijlstra commented Mar 27, 2022
View reviewed changes
Comment thread Doc/library/functools.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.
@JelleZijlstra
Update Doc/library/functools.rst
7041ad3
JelleZijlstra
JelleZijlstra commented Mar 27, 2022
View reviewed changes
Comment thread Doc/library/typing.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.
@JelleZijlstra
Update Doc/library/typing.rst
e26b0db
@JelleZijlstra
Simpler implementation (thanks Guido)
103bfd4
@JelleZijlstra

JelleZijlstra commented Apr 9, 2022

Copy link
Copy Markdown
Member Author

Implemented this approach. I now see a 5x performance ratio:

In [9]: %timeit overload(f)
223 ns ± 1.85 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)

In [10]: %timeit old_overload(f)
44.7 ns ± 0.74 ns per loop (mean ± std. dev. of 7 runs, 10,000,000 loops each)

In [11]: 223 / 44.7
Out[11]: 4.988814317673378
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@JelleZijlstra
JelleZijlstra removed the request for review from rhettinger April 9, 2022 01:55
gvanrossum
gvanrossum reviewed Apr 9, 2022
View reviewed changes
Comment thread Lib/test/test_typing.py
Comment on lines +3922 to +3928
impl, overloads = self.set_up_overloads()
self.assertNotEqual(typing._overload_registry, {})
self.assertEqual(list(get_overloads(impl)), overloads)

clear_overloads()
self.assertEqual(typing._overload_registry, {})
self.assertEqual(get_overloads(impl), [])

@gvanrossum gvanrossum Apr 9, 2022

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add a comment explaining why this sequence is repeated?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions

@JelleZijlstra JelleZijlstra Apr 9, 2022

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. Also added a test case to make sure that clearing one function's overloads doesn't clear another.

Thinking out loud, clear_overloads() is becoming the most complicated part of this. Perhaps we don't need to support clearing overloads per function, which would simplify it greatly. Use cases I can think of are:

  • You want to save some memory after importing all your code -> you call clear_overloads() and get rid of them all.
  • You are constantly reloading a module with overloads in it and changing the line numbers. Then maybe you want to clear_overloads() for just that function. But that seems really exotic.

So maybe we should only support clearing the whole overload registry at once. Can you think of other realistic reasons people may want to use clear_overloads()?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions

@gvanrossum gvanrossum Apr 9, 2022

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm. Given the reload(module) use case, maybe the only options should be clear all or clear a specific module? The new data structure makes that easy. The question would then be whether the argument should be the (full) module name or the module object? (Or either?) The sequence of events would be something like

typing.clear_overload(mod)
importlib.reload(mod)

I can't really think of a use case for clearing a specific function. Did you have one in mind originally? Or was it just convenient for testing?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions

@JelleZijlstra JelleZijlstra Apr 14, 2022

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think for now we should allow only clearing the whole registry. I see the use case for wanting to repeatedly reload modules while wanting to introspect overloads, but it seems unlikely to be common. We can always add an argument later to clear overloads by module if someone asks for it, but once we add it we're stuck with it.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@AlexWaygood
AlexWaygood self-requested a review April 9, 2022 04:34
@JelleZijlstra
More comments and tests
d453f7f
@AlexWaygood AlexWaygood added topic-typing type-feature A feature request or enhancement labels Apr 13, 2022
@JelleZijlstra JelleZijlstra changed the title bpo-45100: Add typing.get_overloads gh-89263: Add typing.get_overloads Apr 14, 2022
JelleZijlstra added 2 commits April 13, 2022 19:00
@JelleZijlstra
Merge remote-tracking branch 'upstream/main' into funcregistry
450afeb
@JelleZijlstra
simplify clear_overloads
ea62287
@JelleZijlstra

JelleZijlstra commented Apr 14, 2022

Copy link
Copy Markdown
Member Author

@gvanrossum @AlexWaygood any further feedback? The last thing to resolve is whether clear_overloads should take arguments (#31716 (comment)).
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
AlexWaygood
AlexWaygood reviewed Apr 14, 2022
View reviewed changes

@AlexWaygood AlexWaygood left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks pretty good now! Just a few thoughts:

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
Comment thread Doc/library/typing.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.
Comment thread Lib/typing.py Resolved

Uh oh!

There was an error while loading. Please reload this page.
Comment thread Lib/typing.py Resolved

Uh oh!

There was an error while loading. Please reload this page.
Comment thread Lib/test/test_typing.py Resolved

Uh oh!

There was an error while loading. Please reload this page.
Comment thread Doc/library/typing.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.
Comment thread Doc/library/typing.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.
Comment thread Doc/library/typing.rst
Comment on lines +2418 to +2419
*func*. *func* is the function object for the implementation of the
overloaded function. For example, given the definition of ``process`` in

@AlexWaygood AlexWaygood Apr 14, 2022

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This sentence sounds quite strange to me:

*func* is the function object for the implementation of the overloaded function.

But I'm not sure I have a better suggestion off the top of my head ://

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions

@JelleZijlstra JelleZijlstra Apr 14, 2022

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, unfortunately the overload docs don't have a clear term for the implementation function either.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
AlexWaygood reacted with confused emoji
All reactions
  • 😕 1 reaction
@gvanrossum

gvanrossum commented Apr 14, 2022

Copy link
Copy Markdown
Member

I'll wait until @AlexWaygood approves, then I'll do one more pass if you want me to.
AlexWaygood reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
JelleZijlstra added 4 commits April 14, 2022 09:41
@JelleZijlstra
use partial
905253c
@JelleZijlstra
add test
debbf8a
@JelleZijlstra
docs changes (thanks Alex)
754c134
@JelleZijlstra
Merge remote-tracking branch 'upstream/main' into funcregistry
1ad8224
AlexWaygood
AlexWaygood reviewed Apr 14, 2022
View reviewed changes
Comment thread Doc/library/typing.rst
the documentation for :func:`@overload <overload>`,
``get_overloads(process)`` will return a sequence of three function objects
for the three defined overloads. If called on a function with no overloads,
``get_overloads`` returns an empty sequence.

@AlexWaygood AlexWaygood Apr 14, 2022

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
``get_overloads`` returns an empty sequence.
return an empty sequence.

Tiny nit -- this paragraph starts off using the imperative mood

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions

@JelleZijlstra JelleZijlstra Apr 14, 2022

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But the previous sentence is in the indicative, so I think this is fine?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions

@AlexWaygood AlexWaygood Apr 14, 2022

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, it's not really a big deal either way :)

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
AlexWaygood
AlexWaygood approved these changes Apr 14, 2022
View reviewed changes

@AlexWaygood AlexWaygood left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hooray!

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@AlexWaygood AlexWaygood added the stdlib Standard Library Python modules in the Lib/ directory label Apr 14, 2022
@JelleZijlstra

JelleZijlstra commented Apr 16, 2022

Copy link
Copy Markdown
Member Author

@gvanrossum would you like to take another look or should I merge this?
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@AlexWaygood AlexWaygood mentioned this pull request Apr 16, 2022
Improve help() by making typing.overload() information accessible at runtime #89263
Open
@gvanrossum

gvanrossum commented Apr 16, 2022

Copy link
Copy Markdown
Member

Go for it!
JelleZijlstra reacted with thumbs up emoji AlexWaygood reacted with hooray emoji AlexWaygood reacted with rocket emoji
All reactions
  • 👍 1 reaction
  • 🎉 1 reaction
  • 🚀 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@JelleZijlstra
JelleZijlstra merged commit 055760e into python:main Apr 16, 2022
@bedevere-bot bedevere-bot removed the awaiting core review label Apr 16, 2022
@JelleZijlstra
JelleZijlstra deleted the funcregistry branch April 16, 2022 16:01
@JelleZijlstra JelleZijlstra mentioned this pull request Apr 16, 2022
Add get_overloads() python/typing#1140
Merged
@TeamSpen210 TeamSpen210 mentioned this pull request Apr 16, 2022
[Feature Request] Support typing.get_overloads() under Python ≥ 3.11 beartype/beartype#54
Open
@AlexWaygood AlexWaygood mentioned this pull request Sep 21, 2023
gh-109653: typing.py: improve import time by creating soft-deprecated members on demand #109651
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gvanrossum gvanrossum gvanrossum left review comments
@Fidget-Spinner Fidget-Spinner Fidget-Spinner left review comments
@AlexWaygood AlexWaygood AlexWaygood approved these changes
+1 more reviewer
@TeamSpen210 TeamSpen210 TeamSpen210 left review comments
Reviewers whose approvals may not affect merge requirements

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

stdlib Standard Library Python modules in the Lib/ directory topic-typing type-feature A feature request or enhancement

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

7 participants

@JelleZijlstra @AlexWaygood @gvanrossum @Fidget-Spinner @TeamSpen210 @the-knights-who-say-ni @bedevere-bot
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402520 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.