Description

AlexWaygood

opened on Mar 15, 2023

Issue body actions

There is currently a difference in behaviour between collections.abc.Callable and typing.Callable for the following edge case involving ParamSpec substitution:

Running PGUpdate|x64 interpreter...
Python 3.12.0a6+ (heads/main:12226bec25, Mar 10 2023, 17:32:23) [MSC v.1932 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import typing as t, collections.abc as c
>>> P, T = t.ParamSpec("P"), t.TypeVar("T")
>>> t.Callable[P, T][[P, str], bool][int]
typing.Callable[[int, str], bool]
>>> c.Callable[P, T][[P, str], bool][int]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<frozen _collections_abc>", line 501, in __getitem__
  File "<frozen _collections_abc>", line 456, in __new__
TypeError: Callable must be used as Callable[[arg, ...], result].

According to PEP-612, this is an invalid substitution, for two reasons:

1. Parameters can only be prepended to a ParamSpec in a parameters list using typing(_extensions).Concatenate.
2. Appending parameters to a ParamSpec in a parameters list is disallowed.

As such, the behaviour of collections.abc.Callable is more correct here, so ideally we'd change the behaviour of typing.Callable to match collections.abc.Callable.

However, this error should hopefully be caught by static type checkers anyway, and this is a false negative rather than a false positive. The runtime makes no promises that it will raise TypeError on all invalid substitutions, so fixing this should be low priority, in my opinion. We should first concentrate on fixing substitutions where the runtime raises exceptions, even though it shouldn't. For example:

• Substitution does not work after ParamSpec substitution of the user generic with a list of TypeVars #88965

This discrepancy in behaviour was first uncovered as part of a broader discussion in:

• Representation of ParamSpecs at runtime compared to Callable #102615

Cc. @sobolevn

Activity

1. AlexWaygood

   added

   type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

   stdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directory

   topic-typing

   3.12only security fixesonly security fixes

   on Mar 15, 2023
2. sobolevn

   self-assigned this

   on Mar 16, 2023
3. sobolevn commented on Mar 16, 2023

   

   sobolevn

   on Mar 16, 2023

   Member

   More actions

   I am going to research, how hard it is to change.

   If this is a rather simple patch, I am going to apply it.
   If this is a complex one, I will close this issue as "won't fix".

   Because parameters and their substitution is a very hard thing to get right.

   Reacted by Alex Waygood
4. sobolevn commented on Mar 16, 2023

   

   sobolevn

   on Mar 16, 2023

   Member

   More actions

   Wait, I am confused again. Why c.Callable[P, T][[P, str], bool] is allowed?

   It seems like P can only be substituted with:

   1. Some other ParamSpec variable: P -> P2
   2. ...
   3. Arguments list without any ParamSpec variables: [int, str]
   4. Concatenate with ParamSpec: Concatenate[int, P]

   Am I missing something?
5. AlexWaygood commented on Mar 16, 2023

   

   AlexWaygood

   on Mar 16, 2023

   MemberAuthor

   More actions

   You're correct, c.Callable[P, T][[P, str], bool] should also, strictly speaking, not be allowed.

   Honestly, I think we should probably leave this one. The runtime allows all sorts of crazy things that should be rejected by type checkers. My personal favourite (we should make this an alias for Any):

   >>> list[r"¯\_(ツ)_/¯"]
   list['¯\\_(ツ)_/¯']

   We've never promised that the runtime would disallow all things that should be rejected by type checkers, and I don't want us to start making that promise now.

   The fact that c.Callable[P, T][[P, str], bool][int] raises a TypeError is, strictly speaking, correct -- but I think it's almost certainly just an accident of the current implementation rather than something that was designed. So we shouldn't spend time trying to change the behaviour (since the behaviour is not incorrect), but we also probably shouldn't spend time trying to make typing.Callable match it.

   #88965 seems like a much more important issue to me, since that involves the runtime incorrectly raising a TypeError for valid substitutions.
6. sobolevn commented on Mar 16, 2023

   

   sobolevn

   on Mar 16, 2023

   Member

   More actions

   Yes, I agree. We should close this as won't fix. Because the correct behaviour is very complex to achieve. And will probably break other things.

   I am going to research, how hard it is to change.

   There's no point to address a minor issue like this one by the cost of increasing code complexity.

   Reacted by Alex Waygood
7. sobolevn

   closed this as not plannedon Mar 16, 2023
8. AlexWaygood commented on Mar 16, 2023

   

   AlexWaygood

   on Mar 16, 2023

   MemberAuthor

   More actions

   Would you be interested in taking a look at #88965? Serhiy assigned it to himself several months ago, but there's been no activity on the issue since, and it would be great to have it fixed.
9. sobolevn commented on Mar 16, 2023

   

   sobolevn

   on Mar 16, 2023

   Member

   More actions

   I will tomorrow!

   Reacted by Alex Waygood

Issue actions