GeezerVison

The World Wide Web, restored to 800 pixels of glory.

GeezerVison is showing github.com — improvement: Add AST subclass constructors by bzoracler · Pull Request #11880 · python/typeshed

Conversation

bzoracler commented May 9, 2024 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link

Copy Markdown

Contributor

Fixes #8378

ast.AST no longer owns attributes which may not exist on all subclasses (node locations, type_comment). These are all shifted to subclasses which actually own the attributes (d5e0838, 75a2e8e, f9a2cb9)
ast.AST no longer has the constructor __init__(self, *args: Any, **kwargs: Any).
All subclasses of ast.AST which have a non-empty _fields: ClassVar[tuple[str, ...]] or _attributes: ClassVar[tuple[str, ...]] now have their own __init__ constructor (d886a45). The constructor has:
- Positional-or-keyword arguments in the order in which they appear in _fields, and
- Non-mandatory keyword-only arguments from _attributes. These are expressed using a combination of typing.TypedDict and typing_extensions.Unpack.
  - These are non-mandatory because I expect most users won't pass explicit locations, but rather use something like ast.fix_missing_locations.

Updates:

Arguments of type ast.expr_context are not required to be passed to the constructor, but they will not exist on node instances for Python < 3.13.
Arguments of type list are not required to be passed to the constructor for Python >= 3.13.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

bzoracler added 5 commits

May 9, 2024 23:56


          improvement: shift node location attributes to classes which have them

d5e0838


          Update _ast.pyi

75a2e8e

improvement: shift `type_comment` attribute to classes which contain it


          improvement: add missing node field

f9a2cb9


          improvement: add precise AST subclass constructors

d886a45


          test: update allowlist

25d4dad

This comment has been minimized.

bzoracler added 2 commits

May 10, 2024 00:33


          fix: delegate version-specific nodes to separate allowlists

70fdb52


          fix: typo in parameter

c88c4aa

AlexWaygood reviewed

May 9, 2024

View reviewed changes

Comment thread

stdlib/_ast.pyi Resolved

Uh oh!

There was an error while loading. Please reload this page.

AlexWaygood reviewed

May 9, 2024

View reviewed changes

Comment thread

stdlib/_ast.pyi Resolved

Uh oh!

There was an error while loading. Please reload this page.

This comment has been minimized.


          style: Shorten signatures

1135b28

AlexWaygood mentioned this pull request

May 9, 2024

Allow using typing_extensions.TypeVar in stubs PyCQA/flake8-pyi#486

Merged

This comment has been minimized.

JelleZijlstra commented May 9, 2024

Copy link

Copy Markdown

Member

Thanks for this, I had been thinking of adding it but hadn't gotten around to it.

bzoracler reacted with thumbs up emoji

All reactions

👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

JelleZijlstra mentioned this pull request

May 9, 2024

Default expr_context parameters in ast to Load() python/cpython#118851

Closed

JelleZijlstra reviewed

May 9, 2024

View reviewed changes

Comment thread

stdlib/_ast.pyi Outdated

		__match_args__ = ("body", "type_ignores")
		body: list[stmt]
		type_ignores: list[TypeIgnore]
		def __init__(self, body: list[stmt], type_ignores: list[TypeIgnore]) -> None: ...

JelleZijlstra May 9, 2024

Copy link

Copy Markdown

Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These list fields should default to the empty list on 3.13+.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

bzoracler May 10, 2024

Copy link

Copy Markdown

Contributor Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in c82a5fa

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

Comment thread

stdlib/_ast.pyi

		decorator_list: list[expr],
		returns: expr \| None = None,
		*,
		type_comment: str \| None = None,

JelleZijlstra May 9, 2024

Copy link

Copy Markdown

Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think type_comment needs to be keyword-only on this overload.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

bzoracler reacted with thumbs up emoji

All reactions

👍 1 reaction

bzoracler May 10, 2024

Copy link

Copy Markdown

Contributor Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 9655775

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

bzoracler and others added 5 commits

May 10, 2024 17:34


          fix: remove unnecessary keyword-only constraint


          improvement: don't require args of type expr_context to be passed t…

bb880f2

…o constructor


          feat: empty list defaults in Python 3.13

c82a5fa


          style: keep _SliceAttributes together with usage location

aea4c01


          [pre-commit.ci] auto fixes from pre-commit.com hooks

1e489b1

This comment has been minimized.

bzoracler commented May 10, 2024 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link

Copy Markdown

Contributor Author

I didn't add these other changes from Python 3.13 runtime to this PR:

AST._field_types - not sure what the intention is for the values of these dictionaries; are they meant to be for runtime introspection, or can they be anything?
The default_value parameter for ast.TypeVar, ast.ParamSpec, ast.TypeVarTuple - the runtime suggests that this defaults to None if not passed, but the grammar does not mark this field as optional. (I misread the grammar, they're definitely optional)

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.


          fix: add missing default_value field for Python 3.13

b82fc4e

This comment has been minimized.

JelleZijlstra commented May 10, 2024

Copy link

Copy Markdown

Member

AST._field_types - not sure what the intention is for the values of these dictionaries; are they meant to be for runtime introspection, or can they be anything?

It's used by the constructor of AST nodes to figure out what defaults to use. I did document it, so I think we should add it to the ast.AST class as ClassVar[dict[str, Any]].

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.


          improvement: add new Python 3.13 class variable

cc50db4

This comment has been minimized.


          fix: wrong attribute conditional location

cc2cd65

github-actions Bot commented May 10, 2024

Copy link

Copy Markdown

Contributor

Diff from mypy_primer, showing the effect of this PR on open source code:

pyp (https://github.com/hauntsaninja/pyp)
+ pyp.py:585: error: "AST" has no attribute "lineno"  [attr-defined]
+ pyp.py:586: error: "AST" has no attribute "end_lineno"  [attr-defined]

sphinx (https://github.com/sphinx-doc/sphinx)
+ sphinx/ext/autodoc/preserve_defaults.py: note: In function "get_default_value":
+ sphinx/ext/autodoc/preserve_defaults.py:114:12: error: "AST" has no attribute "lineno"  [attr-defined]
+ sphinx/ext/autodoc/preserve_defaults.py:114:31: error: "AST" has no attribute "end_lineno"  [attr-defined]
+ sphinx/ext/autodoc/preserve_defaults.py:115:26: error: "AST" has no attribute "lineno"  [attr-defined]
+ sphinx/ext/autodoc/preserve_defaults.py:116:25: error: "AST" has no attribute "col_offset"  [attr-defined]
+ sphinx/ext/autodoc/preserve_defaults.py:116:45: error: "AST" has no attribute "end_col_offset"  [attr-defined]

mypy (https://github.com/python/mypy)
+ mypy/fastparse.py:1998: error: "Index" has no attribute "col_offset"  [attr-defined]
+ mypy/fastparse.py:2001: error: "Slice" has no attribute "col_offset"  [attr-defined]
+ mypy/fastparse.py:2002: error: Argument 1 to "Tuple" has incompatible type "List[slice]"; expected "List[expr]"  [arg-type]

pytest (https://github.com/pytest-dev/pytest)
+ src/_pytest/assertion/rewrite.py:839: error: Argument 1 to "Dict" has incompatible type "list[Constant]"; expected "list[expr | None]"  [arg-type]
+ src/_pytest/assertion/rewrite.py:839: note: "List" is invariant -- see https://mypy.readthedocs.io/en/stable/common_issues.html#variance
+ src/_pytest/assertion/rewrite.py:839: note: Consider using "Sequence" instead, which is covariant
+ src/_pytest/assertion/rewrite.py:932: error: Argument 3 to "If" has incompatible type "list[If]"; expected "list[stmt]"  [arg-type]
+ src/_pytest/assertion/rewrite.py:932: note: "List" is invariant -- see https://mypy.readthedocs.io/en/stable/common_issues.html#variance
+ src/_pytest/assertion/rewrite.py:932: note: Consider using "Sequence" instead, which is covariant
+ src/_pytest/assertion/rewrite.py:938: error: Argument 1 to "Assign" has incompatible type "list[Name]"; expected "list[expr]"  [arg-type]
+ src/_pytest/assertion/rewrite.py:938: note: "List" is invariant -- see https://mypy.readthedocs.io/en/stable/common_issues.html#variance
+ src/_pytest/assertion/rewrite.py:938: note: Consider using "Sequence" instead, which is covariant
+ src/_pytest/assertion/rewrite.py:963: error: Argument 1 to "Assign" has incompatible type "list[Name]"; expected "list[expr]"  [arg-type]
+ src/_pytest/assertion/rewrite.py:963: note: "List" is invariant -- see https://mypy.readthedocs.io/en/stable/common_issues.html#variance
+ src/_pytest/assertion/rewrite.py:963: note: Consider using "Sequence" instead, which is covariant
+ src/_pytest/assertion/rewrite.py:1145: error: Argument 1 to "Tuple" has incompatible type "list[Constant]"; expected "list[expr]"  [arg-type]
+ src/_pytest/assertion/rewrite.py:1145: note: "List" is invariant -- see https://mypy.readthedocs.io/en/stable/common_issues.html#variance
+ src/_pytest/assertion/rewrite.py:1145: note: Consider using "Sequence" instead, which is covariant
+ src/_pytest/assertion/rewrite.py:1146: error: Argument 1 to "Tuple" has incompatible type "list[Name]"; expected "list[expr]"  [arg-type]
+ src/_pytest/assertion/rewrite.py:1146: note: "List" is invariant -- see https://mypy.readthedocs.io/en/stable/common_issues.html#variance
+ src/_pytest/assertion/rewrite.py:1146: note: Consider using "Sequence" instead, which is covariant
+ src/_pytest/assertion/rewrite.py:1147: error: Argument 1 to "Tuple" has incompatible type "list[Constant]"; expected "list[expr]"  [arg-type]
+ src/_pytest/assertion/rewrite.py:1147: note: "List" is invariant -- see https://mypy.readthedocs.io/en/stable/common_issues.html#variance
+ src/_pytest/assertion/rewrite.py:1147: note: Consider using "Sequence" instead, which is covariant
+ src/_pytest/assertion/rewrite.py:1151: error: Argument 2 to "BoolOp" has incompatible type "list[Name]"; expected "list[expr]"  [arg-type]
+ src/_pytest/assertion/rewrite.py:1151: note: "List" is invariant -- see https://mypy.readthedocs.io/en/stable/common_issues.html#variance
+ src/_pytest/assertion/rewrite.py:1151: note: Consider using "Sequence" instead, which is covariant
+ testing/test_assertrewrite.py:133: error: "AST" has no attribute "lineno"  [attr-defined]
+ testing/test_assertrewrite.py:134: error: "AST" has no attribute "col_offset"  [attr-defined]
+ testing/test_assertrewrite.py:135: error: "AST" has no attribute "end_lineno"  [attr-defined]
+ testing/test_assertrewrite.py:136: error: "AST" has no attribute "end_col_offset"  [attr-defined]

werkzeug (https://github.com/pallets/werkzeug)
+ src/werkzeug/routing/rules.py:785: error: Argument 1 to "JoinedStr" has incompatible type "list[AST]"; expected "list[expr]"  [arg-type]
+ src/werkzeug/routing/rules.py:788: error: List item 0 has incompatible type "AST"; expected "expr"  [list-item]
+ src/werkzeug/routing/rules.py:788: error: List item 1 has incompatible type "AST"; expected "expr"  [list-item]
+ src/werkzeug/routing/rules.py:818: error: "AST" has no attribute "lineno"  [attr-defined]
+ src/werkzeug/routing/rules.py:820: error: "AST" has no attribute "end_lineno"  [attr-defined]
+ src/werkzeug/routing/rules.py:820: error: "AST" has no attribute "lineno"  [attr-defined]
+ src/werkzeug/routing/rules.py:822: error: "AST" has no attribute "col_offset"  [attr-defined]
+ src/werkzeug/routing/rules.py:824: error: "AST" has no attribute "end_col_offset"  [attr-defined]
+ src/werkzeug/routing/rules.py:824: error: "AST" has no attribute "col_offset"  [attr-defined]

ibis (https://github.com/ibis-project/ibis)
+ ibis/backends/bigquery/udf/core.py:66: error: Missing positional arguments "args", "keywords" in call to "Call"  [call-arg]
+ ibis/backends/bigquery/udf/core.py:79: error: Missing positional arguments "args", "keywords" in call to "Call"  [call-arg]
+ ibis/backends/bigquery/udf/core.py:85: error: Missing positional arguments "args", "keywords" in call to "Call"  [call-arg]
+ ibis/backends/bigquery/udf/core.py:85: error: Missing positional argument "value" in call to "Attribute"  [call-arg]
+ ibis/backends/bigquery/udf/core.py:95: error: Missing positional arguments "args", "keywords" in call to "Call"  [call-arg]

flake8-pyi (https://github.com/PyCQA/flake8-pyi)
+ pyi.py:2249: error: "AST" has no attribute "lineno"  [attr-defined]
+ pyi.py:2249: error: "AST" has no attribute "col_offset"  [attr-defined]

streamlit (https://github.com/streamlit/streamlit)
+ lib/streamlit/runtime/scriptrunner/magic.py: note: In function "_build_st_write_call":
+ lib/streamlit/runtime/scriptrunner/magic.py:166:12: error: Unexpected keyword argument "kwargs" for "Call"; did you mean "args"?  [call-arg]
+ note: "Call" defined here
+ lib/streamlit/runtime/scriptrunner/magic.py:166:12: error: Unexpected keyword argument "starargs" for "Call"  [call-arg]
+ note: "Call" defined here

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

JelleZijlstra approved these changes

May 18, 2024

View reviewed changes

JelleZijlstra left a comment

Copy link

Copy Markdown

Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

JelleZijlstra merged commit 5bd7150 into python:main

May 18, 2024

JelleZijlstra added a commit to PyCQA/flake8-pyi that referenced this pull request

May 18, 2024


          Fix new type error

a0370f1

python/typeshed#11880 correctly points out that not all AST nodes have a line and column number.

JelleZijlstra mentioned this pull request

May 18, 2024

Fix new type error PyCQA/flake8-pyi#489

Merged

JelleZijlstra added a commit to JelleZijlstra/pytest that referenced this pull request

May 18, 2024


          Fix new typing issues in AST code

8bb493b

python/typeshed#11880 adds more precise types for AST nodes. I'm submitting some changes to adapt pytest to these changes.

JelleZijlstra mentioned this pull request

May 18, 2024

Fix new typing issues in AST code pytest-dev/pytest#12337

Merged

AlexWaygood pushed a commit to PyCQA/flake8-pyi that referenced this pull request

May 18, 2024


          Fix new type error (#489)

f1230fc

python/typeshed#11880 correctly points out that not all AST nodes have a
line and column number.

JelleZijlstra added a commit to JelleZijlstra/werkzeug that referenced this pull request

May 18, 2024


          Improve types for AST handling

8a672ef

Adapting to changes in python/typeshed#11880. This mostly adds
more precise types for individual pieces of AST.

JelleZijlstra mentioned this pull request

May 18, 2024

Improve types for AST handling pallets/werkzeug#2905

Merged

JelleZijlstra added a commit to JelleZijlstra/sphinx that referenced this pull request

May 18, 2024


          More precise types in preserve_defaults.py

7861e6a

Followup from python/typeshed#11880. Not all AST nodes have a lineno; in this file only parameter defaults are passed to this function, which are always ast.expr.

JelleZijlstra mentioned this pull request

May 18, 2024

More precise types in preserve_defaults.py sphinx-doc/sphinx#12385

Merged

bluetech pushed a commit to pytest-dev/pytest that referenced this pull request

May 18, 2024


          Fix new typing issues in AST code (#12337)

ee9ea70

python/typeshed#11880 adds more precise types for AST nodes. I'm submitting some changes to adapt pytest to these changes.

bzoracler deleted the ast-subclass-constructors branch

May 18, 2024 20:12

JelleZijlstra added a commit to JelleZijlstra/streamlit that referenced this pull request

May 18, 2024


          Remove non-existent kwargs in ast.Call() call

fc2ff08

See https://docs.python.org/3.10/library/ast.html#abstract-grammar for a reference on the attributes `ast.Call` takes.

A future version of mypy will give a type error for this code (python/typeshed#11880).

JelleZijlstra mentioned this pull request

May 18, 2024

Remove non-existent kwargs in ast.Call() call streamlit/streamlit#8711

Merged

vdonato added a commit to streamlit/streamlit that referenced this pull request

May 21, 2024


          Remove non-existent kwargs in ast.Call() call (#8711)

9e3b77a

See https://docs.python.org/3.10/library/ast.html#abstract-grammar for a
reference on the attributes `ast.Call` takes.

A future version of mypy will give a type error for this code
(python/typeshed#11880).

Co-authored-by: Vincent Donato <[email protected]>

davidism pushed a commit to JelleZijlstra/werkzeug that referenced this pull request

Aug 20, 2024


          Improve types for AST handling

2ea2540

Adapting to changes in python/typeshed#11880. This mostly adds
more precise types for individual pieces of AST.

benjamin-awd pushed a commit to benjamin-awd/streamlit that referenced this pull request

Sep 29, 2024


          Remove non-existent kwargs in ast.Call() call (streamlit#8711)

4236a89

See https://docs.python.org/3.10/library/ast.html#abstract-grammar for a
reference on the attributes `ast.Call` takes.

A future version of mypy will give a type error for this code
(python/typeshed#11880).

Co-authored-by: Vincent Donato <[email protected]>

Avasam mentioned this pull request

Apr 10, 2025

A couple of _ast related issues #4760

Closed

asmeralt pushed a commit to asmeralt/streamlit that referenced this pull request

Sep 29, 2025


          Remove non-existent kwargs in ast.Call() call (streamlit#8711)

f1ab774

See https://docs.python.org/3.10/library/ast.html#abstract-grammar for a
reference on the attributes `ast.Call` takes.

A future version of mypy will give a type error for this code
(python/typeshed#11880).

Co-authored-by: Vincent Donato <[email protected]>

Reviewers

AlexWaygood

AlexWaygood left review comments

JelleZijlstra

JelleZijlstra approved these changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Typing the ast.AST subclass constructors

Uh oh!

There was an error while loading. Please reload this page.

3 participants

Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.

GeezerVison

Conversation

bzoracler commented May 9, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment has been minimized.

Uh oh!

Uh oh!

This comment has been minimized.

This comment has been minimized.

JelleZijlstra commented May 9, 2024

Uh oh!

JelleZijlstra May 9, 2024

Choose a reason for hiding this comment

Uh oh!

bzoracler May 10, 2024

Choose a reason for hiding this comment

Uh oh!

JelleZijlstra May 9, 2024

Choose a reason for hiding this comment

Uh oh!

bzoracler May 10, 2024

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

bzoracler commented May 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment has been minimized.

JelleZijlstra commented May 10, 2024

Uh oh!

This comment has been minimized.

github-actions Bot commented May 10, 2024

Uh oh!

JelleZijlstra left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

bzoracler commented May 9, 2024 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.

bzoracler commented May 10, 2024 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.