GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-109653: Just import `recursive_repr` in `dataclasses` by sobolevn · Pull Request #109822 · python/cpython

Conversation

@sobolevn

@sobolevn sobolevn commented Sep 25, 2023 •
edited by AlexWaygood
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

Import time is unchanged. Before:

» pyperf timeit 'import dataclasses'
.....................
Mean +- std dev: 106 ns +- 2 ns

After:

» pyperf timeit 'import dataclasses'
.....................
Mean +- std dev: 105 ns +- 2 ns

But, we now drop two dependencies: functools and _thread and add just one: reprlib
https://github.com/python/cpython/blame/f2eaa92b0cc5a37a9e6010c7c6f5ad1a230ea49b/Lib/dataclasses.py#L248-L266

It was strange to have

# This function's logic is copied from "recursive_repr" function in
# reprlib module to avoid dependency.

But import two other heavier modules. The most interesting part is that functools always imports reprlib inside. See #109653 (comment)

  • Issue: Improve import time of various stdlib modules #109653

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@sobolevn
pythongh-109653: Just import recursive_repr in dataclasses
e71ff5c
@sobolevn
sobolevn requested a review from ericvsmith as a code owner September 25, 2023 07:30
@bedevere-app bedevere-app Bot added the awaiting review label Sep 25, 2023
@bedevere-app bedevere-app Bot mentioned this pull request Sep 25, 2023
Improve import time of various stdlib modules #109653
Closed
@sobolevn sobolevn added the skip news label Sep 25, 2023
@sobolevn

sobolevn commented Sep 25, 2023

Copy link
Copy Markdown
Member Author

The windows failure is known and unrelated: #109739

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@ericvsmith

ericvsmith commented Sep 25, 2023

Copy link
Copy Markdown
Member

I don't think this change is worth making. It seems mostly like code churn: there's no actual problem being solved.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@AA-Turner

AA-Turner commented Sep 25, 2023

Copy link
Copy Markdown
Member

Is there a difference if the functools import is deferred to within the vendored _recursive_repr function (& we remove the top-level)?

A

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@sobolevn

sobolevn commented Sep 25, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member Author

@AA-Turner it makes no difference, because inspect imports 80% of all stdlib, including functools :)

inspect is only used in two places:

  • cpython/Lib/dataclasses.py

    Line 970 in f2eaa92

    cls_annotations = inspect.get_annotations(cls)
  • cpython/Lib/dataclasses.py

    Line 1139 in f2eaa92

    text_sig = str(inspect.signature(cls)).replace(' -> None', '')

But, it still does not improve the import time:

» pyperf timeit 'import dataclasses'
.....................
Mean +- std dev: 105 ns +- 1 ns

I also tried to defer inspect, but it is still the same :(

AA-Turner reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@DavidCEllis

DavidCEllis commented Sep 29, 2023

Copy link
Copy Markdown
Contributor

...it makes no difference, because inspect imports 80% of all stdlib, including functools :)

inspect is only used in two places:

  • cpython/Lib/dataclasses.py

    Line 970 in f2eaa92

    cls_annotations = inspect.get_annotations(cls)
  • cpython/Lib/dataclasses.py

    Line 1139 in f2eaa92

    text_sig = str(inspect.signature(cls)).replace(' -> None', '')

But, it still does not improve the import time:

» pyperf timeit 'import dataclasses'
.....................
Mean +- std dev: 105 ns +- 1 ns

I also tried to defer inspect, but it is still the same :(

@sobolevn
Are these pyperf timings actually outputting accurate data when it comes to imports? The numbers seem surprisingly low to me.

I get similar numbers (scale-wise) from pyperf but they don't match with other timings I've done.

pyperf timeit "import dataclasses" 1

.....................
Mean +- std dev: 84.6 ns +- 4.1 ns

python -Ximporttime -c "import dataclasses" (truncated to only show dataclasses):

import time: self [us] | cumulative | imported package
...
import time:       961 |      18112 | dataclasses

Which lines up roughly with the timings I get from hyperfine comparing launching python with no imports to importing dataclasses.

hyperfine -w10 -r100 -N 'python -c "pass"' 'python -c "import dataclasses"'

Benchmark 1: python -c "pass"
  Time (mean ± σ):      11.0 ms ±   0.6 ms    [User: 8.4 ms, System: 2.4 ms]
  Range (min … max):    10.4 ms …  13.0 ms    100 runs
 
Benchmark 2: python -c "import dataclasses"
  Time (mean ± σ):      28.5 ms ±   1.0 ms    [User: 23.8 ms, System: 4.5 ms]
  Range (min … max):    27.0 ms …  32.0 ms    100 runs
 
Summary
  'python -c "pass"' ran
    2.59 ± 0.17 times faster than 'python -c "import dataclasses"'

From previous experiments mentioned in #109653 (comment) I would expect deferring inspect to remove close to half of the import time. However, it's not actually very useful to do so as any use of @dataclass will end up importing inspect in order to get the annotations, so practically you wouldn't actually see any benefit from delaying the import unless none of the modules you use make any dataclasses.


Slightly more directly related, and also mentioned in #109653 (comment) the definition of _recursive_repr you have in dataclasses makes use of functools.wraps which uses functools.partial which in turn uses reprlib.recursive_repr on its own __repr__ so perhaps this change still makes sense from that angle?

Footnotes

  1. I also get surprisingly high values for pyperf timeit 'import sys' in comparison. ↩

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@AA-Turner

AA-Turner commented Oct 1, 2023

Copy link
Copy Markdown
Member

A change in the other direction (per reprlib's own implementation):

diff --git a/Lib/dataclasses.py b/Lib/dataclasses.py
index 84f8d68ce0..8579f6d605 100644
--- a/Lib/dataclasses.py
+++ b/Lib/dataclasses.py
@@ -4,7 +4,6 @@
 import types
 import inspect
 import keyword
-import functools
 import itertools
 import abc
 import _thread
@@ -252,7 +251,6 @@ def _recursive_repr(user_function):
     # call.
     repr_running = set()

-    @functools.wraps(user_function)
     def wrapper(self):
         key = id(self), _thread.get_ident()
         if key in repr_running:
@@ -263,6 +261,15 @@ def wrapper(self):
         finally:
             repr_running.discard(key)
         return result
+
+    wrapper.__module__ = getattr(user_function, '__module__')
+    wrapper.__doc__ = getattr(user_function, '__doc__')
+    wrapper.__name__ = getattr(user_function, '__name__')
+    wrapper.__qualname__ = getattr(user_function, '__qualname__')
+    wrapper.__annotations__ = getattr(user_function, '__annotations__', {})
+    wrapper.__type_params__ = getattr(user_function, '__type_params__', ())
+    wrapper.__wrapped__ = user_function
+
     return wrapper

 class InitVar:

A

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@sobolevn

sobolevn commented Oct 1, 2023

Copy link
Copy Markdown
Member Author

@AA-Turner we still have _thread import (1 _thread vs 1 reprlib) and quite a lot of boilerplate code that can go out of sync with the reprlib.recursive_repr (it happened before #109819) for no real reason.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

encukou
encukou approved these changes Mar 4, 2024
View reviewed changes

@encukou encukou left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

boilerplate code that can go out of sync with the reprlib.recursive_repr (it happened before #109819)

IMO, this is a good reason to merge this PR.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@bedevere-app bedevere-app Bot added awaiting merge and removed awaiting review labels Mar 4, 2024
ericvsmith
ericvsmith approved these changes Mar 5, 2024
View reviewed changes

@ericvsmith ericvsmith left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As long as there's not a performance regression due to importing reprlib, I'm okay with this. I have not checked the performance.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@DavidCEllis

DavidCEllis commented Mar 5, 2024

Copy link
Copy Markdown
Contributor

As long as there's not a performance regression due to importing reprlib, I'm okay with this. I have not checked the performance.

You're already importing reprlib indirectly via functools so this shouldn't change import time if that's a concern (functools will still be imported via inspect so you won't see an improvement by removing that direct import).

sobolevn reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@ericvsmith

ericvsmith commented Mar 5, 2024

Copy link
Copy Markdown
Member

Agreed that reprlib is already imported (checked with 3.11).

sobolevn reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@ericvsmith
ericvsmith merged commit edc9d85 into python:main Mar 5, 2024
@bedevere-app bedevere-app Bot removed the awaiting merge label Mar 5, 2024
adorilson pushed a commit to adorilson/cpython that referenced this pull request Mar 25, 2024
@sobolevn @adorilson
pythongh-109653: Just import recursive_repr in dataclasses (pytho…
b60a0f7
…ngh-109822)
diegorusso pushed a commit to diegorusso/cpython that referenced this pull request Apr 17, 2024
@sobolevn @diegorusso
pythongh-109653: Just import recursive_repr in dataclasses (pytho…
f65e3ab
…ngh-109822)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@encukou encukou encukou approved these changes
@ericvsmith ericvsmith ericvsmith approved these changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

skip news

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

5 participants

@sobolevn @ericvsmith @AA-Turner @DavidCEllis @encukou
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.