GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — Improve the Efficiency of Python3.11.1 __getattr__ · Issue #102213 · python/cpython

Improve the Efficiency of Python3.11.1 __getattr__ #102213

New issue
Copy link
New issue
Copy link
Closed
Closed
Improve the Efficiency of Python3.11.1 __getattr__#102213
Copy link
Labels
performancePerformance or resource usagePerformance or resource usagetype-featureA feature request or enhancementA feature request or enhancement

Description

@wangxiang-hz
wangxiang-hz
opened on Feb 24, 2023
Issue body actions

Feature or enhancement

__getattr__ in Python3.11.1 is much slower than @Property and visiting an object's attribute. It's even slower than Python3.10.4.
_PyObject_GenericGetAttrWithDict is the key reason. If Python3 fails finding an attribute in normal ways, it will return NULL and raise an exception. But raising an exception has performance cost. Python3.11.1 add set_attribute_error_context to support Fine Grained Error Locations in Tracebacks. It makes things worser.

Pitch

We can use this test code:

import time
import sys

class A:
    def foo(self):
        print("Call A.foo!")

    def __getattr__(self, name):
        return 2

    @property
    def ppp(self):
        return 3

class B(A):
    def foo(self):
        print("Call B.foo!")


class C(B):

    def __init__(self) -> None:
        self.pps = 1

    def foo(self):
        print("Call C.foo!")


def main():
    start = time.time()
    for i in range(1, 1000000):
        pass
    end = time.time()
    peer = end - start
    c = C()
    print(f"Python version of {sys.version}")
    start = time.time()
    for i in range(1, 1000000):
        s = c.pps
    end = time.time()
    print(f"Normal getattr spend time: {end - start - peer}")
    start = time.time()
    for i in range(1, 1000000):
        s = c.ppa
    end = time.time()
    print(f"Call __getattr__ spend time: {end - start - peer}")
    start = time.time()
    for i in range(1, 1000000):
        s = c.ppp
    end = time.time()
    print(f"Call property spend time: {end - start - peer}")

if __name__ == "__main__":
    main()

The result shows how slow __getattr__ is:

Python version of 3.11.1 (main, Dec 26 2022, 16:32:50) [GCC 8.3.0]
Normal getattr spend time: 0.03204226493835449
Call __getattr__ spend time: 0.4767305850982666
Call property spend time: 0.06345891952514648

When we define __getattr__, failed to find an attribute is what we expected. If we can get this result and then call __getattr__ without exception handling, it will be faster.
I tried to modify Python3.11.1 like this:

  1. add a new function in object.c:
PyObject *
PyObject_GenericTryGetAttr(PyObject *obj, PyObject *name)
{
    return _PyObject_GenericGetAttrWithDict(obj, name, NULL, 1);
}
  1. change typeobject.c :
if (getattribute == NULL ||
        (Py_IS_TYPE(getattribute, &PyWrapperDescr_Type) &&
         ((PyWrapperDescrObject *)getattribute)->d_wrapped ==
         (void *)PyObject_GenericGetAttr))
        // res = PyObject_GenericGetAttr(self, name);
        res = PyObject_GenericTryGetAttr(self, name);
    else {
        Py_INCREF(getattribute);
        res = call_attribute(self, getattribute, name);
        Py_DECREF(getattribute);
    }
    if (res == NULL) {
        if (PyErr_ExceptionMatches(PyExc_AttributeError))
            PyErr_Clear();
        res = call_attribute(self, getattr, name);
    }
    Py_DECREF(getattr);
    return res;

Rebuild python, it really become faster: spend time: 0.13772845268249512.

Previous discussion


getattr is much slower in Python3.11

Linked PRs

  • gh-102213: Optimizing the Performance of __getattr__ #102213 #102248
  • Revert "gh-102213: Optimize the performance of __getattr__ (GH-102248)" #103332
  • gh-102213: Optimize the performance of __getattr__ #103761

Activity

  1. wangxiang-hz
    added
    type-featureA feature request or enhancementA feature request or enhancement
    on Feb 24, 2023
  2. AlexWaygood
    added
    performancePerformance or resource usagePerformance or resource usage
    on Feb 24, 2023
  3. Fidget-Spinner commented on Feb 25, 2023

    @Fidget-Spinner
    Fidget-Spinner
    on Feb 25, 2023
    Member
    More actions

    Could you submit a pull request please? I'll try to review it.

  4. wangxiang-hz commented on Mar 8, 2023

    @wangxiang-hz
    wangxiang-hz
    on Mar 8, 2023
    ContributorAuthor
    More actions

    Could you submit a pull request please? I'll try to review it.

    I have already submitted a pull request, but the default assigned reviewer is too busy to review it. Could you please review it and provide some suggestions when you are available? Thank you very much.

  5. Fidget-Spinner commented on Mar 8, 2023

    @Fidget-Spinner
    Fidget-Spinner
    on Mar 8, 2023
    Member
    More actions

    Sorry. I didn't see the pull request (pull requests linked to an issue don't ping me even though I'm subscribe to the issue). I'll review it. Thanks!

  6. wangxiang-hz commented on Mar 10, 2023

    @wangxiang-hz
    wangxiang-hz
    on Mar 10, 2023
    ContributorAuthor
    More actions

    Sorry. I didn't see the pull request (pull requests linked to an issue don't ping me even though I'm subscribe to the issue). I'll review it. Thanks!

    Thank you. I know that contributors in open source communities are very busy, so I didn't want to bother you with a “polite ping” :) I really appreciate your review.

    Reacted by Ken Jin
  7. Fidget-Spinner
    added a commit that references this issue on Mar 11, 2023

    gh-102213: Optimize the performance of `__getattr__` (GH-102248)

    aa0a73d
  8. hauntsaninja
    closed this as completedon Mar 11, 2023
  9. iritkatriel
    added a commit that references this issue on Mar 12, 2023

    pythongh-102213: Optimize the performance of `__getattr__` (pythonGH-…

    fc48167
  10. sobolevn
    added a commit that references this issue on Apr 7, 2023

    Revert "pythongh-102213: Optimize the performance of `__getattr__` (p…

    96946fd
  11. Fidget-Spinner
    added a commit that references this issue on Apr 7, 2023

    gh-102213: Revert "gh-102213: Optimize the performance of `__getattr_…

    059bb04
  12. warsaw
    added a commit that references this issue on Apr 11, 2023

    pythongh-102213: Revert "pythongh-102213: Optimize the performance of…

    ef37048
  13. Fidget-Spinner
    added a commit that references this issue on May 1, 2023

    gh-102213: Optimize the performance of `__getattr__` (GH-103761)

    59c27fa
  14. carljm
    added 2 commits that reference this issue on May 1, 2023

    Merge branch 'main' into supermon

    159b2aa

    Merge branch 'main' into inlinecomp2

    1274e2b
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    performancePerformance or resource usagePerformance or resource usagetype-featureA feature request or enhancementA feature request or enhancement

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions

      You are visitor 01402503 — GeezerVison does not store the pages you look up.
      Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.