Description

markshannon

opened on Feb 21, 2024

Issue body actions

Feature or enhancement

With #28802 we only create dictionaries when needed, which means we need to guess how big to create the values array attached to each object.

We use the following heuristic: Start with 30 items and reduce by one each time we create an object until the size of the array is no more than one greater than the number of keys in the shared keys.
This works reasonably well, but it could be improved. Many classes have a fixed set of attributes, but they are not all used during early object creation. Static analysis could give us a better estimate of the size of values array to use, by pre-initializing the shared keys.

For each class body we should have the compiler inject a special attribute, say __expected_attributes__, which can be used at runtime to compute the expected attributes for the class.

For an example of code where static analysis could work, but our current dynamic approach does not, see https://github.com/python/pyperformance/blob/main/pyperformance/data-files/benchmarks/bm_go/run_benchmark.py

Linked PRs

• gh-115775: Compiler adds __static_attributes__ field to classes #115913
• gh-115775: Add whatsnew entry about __static_attributes__ #117909
• GH-115775: Use __static_attributes__ to initialize shared keys #118468

Activity

1. markshannon

   added

   type-featureA feature request or enhancementA feature request or enhancement

   performancePerformance or resource usagePerformance or resource usage

   interpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)

   3.13only security fixesonly security fixes

   on Feb 21, 2024
2. iritkatriel commented on Feb 23, 2024

   

   iritkatriel

   on Feb 23, 2024

   Member

   More actions

   we should have the compiler inject a special attribute, say expected_attributes,

   Do you mean that this should be added as a field to PyTypeObject?
3. iritkatriel

   self-assigned this

   on Feb 23, 2024
4. markshannon commented on Feb 23, 2024

   

   markshannon

   on Feb 23, 2024

   MemberAuthor

   More actions

   No need to mess with the C structs.
   I was thinking that it would be done mostly in the compiler.
   The compiler would convert:

   class C:
       def meth(self):
           self.arg = ...

   into

   class C:
       def meth(self):
           self.arg = ...
       __expected_attributes__ = ("arg", ...)

   We would also need some code in PyType_Ready() to consume the __expected_attributes__ and deduce the correct size limits for the shared keys.
5. iritkatriel

   added 2 commits that reference this issue on Feb 25, 2024

   pythongh-115775: Use static analysis to estimate how many items a cla…

   95432c1

   pythongh-115775: Use static analysis to estimate how many items a cla…

   3306f6e
6. iritkatriel commented on Mar 3, 2024

   

   iritkatriel

   on Mar 3, 2024

   Member

   More actions

   We use the following heuristic: Start with 30 items and reduce by one each time we create an object until the size of the array is no more than one greater than the number of keys in the shared keys.

   As discussed offline, we need to redesign this heuristic. The number of shared keys after an object is created is no longer assumed to be sufficient - we want to leave enough room for the expected attributes.

   We now have

   • a const upper bound on the size of the array
   • a lower bound based on the __expected_attributes__
7. iritkatriel

   added 2 commits that reference this issue on Mar 23, 2024

   pythongh-115775: Use static analysis to estimate how many items a cla…

   e264563

   gh-115775: Compiler adds __static_attributes__ field to classes (#115913

   79be757
8. NGRsoftlab

   added 2 commits that reference this issue on Apr 2, 2024

   Squashed commit of the following:

   1b5fe4c

   Squashed commit of the following:

   a016a54
9. encukou commented on Apr 15, 2024

   

   encukou

   on Apr 15, 2024

   Member

   More actions

   Apparently, adding a new attribute to classes breaks several real-world packages that do some kind of introspection -- at least zope.interface and typing_extensions.

   Could you add an entry about this change to What's New?

   Reacted by Erlend E. Aasland and Edgar Ramírez Mondragón
10. iritkatriel

    removed their assignment

    on Apr 15, 2024
11. iritkatriel

    added a commit that references this issue on Apr 15, 2024

    pythongh-115775: Add whatnew entry about __static_attributes__

    1929797
12. diegorusso

    added a commit that references this issue on Apr 17, 2024

    pythongh-115775: Compiler adds __static_attributes__ field to classes (…

    7822025
13. encukou

    added a commit that references this issue on Apr 19, 2024

    gh-115775: Add whatsnew entry about __static_attributes__ (GH-117909)

    8a01fd7
14. markshannon commented on May 1, 2024

    

    markshannon

    on May 1, 2024

    MemberAuthor

    More actions

    For 3.13 we are going to use the __static_attributes__ as a template for the shared keys. It is not ideal as it doesn't account for superclasses, but it better than nothing (which is what we have now). #118468

    For 3.14 we can be a bit more sophisticated.
15. markshannon

    added a commit that references this issue on Aug 27, 2024

    GH-115775: Use `__static_attributes__` to initialize shared keys (GH-…

    89328f7

Issue actions