Description

brandtbucher

opened on Feb 28, 2024

Issue body actions

The JIT has a minimum allocation granularity of 1 page, with code and data on separate pages. Reducing this minimum would mean having pages of memory shared between executors, which is much harder to manage (and isn't thread-safe).

So the minimum amount of memory owned by an executor is 2 pages. This equals 8KB of memory on most platforms (the notable exception being newer Macs which have 16KB pages, and thus a whopping 32KB minimum allocation per trace). And all of that memory is paged in, since there's at least some memory being used on every page.

We should tune the trace collection machinery to find a sensible minimum trace length for tier two. We might also consider compiling fewer traces.

A separate, but hairier, issue is what to do about the array of cold exit executors. We have 512 of them, and they're one instruction each, adding up to 4MB of mostly empty memory (16MB on macOS). For reference, if optimally packed (i.e. compiled as a single trace of length 512), they would take up about 200KB.

@mdboom has been working on measuring the memory overhead of tier two on our benchmark suite. Early results show that:

• The tier two interpreter uses ~1% more memory (max RSS).
• On Linux, the JIT used ~5% more memory prior to GH-112354: Initial implementation of warm up on exits and trace-stitching #114142, and ~15% after that change landed.
• On M1 Mac, the JIT used ~15% more memory prior to GH-112354: Initial implementation of warm up on exits and trace-stitching #114142, and ~80% after that change landed.

(Keep in mind that most of the benchmarks have a tiny memory footprint to begin with, so moderate increases can manifest as large percentages.)

CC: @gvanrossum, @markshannon

Linked PRs

• GH-116017: Put JIT code and data on the same page #116845
• GH-116017: Get rid of _COLD_EXIT #120960
• GH-123516: Improve JIT memory consumption by invalidating cold executors #124443

Activity

1. brandtbucher

   added

   performancePerformance or resource usagePerformance or resource usage

   interpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)

   3.13only security fixesonly security fixes

   on Feb 28, 2024
2. brandtbucher commented on Feb 28, 2024

   

   brandtbucher

   on Feb 28, 2024

   MemberAuthor

   More actions

   We may also want to consider a scheme for garbage-collecting cold traces. We already have a linked list of executors and a way of invalidating/removing them, so it probably wouldn't be too big of a project.
3. gvanrossum commented on Feb 28, 2024

   

   gvanrossum

   on Feb 28, 2024

   Member

   More actions

   Yeah, for a hot executor the memory waste feels less of an issue than for a cold executor. Without measuring what's hot, we could at least have separate min-trace-size limits if the executor has a loop in it and if it doesn't. Mark probably has more interesting things to say. Meeting agenda for tomorrow?

   Reacted by Michael Droettboom
4. brandtbucher commented on Feb 28, 2024

   

   brandtbucher

   on Feb 28, 2024

   MemberAuthor

   More actions

   Action item for me here is to investigate a more efficient allocation scheme (with multiple executors per page).
5. brandtbucher

   added a commit that references this issue on Mar 19, 2024

   GH-116017: Put JIT code and data on the same page (GH-116845)

   2c82592
6. vstinner

   added a commit that references this issue on Mar 20, 2024

   pythonGH-116017: Put JIT code and data on the same page (pythonGH-116845

   318e6f1
7. adorilson

   added a commit that references this issue on Mar 25, 2024

   pythonGH-116017: Put JIT code and data on the same page (pythonGH-116845

   62ef665
8. diegorusso

   added a commit that references this issue on Apr 17, 2024

   pythonGH-116017: Put JIT code and data on the same page (pythonGH-116845

   6b82138
9. brandtbucher

   assigned

   brandtbucher

   and unassigned

   brandtbucher

   on May 1, 2024
10. savannahostrowski commented on May 1, 2024

    

    savannahostrowski

    on May 1, 2024

    Member

    More actions

    I'll take a look at this ahead of 3.14!

    Reacted by Brandt Bucher
11. brandtbucher

    assigned

    savannahostrowski

    on May 1, 2024
12. brandtbucher

    added

    topic-JIT

    on May 31, 2024
13. markshannon commented on Jun 26, 2024

    

    markshannon

    on Jun 26, 2024

    Member

    More actions

    We will need lots of "tiny traces" (short executors with only a few uops), to link longer traces, form loops, handled specialized entries and exits, etc.
    So avoiding "tiny" traces is not an option.

    The solution to this seems to be compacting traces together during GC.
    It should give us high density and allow as many "tiny traces" as we need.
    (This is not my idea, but I don't see it written down anywhere).

    @brandtbucher or @savannahostrowski could you write up an issue for compaction (or link to an existing one)?

    Reacted by Savannah Ostrowski
14. brandtbucher commented on Jun 26, 2024

    

    brandtbucher

    on Jun 26, 2024

    MemberAuthor

    More actions

    Yep, I’ll write one up when I get back on Monday.

    Reacted by Savannah Ostrowski
15. brandtbucher

    added a commit that references this issue on Jul 1, 2024

    GH-116017: Get rid of _COLD_EXITs (GH-120960)

    33903c5
16. Akasurde

    added a commit that references this issue on Jul 3, 2024

    pythonGH-116017: Get rid of _COLD_EXITs (pythonGH-120960)

    d7dedc3
17. estyxx

    added a commit that references this issue on Jul 17, 2024

    pythonGH-116017: Get rid of _COLD_EXITs (pythonGH-120960)

    6fcc55e
18. savannahostrowski

    added

    3.14bugs and security fixesbugs and security fixes

    and removed

    3.13only security fixesonly security fixes

    on Sep 6, 2024
19. savannahostrowski

    assigned

    brandtbucher

    and unassigned

    savannahostrowski

    on Nov 2, 2024
20. savannahostrowski commented on Nov 2, 2024

    

    savannahostrowski

    on Nov 2, 2024

    Member

    More actions

    I think that the remaining work here is related to trace compaction so I will reassign to @brandtbucher :)

Issue actions