Description

tiran

opened on Jun 7, 2022

Issue body actions

Bug report

setup.py has a helper method add_multiarch_paths. The method creates a temporary file build/lib.{platform}/multiarch and unlinks it at the end of the function call. This is not parallel-safe.

PGO builds of Python use $(MAKE), so called recursive make. Recursive makes are considered harmful because the main make process has no understanding what the child make process is doing. With heavy parallel makes this can cause race conditions.

The combination of unsafe add_multiarch_paths(), recursive make and loooots of CPU cores can lead to build issues like #84461 (comment).

Possible workarounds

1. Rewrite our Makefile to not use $(MAKE)
2. Move CC -print-multiarch and dpkg-architecture ... -qDEB_HOST_MULTIARCH calls to configure.ac
3. Fix _bootsubprocess.check_output() and add_multiarch_paths() to use a more safe tmp file

Option (3) is the simplest approach. Even tmpfile = os.path.join(self.build_temp, f'multiarch-{os.getpid()}') would be good enough to avoid file conflicts in parallel builds.

Reacted by Eric Snow

Activity

1. tiran

   added

   type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

   buildThe build process and cross-buildThe build process and cross-build

   3.11only security fixesonly security fixes

   3.12only security fixesonly security fixes

   on Jun 7, 2022
2. tiran

   self-assigned this

   on Jun 7, 2022
3. pablogsal

   added

   release-blocker

   on Jun 7, 2022
4. tiran commented on Jun 7, 2022

   

   tiran

   on Jun 7, 2022

   MemberAuthor

   More actions

   My PR solved the race condition in add_multiarch_paths() but revealed another race condition later.

   @pablogsal 's problem is caused by the fact that PGO builds use recursive make. The sharedmods target and the profile-opt target run ./python setup.py build at the same time. setup.py is not designed to be parallel-safe.

   Reacted by Emma Smith
5. emmatyping commented on Jun 7, 2022

   

   emmatyping

   on Jun 7, 2022

   Member

   More actions

   Weird, I never hit a compilation failure after 2 hours of trying on my 32 core machine, maybe this is an edge case you can only hit with the macOS scheduler.
6. tiran commented on Jun 7, 2022

   

   tiran

   on Jun 7, 2022

   MemberAuthor

   More actions

   Did you create a build with configure --enable-optimizations? The issue only occurs with PGO builds.
7. emmatyping commented on Jun 7, 2022

   

   emmatyping

   on Jun 7, 2022

   Member

   More actions

   Yeah, I was running a script in a loop using the steps from Pablo's reproducer here: #84461 (comment)

   After 2 hours it still hadn't failed
8. pablogsal commented on Jun 7, 2022

   

   pablogsal

   on Jun 7, 2022

   Member

   More actions

   Yeah, I was running a script in a loop using the steps from Pablo's reproducer here: #84461 (comment)

   After 2 hours it still hadn't failed

   That's odd. I have been able to reproduce in at least 3 systems, including Linux with arch Linux and RHEL7
9. tiran commented on Jun 8, 2022

   

   tiran

   on Jun 8, 2022

   MemberAuthor

   More actions

   In comment #93589 (comment) @nascheme suggested to rename some of the targets to make the intention more clear. I like the idea, but I don't want to mix improvements with a release blocker fix.
10. tiran

    changed the title [-]setup.py add_multiarch_paths() is not parallel-safe with recursive make[/-] [+]make altinstall for PGO is not parallel-save[/+] on Jun 8, 2022
11. tiran

    added 3 commits that reference this issue on Jun 8, 2022

    gh-93584: Make all install+tests targets depends on all (GH-93589)

    243ed54

    [3.11] pythongh-93584: Make all install+tests targets depends on all (p…

    8c30e65

    [3.11] gh-93584: Make all install+tests targets depends on all (GH-93589

    abd0223
12. gpshead

    changed the title [-]make altinstall for PGO is not parallel-save[/-] [+]make altinstall for PGO is not parallel-safe[/+] on Jun 15, 2022
13. tiran commented on Jun 15, 2022

    

    tiran

    on Jun 15, 2022

    MemberAuthor

    More actions

    @pablogsal As far as I know this bug has been fixed by GH-93589.

    Reacted by Pablo Galindo Salgado
14. pablogsal

    closed this as completedon Jun 15, 2022
15. serhiy-storchaka

    added a commit that references this issue on Jun 26, 2022

    Revert 30935 defer bpo45162 to 312 (#94285)

    1ffde3e

Issue actions