Description

samtygier

opened on Apr 15, 2022

Issue body actions

SharedMemory.unlink() contains a import which can fail if called during interpreter shutdown.

For example when working with SharedMemory as the buffer for a numpy array it is useful to create a wrapper object to ensure that clean up is done with the object is deleted.

#import numpy as np
from multiprocessing import shared_memory

class ShArray:
    def __init__(self):
        self.shared_memory = shared_memory.SharedMemory(create=True, size=400)
        #self.array = np.ndarray((10,10), dtype="i4", buffer=self.shared_memory.buf)

    def __del__(self):
        self.shared_memory.close()
        self.shared_memory.unlink()

a1 = ShArray()

(Commented lines show my typical usage, but are not need to demonstrate the issue)

Exception ignored in: <function ShArray.__del__ at 0x7f36eee510d0>
Traceback (most recent call last):
  File "/home/sam/git/cpython/test_shared_mem2.py", line 13, in __del__
  File "/home/sam/git/cpython/Lib/multiprocessing/shared_memory.py", line 240, in unlink
ImportError: sys.meta_path is None, Python is likely shutting down
/home/sam/git/cpython/Lib/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '

This happens on 3.8 and current main, both on Linux x86-64.

Patch to follow

Activity

1. AlexWaygood

   added

   type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

   on Apr 15, 2022
2. vstinner

   added

   topic-multiprocessing

   on Jun 6, 2022
3. samtygier

   added a commit that references this issue on Jun 16, 2022

   pythongh-91577: SharedMemory move imports out of methods

   296ddaa
4. vstinner

   closed this as completedin #91579on Jun 16, 2022
5. vstinner

   added a commit that references this issue on Jun 16, 2022

   gh-91577: SharedMemory move imports out of methods (#91579)

   9a458be
6. miss-islington

   added 2 commits that reference this issue on Jun 16, 2022

   pythongh-91577: SharedMemory move imports out of methods (pythonGH-91579

   d6e406f

   pythongh-91577: SharedMemory move imports out of methods (pythonGH-91579

   59bbbba
7. vstinner commented on Jun 16, 2022

   

   vstinner

   on Jun 16, 2022

   Member

   More actions

   I'm not a supporter of multiprocessing programs relying on the GC to call __del__() finalizers. You should close resources explicitly, using with: statement if possible. But well, the change seems acceptable to me, so I merged it.

   cc @pitrou
8. miss-islington

   added 2 commits that reference this issue on Jun 16, 2022

   gh-91577: SharedMemory move imports out of methods (GH-91579)

   5c10c36

   gh-91577: SharedMemory move imports out of methods (GH-91579)

   0c49a09
9. pitrou commented on Jun 16, 2022

   

   pitrou

   on Jun 16, 2022

   Member

   More actions

   Seems fine to me. with statements are not always usable.
10. vstinner commented on Jun 16, 2022

    

    vstinner

    on Jun 16, 2022

    Member

    More actions

    with statements are not always usable.

    So we should make them usable in more cases :-)

    Even without with:, the idea is to explicitly release resources, not rely on the GC for that. The objects are not finalized in a specific order by the GC. The exact order is really complicated and depends on too many things. So it's likely that your code will fall into many corner cases. ImportError: sys.meta_path is None, Python is likely shutting down is just one example of such issue that you can get if you rely on the GC.

    In PyPy, it expect that it will be even worse.
11. pitrou commented on Jun 16, 2022

    

    pitrou

    on Jun 16, 2022

    Member

    More actions

    So we should make them usable in more cases :-)

    I mean that with statements only work if your lifetime is lexically-defined. It's not the case for more sophisticated use cases unfortunately.
12. serhiy-storchaka

    added a commit that references this issue on Jun 26, 2022

    Revert 30935 defer bpo45162 to 312 (#94285)

    1ffde3e

Issue actions