GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-103134: Update multiprocessing.managers.ListProxy and multiprocessing.managers.DictProxy by invisibleroads · Pull Request #103133 · python/cpython

Conversation

@invisibleroads

@invisibleroads invisibleroads commented Mar 30, 2023 •
edited by bedevere-bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor
from multiprocessing import Manager
with Manager() as manager:
    xs = manager.list()
    xs.clear()

For now, we can use the workaround del xs[:]

  • Issue: Update multiprocessing.managers.ListProxy and multiprocessing.managers.DictProxy #103134

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@invisibleroads
Add support for manager.list().clear()
6fb4cba
```
from multiprocessing import Manager
with Manager() as manager:
    xs = manager.list()
    xs.clear()
```

For now, we can use the workaround `del xs[:]`
@bedevere-bot

bedevere-bot commented Mar 30, 2023

Copy link
Copy Markdown

Most changes to Python require a NEWS entry.

Please add it using the blurb_it web app or the blurb command-line tool.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot bedevere-bot added the awaiting review label Mar 30, 2023
@ghost

ghost commented Mar 30, 2023 •
edited by ghost
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown

All commit authors signed the Contributor License Agreement.
CLA signed

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@arhadthedev arhadthedev added stdlib Standard Library Python modules in the Lib/ directory topic-multiprocessing labels Mar 30, 2023
@invisibleroads invisibleroads changed the title Add support for manager.list().clear() gh-103134: Add support for manager.list().clear() Mar 30, 2023
@bedevere-bot bedevere-bot mentioned this pull request Mar 30, 2023
Update multiprocessing.managers.ListProxy and multiprocessing.managers.DictProxy #103134
Closed
blurb-it Bot and others added 2 commits March 30, 2023 18:19
@blurb-it
📜🤖 Added by blurb_it.
b3c8810
@invisibleroads
Update 2023-03-30-18-19-53.gh-issue-103134.bHrn91.rst
fc803ca
terryjreedy
terryjreedy reviewed Mar 30, 2023
View reviewed changes

@terryjreedy terryjreedy left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are there any tests for BaseList Proxy or derivatives that are not automatically expanded by expanding this list? And which should be manually augmented?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Lib/multiprocessing/managers.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

terryjreedy
terryjreedy reviewed Mar 30, 2023
View reviewed changes
Comment thread Misc/NEWS.d/next/Library/2023-03-30-18-19-53.gh-issue-103134.bHrn91.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

terryjreedy
terryjreedy reviewed Mar 30, 2023
View reviewed changes
Comment thread Lib/multiprocessing/managers.py Outdated
'__mul__', '__reversed__', '__rmul__', '__setitem__',
'append', 'count', 'extend', 'index', 'insert', 'pop', 'remove',
'append', 'clear', 'count', 'extend', 'index', 'insert', 'pop', 'remove',
'reverse', 'sort', '__imul__'

@terryjreedy terryjreedy Mar 30, 2023

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given the definition of method __iadd__ in the definition of ListProxy below, it would seem that it should be in the list above. On the other hand, adding xs += [2] to your example works as is. Maybe '_imul' does not need to be listed. Does it hurt to list 'iadd'?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@invisibleroads invisibleroads Mar 31, 2023

Copy link
Copy Markdown
Contributor Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like ListProxy.__iadd__ uses BaseListProxy.extend and ListProxy.__imul__ calls BaseListProxy.__imul__. I'm not sure of the reasoning behind why ListProxy defines __iadd__ and __imul__ separately instead of including them directly in BaseListProxy. Testing xs += [2] and xs *= 2 works when tested manually.

I'm afraid of replacing BaseListProxy with ListProxy for fear of breaking something. Looking at git blame, it looks like this part of the code was last changed 15 years ago? @benjaminp would it be safe to replace BaseListProxy with ListProxy and just include __iadd__ and __imul__?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@terryjreedy

terryjreedy commented Mar 30, 2023

Copy link
Copy Markdown
Member

As mentioned on issue, DictProxy and maybe others need updating.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

invisibleroads and others added 3 commits March 31, 2023 01:15
@invisibleroads
Update ListProxy and DictProxy
a748d41
```
from multiprocessing import Manager
with Manager() as manager:
    xs = manager.list()
    xs.clear
    xs.copy
    d = manager.dict()
    d | {}  # __or__
    {} | d  # __ror__
    reversed(d)  # __reversed__
    d.fromkeys
```

suggested by @terryjreedy
tested manually in Python 3.10.8

python#103134
@invisibleroads @terryjreedy
Update Misc/NEWS.d/next/Library/2023-03-30-18-19-53.gh-issue-103134.b…
22de5a1
…Hrn91.rst

Co-authored-by: Terry Jan Reedy <[email protected]>
@invisibleroads
Update 2023-03-30-18-19-53.gh-issue-103134.bHrn91.rst
8602f0a
@invisibleroads

invisibleroads commented Mar 31, 2023

Copy link
Copy Markdown
Contributor Author

As mentioned on issue, DictProxy and maybe others need updating.

Added a few more bits to DictProxy to support the cases you mentioned, including fromkeys, d | {}, {} | d, reversed(d).

After your comments, I'm tempted to replace BaseListProxy with ListProxy but would like to hear back from @benjaminp if @benjaminp remembers why BaseListProxy was defined separately.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@invisibleroads
Add DictProxy.__ior__ for d |= {'b': 2}
6fe264a
@invisibleroads

invisibleroads commented Mar 31, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor Author

As an update, I did try removing BaseListProxy and just adding __iadd__ and __imul__ directly to ListProxy and it didn't work. I'm not sure why. Here is the code that I used to test this functionality manually.

from multiprocessing import Manager, Process

def f(x):
    x += [1]
    x *= 2
    print(x)

def g(d):
    d |= {'b': 2}
    print(d)

with Manager() as manager:

    x = manager.list()
    x.append(0)
    p = Process(target=f, args=(x,))
    p.start()
    p.join()
    q = Process(target=print, args=(x,))
    q.start()
    q.join()

    d = manager.dict()
    d['a'] = 1
    p = Process(target=g, args=(d,))
    p.start()
    p.join()
    q = Process(target=print, args=(d,))
    q.start()
    q.join()

RESULTS

ACTUAL after replacing BaseListProxy with ListProxy (the __imul__ doesn't seem to work).
ACTUAL when DictProxy is missing __ior__

[0, 1, 0, 1]
[0, 1]
{'a': 1, 'b': 2}
{'a': 1}

EXPECTED and ACTUAL with current commits for this pull request -- works with this pull request

[0, 1, 0, 1]
[0, 1, 0, 1]
{'a': 1, 'b': 2}
{'a': 1, 'b': 2}

The pull request passes the above manual test.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@invisibleroads
Update 2023-03-30-18-19-53.gh-issue-103134.bHrn91.rst
8c2afdb
@invisibleroads invisibleroads changed the title gh-103134: Add support for manager.list().clear() gh-103134: Update multiprocessing.managers.ListProxy and multiprocessing.managers.DictProxy Mar 31, 2023
@invisibleroads

invisibleroads commented Mar 31, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor Author

We could write a test that uses a process to modify a manager.list or manager.list, then check that the list or dict has actually been modified.

I found the multiprocessing tests here
https://github.com/python/cpython/blob/main/Lib/test/_test_multiprocessing.py#L5703

will try to add tests over the weekend

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

invisibleroads added 4 commits April 29, 2023 09:19
@invisibleroads
Merge branch 'python:main' into patch-1
3ae5749
@invisibleroads
Fix __ior__ and add tests thanks to python#103134 (comment) @sunmy2019
1cf0985
@invisibleroads
Merge branch 'main' into patch-1
5ef9850
@invisibleroads
Merge branch 'main' into patch-1
dd54efc
@invisibleroads

invisibleroads commented Apr 30, 2023

Copy link
Copy Markdown
Contributor Author

This pull request is ready for review. Thanks to @arhadthedev for accepting this issue, @terryjreedy for the suggestions and @sunmy2019 for the very important hints.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

invisibleroads added 2 commits April 30, 2023 10:44
@invisibleroads
Merge branch 'main' into patch-1
8354c5b
@invisibleroads
Merge branch 'main' into patch-1
b826c7b
invisibleroads added 3 commits May 20, 2023 11:54
@invisibleroads
Merge branch 'main' into patch-1
847c072
@invisibleroads
Merge branch 'main' into patch-1
820b45b
@invisibleroads
Merge branch 'main' into patch-1
e94d306
@invisibleroads

invisibleroads commented May 20, 2023

Copy link
Copy Markdown
Contributor Author

This minor enhancement backports recently added list and dictionary functionality into multiprocessing.managers.ListProxy and multiprocessing.managers.DictProxy so that scripts that use multiprocessing.Manager can use the same list and dictionary functionality. Would it be possible for someone to review these proposed enhancements?

@kumaraditya303
@serhiy-storchaka
@pablogsal
@iritkatriel
@gvanrossum

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

serhiy-storchaka
serhiy-storchaka reviewed May 22, 2023
View reviewed changes

@serhiy-storchaka serhiy-storchaka left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please test the type and the value of results.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Lib/multiprocessing/managers.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Lib/test/_test_multiprocessing.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Lib/test/_test_multiprocessing.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Lib/test/_test_multiprocessing.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Lib/multiprocessing/managers.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

@invisibleroads
Merge branch 'main' into patch-1
083ad3d
@invisibleroads
invisibleroads requested a review from gpshead as a code owner April 27, 2024 17:03
@invisibleroads
Test the type and the value of the results as requested by @serhiy-st…
83ecdb3
…orchaka
@invisibleroads

invisibleroads commented Apr 27, 2024

Copy link
Copy Markdown
Contributor Author

It's been almost a year, but I finally got around to making the changes requested by @serhiy-storchaka.

Would it be possible for someone to review these changes?
@pablogsal
@iritkatriel
@gvanrossum
@kumaraditya303

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@invisibleroads
Remove trailing whitespace
6f2ea5b
@invisibleroads
invisibleroads requested a review from serhiy-storchaka April 27, 2024 19:56
invisibleroads added 7 commits April 29, 2024 11:49
@invisibleroads
Merge branch 'main' into patch-1
b95f840
@invisibleroads
Merge branch 'main' into patch-1
0e94e31
@invisibleroads
Merge branch 'main' into patch-1
15089ce
@invisibleroads
Merge branch 'main' into patch-1
d9b3bf6
@invisibleroads
Merge branch 'main' into patch-1
9eb39c9
@invisibleroads
Merge branch 'main' into patch-1
80a2f64
@invisibleroads
Update _test_multiprocessing.py
577e5a3
Check that the dictionary returned by dict_proxy.from_keys is a dict as defined in dict.from_keys as requested by @encukou
@invisibleroads

invisibleroads commented May 20, 2024

Copy link
Copy Markdown
Contributor Author

Thanks to @encukou for taking the time to review this during the PyCon 2024 sprints!!

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@encukou
Add links to the NEWS blurb
72d1055
@encukou
encukou enabled auto-merge (squash) May 20, 2024 14:08
encukou
encukou approved these changes May 20, 2024
View reviewed changes
@bedevere-app bedevere-app Bot added awaiting merge and removed awaiting review labels May 20, 2024
@encukou
encukou merged commit bbb4988 into python:main May 20, 2024
@bedevere-app bedevere-app Bot removed the awaiting merge label May 20, 2024
estyxx pushed a commit to estyxx/cpython that referenced this pull request Jul 17, 2024
@invisibleroads @estyxx
pythongh-103134: Update multiprocessing.managers.ListProxy and DictPr…
df129f0
…oxy (pythonGH-103133)
@gpshead gpshead mentioned this pull request Nov 11, 2024
Register multiprocessing proxy types to an appropriate collections.abc class #126417
Closed
donbarbos added a commit to donbarbos/typeshed that referenced this pull request Feb 14, 2026
@donbarbos
[multiprocessing] Update to 3.14
96fdb1a
Source: python/cpython#103133
@donbarbos donbarbos mentioned this pull request Feb 14, 2026
[multiprocessing] Update to 3.14 python/typeshed#15427
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@encukou encukou encukou approved these changes
@terryjreedy terryjreedy Awaiting requested review from terryjreedy
@gpshead gpshead Awaiting requested review from gpshead gpshead is a code owner
@serhiy-storchaka serhiy-storchaka Awaiting requested review from serhiy-storchaka

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

stdlib Standard Library Python modules in the Lib/ directory topic-multiprocessing

Projects

Multiprocessing issues
Status: Done

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

6 participants

@invisibleroads @bedevere-bot @terryjreedy @encukou @serhiy-storchaka @arhadthedev
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.