GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — bpo-30256: Add test for nested queues by finefoot · Pull Request #16341 · python/cpython

Conversation

@finefoot

@finefoot finefoot commented Sep 24, 2019 •
edited by bedevere-bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

I stumbled upon this problem: https://stackoverflow.com/questions/56716470/python-multiprocessing-nested-shared-objects-doesnt-work-with-queue

Right now, Python docs state

Shared objects are capable of being nested. For example, a shared container object such as a shared list can contain other shared objects which will all be managed and synchronized by the SyncManager.

Source: https://docs.python.org/3/library/multiprocessing.html

These changes have been introduced with https://hg.python.org/cpython/rev/39e7307f9aee along with some tests. However, the tests only use nested ListProxy and DictProxy.

Because a test for AutoProxy is missing, the mentioned issue from the StackOverflow post remained undetected. This pull request adds a test for AutoProxy[Queue] for a queue nested inside a list and inside a dict.

Hence, the test added by this pull request will currently fail due to the existing bug regarding nested AutoProxy which already has a fix in #4819. After applying the patch from #4819, this test will succeed. So this pull request should be merged together with #4819.

https://bugs.python.org/issue30256

https://bugs.python.org/issue30256

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@finefoot
Add test for nested queues
9268fb5
@the-knights-who-say-ni the-knights-who-say-ni added the CLA signed label Sep 24, 2019
@bedevere-bot bedevere-bot added the awaiting review label Sep 24, 2019
@brandtbucher brandtbucher added needs backport to 3.7 tests Tests in the Lib/test dir labels Sep 24, 2019
@finefoot

finefoot commented Sep 25, 2019

Copy link
Copy Markdown
Contributor Author

@brandtbucher Thanks for adding the labels. :) Do you mind adding the "skip news" label to this pull request and the bugfix one #4819, too? It's the only CI check missing from #4819 and as far as I understand, a news entry for that small patch is not necessary. The failed CI check might be the only thing missing from getting #4819 merged.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

brandtbucher
brandtbucher requested changes Sep 25, 2019
View reviewed changes

@brandtbucher brandtbucher left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @finefoot! Thanks and welcome to CPython! 😎

I think that this PR (and certainly the other one) still need NEWS entries. "Trivial" changes and "small" changes don't always mean the same thing, and my personal litmus test is whether or not the changes touch live code. A NEWS entry only takes seconds to write, but can be extremely valuable when digging through changes or debugging regressions later. It seems, at least in my experience, that first-timers are usually a bit too hesitant to include them.

I'll take a look over at #4819 and see what the holdup is.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

finefoot reacted with thumbs up emoji
All reactions
  • 👍 1 reaction
@bedevere-bot bedevere-bot added awaiting core review and removed awaiting review labels Sep 25, 2019
@brandtbucher brandtbucher mentioned this pull request Sep 25, 2019
bpo-30256: pass all BaseProxy arguments through AutoProxy #4819
Closed
@finefoot
Create 2019-09-25-18-10-10.bpo-30256.A5i76Q.rst
3f6cd15
brandtbucher
brandtbucher approved these changes Sep 25, 2019
View reviewed changes
@finefoot

finefoot commented Sep 25, 2019

Copy link
Copy Markdown
Contributor Author

Thanks for the response :) Sorry, you're right - I had a look at only the most recent pull request which introduced a change to _test_multiprocessing.py which doesn't have a NEWS entry and makes use of the "skip news" label. I should have also looked at a few old pull requests which all do have a NEWS entry. ;) So 3f6cd15 should fix that now, I think.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@uSpike Jordan Speicher
bpo-30256: pass all BaseProxy arguments through AutoProxy
2cc5894
@matrixise
matrixise requested a review from applio September 26, 2019 09:20
@pitrou

pitrou commented Feb 21, 2020

Copy link
Copy Markdown
Member

Sorry for the delay. @finefoot is it possible for you to integrate the #4819 fix in this PR? It will make it easier to review and merge.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@finefoot

finefoot commented Feb 21, 2020

Copy link
Copy Markdown
Contributor Author

@pitrou I thought, I'd just adopt the changes directly from master once #4819 got merged. (To show the tests will pass, then.) But you want me to add the changes to managers.py with an individual commit before that, yes?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@pitrou

pitrou commented Feb 21, 2020

Copy link
Copy Markdown
Member

The fix for a bug and the tests for it should be logically contained in a single PR, so I'd like this PR (or the other one) to contain both.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@finefoot
Merge pull request #2 from uSpike/multiprocessing_autoproxy_fix
b5877a0
Multiprocessing autoproxy fix
@finefoot

finefoot commented Feb 21, 2020

Copy link
Copy Markdown
Contributor Author

@pitrou 👍 Done, I've merged the changes from #4819 and the tests just passed.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@codecov

codecov Bot commented Feb 21, 2020 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown

Codecov Report

Merging #16341 into master will increase coverage by 2.50%.
The diff coverage is n/a.

Impacted file tree graph

@@             Coverage Diff             @@
##           master   #16341       +/-   ##
===========================================
+ Coverage   79.56%   82.06%    +2.50%     
===========================================
  Files         381     1955     +1574     
  Lines      168109   584066   +415957     
  Branches        0    44458    +44458     
===========================================
+ Hits       133753   479337   +345584     
- Misses      34356    95105    +60749     
- Partials        0     9624     +9624     
Impacted Files Coverage Δ
Modules/_decimal/libmpdec/umodarith.h 80.76% <0.00%> (-19.24%) ⬇️
Modules/fcntlmodule.c 71.17% <0.00%> (-10.26%) ⬇️
Modules/_ctypes/_ctypes_test.c 62.67% <0.00%> (-7.90%) ⬇️
Include/cpython/abstract.h 97.36% <0.00%> (-2.64%) ⬇️
Modules/clinic/mathmodule.c.h 85.71% <0.00%> (-2.33%) ⬇️
Modules/clinic/fcntlmodule.c.h 60.91% <0.00%> (-2.30%) ⬇️
Objects/codeobject.c 82.59% <0.00%> (-2.20%) ⬇️
Modules/syslogmodule.c 81.25% <0.00%> (-2.09%) ⬇️
Modules/_sqlite/util.c 64.86% <0.00%> (-1.81%) ⬇️
Modules/readline.c 73.51% <0.00%> (-1.43%) ⬇️
... and 1724 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bfd0c96...b5877a0. Read the comment docs.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@pitrou

pitrou commented Mar 8, 2020

Copy link
Copy Markdown
Member

@finefoot You can leave the codecov failure alone, it's not relevant. The patch is in my todo list, it's just that I need to familiarize a bit with the code around that feature.

finefoot reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@applio

applio commented May 6, 2020

Copy link
Copy Markdown
Member

I first left his comment on #4819 but I perhaps should have added it here instead:

When testing this fix, this post on StackOverflow suggests they experienced a seg fault: https://stackoverflow.com/questions/56716470/multiprocessing-manager-nested-shared-objects-doesnt-work-with-queue

Has this been investigated and resolved?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@finefoot

finefoot commented May 6, 2020

Copy link
Copy Markdown
Contributor Author

Note that the solution I referenced in comments in @Darkonaut's answer produced seg faults in my own testing, but this solution did not.

From https://stackoverflow.com/a/58668088

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@uSpike

uSpike commented Feb 24, 2021

Copy link
Copy Markdown
Contributor

Thank you @finefoot for creating this MR with additional tests. As I mentioned in #4819 and @finefoot noted above, it seems to me that the noted stackoverflow comment was not suggesting that this change causes crashes. And, two other commenters in #4819 including myself are unable to reproduce it. It'd be really, really, really nice to have this in 3.10.

tstordyallison, malmiteria, and finefoot reacted with thumbs up emoji
All reactions
  • 👍 3 reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@ambv ambv removed needs backport to 3.7 labels May 4, 2021
@gvanrossum gvanrossum added needs backport to 3.9 labels Jul 2, 2021
gvanrossum
gvanrossum approved these changes Jul 2, 2021
View reviewed changes

@gvanrossum gvanrossum left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. I'm not sure how the test relates to the fix, but I'm too lazy to run all the tests and I trust this. I'll add a Co-Authored-By: uSpike header since the patch is @uSpike's.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

uSpike reacted with heart emoji
All reactions
  • ❤️ 1 reaction
@bedevere-bot bedevere-bot added awaiting merge and removed awaiting core review labels Jul 2, 2021
@gvanrossum

gvanrossum commented Jul 2, 2021

Copy link
Copy Markdown
Member

I'll close and reopen this PR to force the tests to re-run, otherwise I can't merge it.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@gvanrossum gvanrossum closed this Jul 2, 2021
@gvanrossum

gvanrossum commented Jul 2, 2021

Copy link
Copy Markdown
Member

...Aaaaand we're back!

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@gvanrossum gvanrossum reopened this Jul 2, 2021
@gvanrossum

gvanrossum commented Jul 2, 2021

Copy link
Copy Markdown
Member
  • What I should add:
Co-Authored-By:  Jordan Speicher <[email protected]>
  • Waiting for tests to pass.
  • Hopefully I can merge despite the failing codecov test.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@gvanrossum
gvanrossum merged commit 85b9204 into python:main Jul 2, 2021
@miss-islington

miss-islington commented Jul 2, 2021

Copy link
Copy Markdown
Contributor

Thanks @finefoot for the PR, and @gvanrossum for merging it 🌮🎉.. I'm working now to backport this PR to: 3.10, 3.9.
🐍🍒⛏🤖

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot bedevere-bot removed the awaiting merge label Jul 2, 2021
@bedevere-bot

bedevere-bot commented Jul 2, 2021

Copy link
Copy Markdown

GH-26987 is a backport of this pull request to the 3.10 branch.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot bedevere-bot removed the needs backport to 3.10 label Jul 2, 2021
miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Jul 2, 2021
@finefoot @miss-islington
bpo-30256: Add manager_owned keyword arg to AutoProxy (pythonGH-16341)
f1e26ec
Co-authored-by: Jordan Speicher <[email protected]>
(cherry picked from commit 85b9204)

Co-authored-by: finefoot <[email protected]>
miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Jul 2, 2021
@finefoot @miss-islington
bpo-30256: Add manager_owned keyword arg to AutoProxy (pythonGH-16341)
5da992e
Co-authored-by: Jordan Speicher <[email protected]>
(cherry picked from commit 85b9204)

Co-authored-by: finefoot <[email protected]>
gvanrossum pushed a commit that referenced this pull request Jul 2, 2021
@miss-islington @finefoot
bpo-30256: Add manager_owned keyword arg to AutoProxy (GH-16341) (#26987
3ec3e0f
)

Co-authored-by: Jordan Speicher <[email protected]>
(cherry picked from commit 85b9204)

Co-authored-by: finefoot <[email protected]>
@gvanrossum gvanrossum added needs backport to 3.9 and removed needs backport to 3.9 labels Jul 2, 2021
@miss-islington

miss-islington commented Jul 2, 2021

Copy link
Copy Markdown
Contributor

Thanks @finefoot for the PR, and @gvanrossum for merging it 🌮🎉.. I'm working now to backport this PR to: 3.9.
🐍🍒⛏🤖

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Jul 2, 2021
@finefoot @miss-islington
bpo-30256: Add manager_owned keyword arg to AutoProxy (pythonGH-16341)
0aa279a
Co-authored-by: Jordan Speicher <[email protected]>
(cherry picked from commit 85b9204)

Co-authored-by: finefoot <[email protected]>
@bedevere-bot

bedevere-bot commented Jul 2, 2021

Copy link
Copy Markdown

GH-26989 is a backport of this pull request to the 3.9 branch.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot bedevere-bot removed the needs backport to 3.9 label Jul 2, 2021
gvanrossum pushed a commit that referenced this pull request Jul 2, 2021
@miss-islington @finefoot
bpo-30256: Add manager_owned keyword arg to AutoProxy (GH-16341) (GH-…
8aa45de
…26989)

Co-authored-by: Jordan Speicher <[email protected]>
(cherry picked from commit 85b9204)

Co-authored-by: finefoot <[email protected]>
@pablogsal pablogsal mentioned this pull request Jul 18, 2021
[3.10] Correct the order of regen-abidump #27228
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gvanrossum gvanrossum gvanrossum approved these changes
@brandtbucher brandtbucher brandtbucher approved these changes
@applio applio Awaiting requested review from applio

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

tests Tests in the Lib/test dir

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

10 participants

@finefoot @pitrou @applio @uSpike @gvanrossum @miss-islington @bedevere-bot @brandtbucher @ambv @the-knights-who-say-ni
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.