GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-109370: Support closing Connection and PipeConnection from other thread by serhiy-storchaka · Pull Request #109397 · python/cpython

Conversation

@serhiy-storchaka

@serhiy-storchaka serhiy-storchaka commented Sep 14, 2023 •
edited by bedevere-app Bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member
  • Issue: Unexpected traceback output in test_concurrent_futures and crash #109370

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@serhiy-storchaka
pythongh-109370: Support closing Connection and PipeConnection from o…
a384425
…ther thread
@serhiy-storchaka
serhiy-storchaka requested a review from vstinner September 14, 2023 09:58
@bedevere-app bedevere-app Bot mentioned this pull request Sep 14, 2023
Unexpected traceback output in test_concurrent_futures and crash #109370
Closed
@serhiy-storchaka

serhiy-storchaka commented Sep 14, 2023

Copy link
Copy Markdown
Member Author

I do not like this.

  1. It is too complicated and verbose.
  2. There is still a chance for race condition, if between pushing self.handler on the stack and calling the corresponding OS function, the handler was closed and a new file was open that reuses the same fileno.

I have not tested the code on Windows yet, very likely more try/exept or return code checks should be added in Windows specific code.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@serhiy-storchaka serhiy-storchaka added the DO-NOT-MERGE label Sep 14, 2023
vstinner
vstinner reviewed Sep 14, 2023
View reviewed changes
Comment thread Lib/multiprocessing/connection.py
_winapi.PeekNamedPipe(self._handle)[0] != 0):
return True
except OSError as err:
if err.errno == errno.EBADF:

@vstinner vstinner Sep 14, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should not get EBADF. We should just not call functions on closed handle. If we land in this case, we failed badly before, no?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@serhiy-storchaka serhiy-storchaka Sep 23, 2023

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How can we know that it was closed in other thread without using locks?

You can get EBADF in the following scenario:

  • Thread A reads self._handle and puts in on the stack.
  • Thread B closes it.
  • Thread A calls function with an already closed file descriptor on the stack.
  • That function fails with EBADF errno.

It is the bast case. The worst case is:

  • Thread A reads self._handle and puts in on the stack.
  • Thread B closes it.
  • Thread B or C opens a new file descriptor which reuses the same integer value.
  • Thread A calls function with a file descriptor which now refers to unrelated file or socket.
  • Unpredictable consequences.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
vstinner
vstinner reviewed Sep 15, 2023
View reviewed changes
Comment thread Lib/multiprocessing/connection.py
raise
except TypeError:
self._check_closed()
raise

@vstinner vstinner Sep 15, 2023

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be possible to split this change in two parts? Keep this change for _check_closed() change which now raises BrokenPipeError, and you changes around close(): these ones look straightforward are correct. But write a separated PR to add these try/except?

I dislike these try/except. If there is a risk to land into EBADF case with the current code, maybe some kind of locking is needed to not call the Windows API with a handle, while another thread call close() which invalidates the handle.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@serhiy-storchaka

serhiy-storchaka commented Sep 23, 2023

Copy link
Copy Markdown
Member Author

The more I look at this, the more I dislike it. I do not see how is it possible to safely close a file descriptor on other thread without locking. Making Connection and PipeConnection truly thread-safe in a task on completely different level.

I opened #109780 instead.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@serhiy-storchaka serhiy-storchaka closed this Sep 23, 2023
@vstinner

vstinner commented Sep 25, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

Making Connection and PipeConnection truly thread-safe in a task on completely different level.

Maybe we should just give up and document that they are not thread-safe.

I recall that I did that once on some asyncio classes.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@serhiy-storchaka serhiy-storchaka mentioned this pull request Sep 25, 2023
gh-109370: Fix unexpected traceback output in test_concurrent_futures #109780
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@vstinner vstinner vstinner left review comments
@gpshead gpshead Awaiting requested review from gpshead gpshead will be requested when the pull request is marked ready for review gpshead is a code owner

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

DO-NOT-MERGE

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

2 participants

@serhiy-storchaka @vstinner
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.