GeezerVison

The World Wide Web, restored to 800 pixels of glory.

GeezerVison is showing github.com — bpo-31976: Fix race condition when flushing a file is slow. by benfogle · Pull Request #4331 · python/cpython

Conversation

benfogle commented Nov 8, 2017 •

edited by bedevere-bot

Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link

Copy Markdown

Contributor

Move IS_CLOSED check to after the acquiring the lock
Add a check for self->buffer == NULL in IS_CLOSED
Make _pyio.BufferedWriter close the same way the C version does
Add unit tests

https://bugs.python.org/issue31976

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions


          Fix race condition when flushing a file is slow.

60edd49

* Move IS_CLOSED check to after the acquiring the lock
* Add a check for self->buffer == NULL in IS_CLOSED
* Make _pyio.BufferedWriter close the same way the C version does
* Add unit tests

the-knights-who-say-ni added the CLA signed label

Nov 8, 2017

bedevere-bot added the awaiting review label

Nov 8, 2017

pitrou requested changes

Nov 9, 2017

View reviewed changes

pitrou left a comment

Copy link

Copy Markdown

Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for spotting this and proposing this fix!
Before this goes in, you should address the comments and also a NEWS entry. For the latter, you can use the "blurb" utility.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

Comment thread

Lib/_pyio.py

		with self._write_lock:
		if self.raw is None or self.closed:
		return
		try:

pitrou Nov 9, 2017

Copy link

Copy Markdown

Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't look right. The close() method should take the lock once, not take it three times in a row (one of them implicit in flush()). Note _flush_unlocked() can help you here.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

benfogle Nov 10, 2017

Copy link

Copy Markdown

Contributor Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That made a bunch of tests (test_flush_error_on_close, etc.) fail since close() no longer calls flush() if the file object is subclassed or fileobj.flush is set to something.

This is actually the same behavior as buffered_close in the C implementation. I have added a comment to that effect.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

Comment thread

Modules/_io/bufferedio.c

		if (!ENTER_BUFFERED(self))
		return NULL;

		if (IS_CLOSED(self)) {

pitrou Nov 9, 2017

Copy link

Copy Markdown

Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add a comment and a reference to the bug entry here?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

Comment thread

Lib/test/test_io.py

		self.assertTrue(b.closed)

		def test_slow_close_from_thread(self):
		rawio = self.SlowFlushRawIO()

pitrou Nov 9, 2017

Copy link

Copy Markdown

Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add a comment pointing to the bug entry here?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

pitrou Nov 10, 2017

Copy link

Copy Markdown

Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like you forgot to address this comment.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

benfogle Nov 10, 2017

Copy link

Copy Markdown

Contributor Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, whoops. Thank you.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

Comment thread

Lib/test/test_io.py Outdated

		bufio = self.tp(rawio, 8)
		t = threading.Thread(target=bufio.close)
		t.start()
		time.sleep(0.1) # Long enough that t is sleeping in bufio.close()

pitrou Nov 9, 2017

Copy link

Copy Markdown

Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You could probably use an Event that you would set in SlowFlushRawIO.flush() instead.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

bedevere-bot commented Nov 9, 2017

Copy link

Copy Markdown

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

bedevere-bot added awaiting changes and removed awaiting review labels

Nov 9, 2017


          Requested changes from review.

6f3ca75

benfogle commented Nov 10, 2017

Copy link

Copy Markdown

Contributor Author

I have made the requested changes; please review again

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

bedevere-bot commented Nov 10, 2017

Copy link

Copy Markdown

Thanks for making the requested changes!

@pitrou: please review the changes made to this pull request.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

bedevere-bot added awaiting change review and removed awaiting changes labels

Nov 10, 2017

pitrou and others added 2 commits

November 10, 2017 21:39


          Improve precision of NEWS entry

be568bf


          Added reference to issue number.

d3f86a4

benfogle commented Nov 10, 2017

Copy link

Copy Markdown

Contributor Author

I have made the requested changes; please review again.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

bedevere-bot commented Nov 10, 2017

Copy link

Copy Markdown

Thanks for making the requested changes!

@pitrou: please review the changes made to this pull request.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

pitrou approved these changes

Nov 10, 2017

View reviewed changes

bedevere-bot added awaiting merge and removed awaiting change review labels

Nov 10, 2017

pitrou commented Nov 10, 2017

Copy link

Copy Markdown

Member

Thanks. I'm inclined to merge this after CI approval. I'm not sure I want to backport it to 3.6, though, because there is a tiny but non-zero chance that this might break some odd usage somewhere and I don't want that to happen in a bugfix release.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

pitrou merged commit 9703f09 into python:master

Nov 10, 2017

bedevere-bot removed the awaiting merge label

Nov 10, 2017

Reviewers

pitrou

pitrou approved these changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

4 participants

Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.

Conversation

benfogle commented Nov 8, 2017 • edited by bedevere-bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pitrou left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bedevere-bot commented Nov 9, 2017

Uh oh!

benfogle commented Nov 10, 2017

Uh oh!

bedevere-bot commented Nov 10, 2017

Uh oh!

benfogle commented Nov 10, 2017

Uh oh!

bedevere-bot commented Nov 10, 2017

Uh oh!

pitrou commented Nov 10, 2017

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

benfogle commented Nov 8, 2017 •

edited by bedevere-bot

Loading

Uh oh!

There was an error while loading. Please reload this page.