GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — bpo-44439: _ZipWriteFile.write() handle buffer protocol correctly · Pull Request #29468 · python/cpython

Conversation

@ghost

@ghost ghost commented Nov 8, 2021 •
edited by bedevere-bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown

https://bugs.python.org/issue44439

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
wjssz and others added 2 commits November 8, 2021 20:42
@wjssz @marcoffee
1. _ZipWriteFile.write() handle buffer protocol correctly
f53ae11
No longer use len() to get the length of the input data. For some buffer protocol objects,
the length obtained by using len() is wrong.

Co-authored-by: Marco Ribeiro <[email protected]>
@wjssz
2. zlib: fix thread lock may go wrong in rare cases
d103eb0
Getting `self->unconsumed_tail` before acquiring the thread lock may mix up decompress state.
@the-knights-who-say-ni the-knights-who-say-ni added the CLA signed label Nov 8, 2021
@bedevere-bot bedevere-bot added the awaiting review label Nov 8, 2021
eryksun
eryksun reviewed Nov 8, 2021
View reviewed changes
Comment thread Lib/zipfile.py
if isinstance(data, (bytes, bytearray)):
nbytes = len(data)
else:
data = memoryview(data)

@eryksun eryksun Nov 8, 2021

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's recommended to manually release a memoryview. Implementations other than CPython may use a garbage collector that doesn't immediately finalize unreferenced objects. Maybe it's simpler to always use a memoryview:

        # Accept any data that supports the buffer protocol
        with memoryview(data) as data:
            nbytes = data.nbytes
            self._file_size += nbytes
            self._crc = crc32(data, self._crc)
            if self._compressor:
                data = self._compressor.compress(data)
                self._compress_size += len(data)
            self._fileobj.write(data)
            return nbytes

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@ghost ghost Nov 8, 2021 •
edited by ghost
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point.
There are some sites in stdlib have this problem, maybe they can be solved in another issue together.
In addition, I suspect if other Python implementations don't release the underlying buffer of memoryview, the program will run abnormally.

Let @serhiy-storchaka decide.

update: I ran a benchmark, the performances are no significant different.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@serhiy-storchaka serhiy-storchaka Mar 7, 2022

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@eryksun is right. But it may be simpler to merge the current solution and fix issues with releasing memoryviews in all places uniformly later.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@wjssz
Revert "2. zlib: fix thread lock may go wrong in rare cases"
d6d5548
Fix this in another issue.
@github-actions

github-actions Bot commented Dec 15, 2021

Copy link
Copy Markdown

This PR is stale because it has been open for 30 days with no activity.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@github-actions github-actions Bot added the stale Stale PR or inactive for long period of time. label Dec 15, 2021
MaxwellDupre
MaxwellDupre approved these changes Mar 3, 2022
View reviewed changes

@MaxwellDupre MaxwellDupre left a comment

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ran 258 tests in 27.542s

OK (skipped=1)

== Tests result: SUCCESS ==
Look ok to me.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@bedevere-bot bedevere-bot added awaiting core review and removed awaiting review labels Mar 3, 2022
serhiy-storchaka
serhiy-storchaka reviewed Mar 7, 2022
View reviewed changes
Comment thread Lib/test/test_zipfile.py Outdated
with zipfile.ZipFile(io.BytesIO(), 'w') as zip:
with zip.open('data', 'w') as data:
self.assertEqual(data.write(q), LENGTH)
self.assertEqual(data._file_size, LENGTH)

@serhiy-storchaka serhiy-storchaka Mar 7, 2022

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_file_size is a private attribute. It is better to use a public API in tests. For example read the content of the file.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Lib/test/test_zipfile.py Outdated
# quickly.
self.assertRaises(OSError, zipfile.ZipFile, TESTFN)

def test_issue44439(self):

@serhiy-storchaka serhiy-storchaka Mar 7, 2022

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be better to move this test to AbstractWriterTests and test with different compressions.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Lib/zipfile.py
if isinstance(data, (bytes, bytearray)):
nbytes = len(data)
else:
data = memoryview(data)

@serhiy-storchaka serhiy-storchaka Mar 7, 2022

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@eryksun is right. But it may be simpler to merge the current solution and fix issues with releasing memoryviews in all places uniformly later.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Misc/NEWS.d/next/Library/2021-11-08-20-27-41.bpo-44439.I_8qro.rst Outdated
@@ -0,0 +1,2 @@
Fix in `_ZipWriteFile.write()` method, when the input data is an object that

@serhiy-storchaka serhiy-storchaka Mar 7, 2022

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_ZipWriteFile is a private class. It would be better to reword the NEWS entry in terms of the public API.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@serhiy-storchaka serhiy-storchaka removed the stale Stale PR or inactive for long period of time. label Mar 7, 2022
@wjssz
address comments
6ba267a
serhiy-storchaka
serhiy-storchaka approved these changes Mar 7, 2022
View reviewed changes

@serhiy-storchaka serhiy-storchaka left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. LGTM.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@bedevere-bot bedevere-bot added awaiting merge and removed awaiting core review labels Mar 7, 2022
@wjssz
add a newline to NEWS
bdb8487
@ghost

ghost commented Mar 7, 2022

Copy link
Copy Markdown
Author

Thanks for your review.

Close and reopen to trigger tests.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@ghost ghost closed this Mar 7, 2022
@ghost ghost reopened this Mar 7, 2022
@wjssz
fix NEWS
451e24f
@ghost

ghost commented Mar 7, 2022

Copy link
Copy Markdown
Author

@serhiy-storchaka
I modified the NEWS file slightly, the tests are all green now.

serhiy-storchaka reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@serhiy-storchaka
serhiy-storchaka merged commit 36dd739 into python:main Mar 8, 2022
@bedevere-bot bedevere-bot removed the awaiting merge label Mar 8, 2022
@serhiy-storchaka serhiy-storchaka added needs backport to 3.9 type-bug An unexpected behavior, bug, or error labels Mar 8, 2022
@miss-islington

miss-islington commented Mar 8, 2022

Copy link
Copy Markdown
Contributor

Thanks @animalize for the PR, and @serhiy-storchaka for merging it 🌮🎉.. I'm working now to backport this PR to: 3.9.
🐍🍒⛏🤖

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@miss-islington

miss-islington commented Mar 8, 2022

Copy link
Copy Markdown
Contributor

Thanks @animalize for the PR, and @serhiy-storchaka for merging it 🌮🎉.. I'm working now to backport this PR to: 3.10.
🐍🍒⛏🤖

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot

bedevere-bot commented Mar 8, 2022

Copy link
Copy Markdown

GH-31755 is a backport of this pull request to the 3.9 branch.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot bedevere-bot removed the needs backport to 3.9 label Mar 8, 2022
miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Mar 8, 2022
@miss-islington
bpo-44439: _ZipWriteFile.write() handle buffer protocol correctly (py…
fca2939
…thonGH-29468)

Co-authored-by: Marco Ribeiro <[email protected]>
(cherry picked from commit 36dd739)

Co-authored-by: Ma Lin <[email protected]>
@bedevere-bot bedevere-bot removed the needs backport to 3.10 label Mar 8, 2022
@bedevere-bot

bedevere-bot commented Mar 8, 2022

Copy link
Copy Markdown

GH-31756 is a backport of this pull request to the 3.10 branch.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Mar 8, 2022
@miss-islington
bpo-44439: _ZipWriteFile.write() handle buffer protocol correctly (py…
fa08197
…thonGH-29468)

Co-authored-by: Marco Ribeiro <[email protected]>
(cherry picked from commit 36dd739)

Co-authored-by: Ma Lin <[email protected]>
miss-islington added a commit that referenced this pull request Mar 8, 2022
@miss-islington
bpo-44439: _ZipWriteFile.write() handle buffer protocol correctly (GH…
21c5b3f
…-29468)

Co-authored-by: Marco Ribeiro <[email protected]>
(cherry picked from commit 36dd739)

Co-authored-by: Ma Lin <[email protected]>
miss-islington added a commit that referenced this pull request Mar 8, 2022
@miss-islington
bpo-44439: _ZipWriteFile.write() handle buffer protocol correctly (GH…
0663ca1
…-29468)

Co-authored-by: Marco Ribeiro <[email protected]>
(cherry picked from commit 36dd739)

Co-authored-by: Ma Lin <[email protected]>
@ghost
ghost deleted the fix_compression branch March 8, 2022 12:11
@animalize animalize mannequin mentioned this pull request Apr 12, 2022
stdlib wrongly uses len() for bytes-like object #88605
Closed
hello-adam pushed a commit to hello-adam/cpython that referenced this pull request Jun 2, 2022
@miss-islington @hello-adam
bpo-44439: _ZipWriteFile.write() handle buffer protocol correctly (py…
c2724a0
…thonGH-29468)

Co-authored-by: Marco Ribeiro <[email protected]>
(cherry picked from commit 36dd739)

Co-authored-by: Ma Lin <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

type-bug An unexpected behavior, bug, or error

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

7 participants

@miss-islington @bedevere-bot @eryksun @serhiy-storchaka @MaxwellDupre @the-knights-who-say-ni @wjssz
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.