Description

assignee = None
closed_at = <Date 2022-03-19.13:19:59.142>
created_at = <Date 2021-06-17.05:05:24.162>
labels = ['type-bug', 'library', '3.9', '3.10', '3.11']
title = 'stdlib wrongly uses len() for bytes-like object'
updated_at = <Date 2022-03-19.13:19:59.141>
user = 'https://github.com/animalize'

activity = <Date 2022-03-19.13:19:59.141>
actor = 'malin'
assignee = 'none'
closed = True
closed_date = <Date 2022-03-19.13:19:59.142>
closer = 'malin'
components = ['Library (Lib)']
creation = <Date 2021-06-17.05:05:24.162>
creator = 'malin'
dependencies = []
files = []
hgrepos = []
issue_num = 44439
keywords = ['patch']
message_count = 14.0
messages = ['395971', '395973', '395976', '396305', '396309', '396334', '396335', '396336', '405948', '414737', '414742', '414743', '415528', '415549']
nosy_count = 6.0
nosy_names = ['pitrou', 'nadeem.vawda', 'serhiy.storchaka', 'malin', 'miss-islington', 'iritkatriel']
pr_nums = ['26764', '26845', '26846', '29468', '31755', '31756']
priority = 'normal'
resolution = 'later'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue44439'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

Activity

1. animalize commented on Jun 17, 2021

   

   animalizemannequin

   on Jun 17, 2021

   MannequinAuthor

   More actions

   If run this code, it will raise an exception:

       import pickle
       import lzma
       import pandas as pd
       with lzma.open("test.xz", "wb") as file:
           pickle.dump(pd.DataFrame(range(1_000_000)), file, protocol=5)

   The exception:

       Traceback (most recent call last):
         File "E:\testlen.py", line 7, in <module>
           pickle.dump(pd.DataFrame(range(1_000_000)), file, protocol=5)
         File "D:\Python39\lib\lzma.py", line 234, in write
           self._pos += len(data)
       TypeError: object of type 'pickle.PickleBuffer' has no len()
       
   The exception is raised in lzma.LZMAFile.write() method:
   https://github.com/python/cpython/blob/v3.10.0b2/Lib/lzma.py#L238
           
   PickleBuffer doesn't have .__len__ method, is it intended?
2. serhiy-storchaka commented on Jun 17, 2021

   

   serhiy-storchaka

   on Jun 17, 2021

   Member

   More actions

   Oh, LZMAFile.write() should not use len() directly on input data because it does not always work correctly with memoryview and other objects supporting the buffer protocol. It should use memoryview(data).nbytes or data = memoryview(data).cast('B') if other byte-oriented operations (indexing, slicing) are used. See for example Lib/gzip.py, Lib/_pyio.py, Lib/_compression.py, Lib/ssl.py, Lib/socketserver.py, Lib/wave.py.
3. animalize commented on Jun 17, 2021

   

   animalizemannequin

   on Jun 17, 2021

   MannequinAuthor

   More actions

   Ok, I'm working on a PR.
4. animalize commented on Jun 22, 2021

   

   animalizemannequin

   on Jun 22, 2021

   MannequinAuthor

   More actions

   I am checking all the .py files in Lib folder.
   hmac.py has two len() bugs:
   https://github.com/python/cpython/blob/v3.10.0b3/Lib/hmac.py#L212
   https://github.com/python/cpython/blob/v3.10.0b3/Lib/hmac.py#L214

   I think PR 26764 is prepared, it fixes the len() bugs in bz2.py/lzma.py files.
5. animalize

   changed the title [-]PickleBuffer doesn't have __len__ method[/-] [+]stdlib wrongly uses len() for bytes-like object[/+] on Jun 22, 2021
6. animalize

   changed the title [-]PickleBuffer doesn't have __len__ method[/-] [+]stdlib wrongly uses len() for bytes-like object[/+] on Jun 22, 2021
7. serhiy-storchaka commented on Jun 22, 2021

   

   serhiy-storchaka

   on Jun 22, 2021

   Member

   More actions

   New changeset bc6c12c by Ma Lin in branch 'main':
   bpo-44439: BZ2File.write() / LZMAFile.write() handle buffer protocol correctly (GH-26764)
   bc6c12c
8. serhiy-storchaka commented on Jun 22, 2021

   

   serhiy-storchaka

   on Jun 22, 2021

   Member

   More actions

   New changeset 8bc26d8 by Ma Lin in branch '3.9':
   bpo-44439: BZ2File.write()/LZMAFile.write() handle length correctly (GH-26846)
   8bc26d8
9. serhiy-storchaka commented on Jun 22, 2021

   

   serhiy-storchaka

   on Jun 22, 2021

   Member

   More actions

   Thank you for your contribution Ma Lin.
10. serhiy-storchaka

    added

    3.9 (EOL)end of lifeend of life

    3.10 (EOL)end of lifeend of life

    3.11only security fixesonly security fixes

    stdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directory

    on Jun 22, 2021
11. serhiy-storchaka

    closed this as completedon Jun 22, 2021
12. serhiy-storchaka

    added

    type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

    on Jun 22, 2021
13. 5 remaining items
14. serhiy-storchaka

    added

    type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

    on Jun 22, 2021
15. serhiy-storchaka commented on Jun 22, 2021

    

    serhiy-storchaka

    on Jun 22, 2021

    Member

    More actions

    New changeset 01858fb by Miss Islington (bot) in branch '3.10':
    bpo-44439: BZ2File.write() / LZMAFile.write() handle buffer protocol correctly (GH-26764) (GH-26845)
    01858fb
16. animalize commented on Nov 8, 2021

    

    animalizemannequin

    on Nov 8, 2021

    MannequinAuthor

    More actions

    Serhiy Storchaka:

    Sorry, I found zipfile module also has this bug, fixed in PR29468.

    This bug was reported & fixed by GitHub user marcoffee firstly, so I list him as a co-author, his work:
    https://github.com/animalize/pyzstd/issues/4

    The second commit fixes an omission of bpo-41735, a very simple fix, I fix it in PR29468 by the way.
17. animalize

    reopened this on Nov 8, 2021
18. animalize

    reopened this on Nov 8, 2021
19. serhiy-storchaka commented on Mar 8, 2022

    

    serhiy-storchaka

    on Mar 8, 2022

    Member

    More actions

    New changeset 36dd739 by Ma Lin in branch 'main':
    bpo-44439: _ZipWriteFile.write() handle buffer protocol correctly (GH-29468)
    36dd739
20. miss-islington commented on Mar 8, 2022

    

    miss-islington

    on Mar 8, 2022

    Contributor

    More actions

    New changeset 21c5b3f by Miss Islington (bot) in branch '3.10':
    bpo-44439: _ZipWriteFile.write() handle buffer protocol correctly (GH-29468)
    21c5b3f
21. miss-islington commented on Mar 8, 2022

    

    miss-islington

    on Mar 8, 2022

    Contributor

    More actions

    New changeset 0663ca1 by Miss Islington (bot) in branch '3.9':
    bpo-44439: _ZipWriteFile.write() handle buffer protocol correctly (GH-29468)
    0663ca1
22. iritkatriel commented on Mar 18, 2022

    

    iritkatriel

    on Mar 18, 2022

    Member

    More actions

    Can this be closed now or is there anything else to do?
23. animalize commented on Mar 19, 2022

    

    animalizemannequin

    on Mar 19, 2022

    MannequinAuthor

    More actions

    _Stream.write method in tarfile.py also has this code:
    https://github.com/python/cpython/blob/v3.11.0a6/Lib/tarfile.py#L434

    But this bug will not be triggered. When calling this method, always pass bytes data.

    _ConnectionBase.send_bytes method in multiprocessing\connection.py can be micro-optimized:
    https://github.com/python/cpython/blob/v3.11.0a6/Lib/multiprocessing/connection.py#L193
    This can be done in another issue.

    So I think this issue can be closed.
24. animalize

    closed this as completedon Mar 19, 2022
25. animalize

    closed this as completedon Mar 19, 2022
26. ezio-melotti

    transferred this issue fromon Apr 10, 2022
27. jakirkham commented on Apr 12, 2022

    

    jakirkham

    on Apr 12, 2022

    More actions

    Would it make sense to backport this change to Python 3.8 or would that not make sense?
28. iritkatriel commented on Apr 12, 2022

    

    iritkatriel

    on Apr 12, 2022

    Member

    More actions

    3.8 is in security fix only mode. Unless this is a security issue it cannot be backported there.

    Reacted by jakirkham

Issue actions