GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — io: improve bytes handling by sobolevn · Pull Request #9059 · python/typeshed

Conversation

@sobolevn

@sobolevn sobolevn commented Nov 1, 2022

Copy link
Copy Markdown
Member

IncrementalNewlineDecoder.decode does not support str, but supports bytearray and memoryview:

>>> import codecs, io
>>> dec = codecs.getincrementaldecoder('utf8')()
>>> d = io.IncrementalNewlineDecoder(dec, translate=True)
>>> d.decode('str')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/sobolev/.pyenv/versions/3.10.0/lib/python3.10/codecs.py", line 321, in decode
    data = self.buffer + input
TypeError: can't concat str to bytes

Buffers:

>>> d.decode(bytearray(b'\r'))
'\n'
>>> d.decode(memoryview(b'\r'))
'\n'
>>> d.decode(b'\r')
'\n'

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@sobolevn
io: improve bytes handling
05a2d77
@github-actions

This comment has been minimized.

Sign in to view
@AlexWaygood AlexWaygood mentioned this pull request Nov 1, 2022
Track review of bytes types #9006
Open
@srittau

srittau commented Nov 1, 2022 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Collaborator

It isn't that easy, unfortunately:

Python 3.9.2 (default, Feb 28 2021, 17:03:44) 
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import io
>>> d = io.IncrementalNewlineDecoder(None, translate=True)
>>> d.decode('str')
'str'
>>> d.decode(b'bytes')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: decoder should return a string result, not 'bytes'

This is the implemenation: https://github.com/python/cpython/blob/934b25dcc492dcbca4da9d63d0d71dc940fc0375/Modules/_io/textio.c#L284

PyObject *
_PyIncrementalNewlineDecoder_decode(PyObject *myself,
                                    PyObject *input, int final)
{
    PyObject *output;
    Py_ssize_t output_len;
    nldecoder_object *self = (nldecoder_object *) myself;

    if (self->decoder == NULL) {
        PyErr_SetString(PyExc_ValueError,
                        "IncrementalNewlineDecoder.__init__ not called");
        return NULL;
    }

    /* decode input (with the eventual \r from a previous pass) */
    if (self->decoder != Py_None) {
        output = PyObject_CallMethodObjArgs(self->decoder,
            &_Py_ID(decode), input, final ? Py_True : Py_False, NULL);
    }
    else {
        output = input;
        Py_INCREF(output);
    }

    if (check_decoded(output) < 0)
        return NULL;

    /* ... */

So, there are code paths, where only str is allowed, as check_decoded() explicity checks for a str.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@sobolevn
Update io.pyi
f27459f
@sobolevn

sobolevn commented Nov 1, 2022

Copy link
Copy Markdown
Member Author

Ok, let's go with an easy solution. I don't have any other ideas :(

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@github-actions

This comment has been minimized.

Sign in to view
@JelleZijlstra

JelleZijlstra commented Nov 2, 2022

Copy link
Copy Markdown
Member

I'm not sure this is quite right either. The argument gets passed to codecs.IncrementalDecoder.decode, which is an abstract method. I left it as taking bytes when I reviewed codecs, because that's what BufferedIncrementalDecoder initializes its buffer to. There may be decoders that accept ReadableBuffer, but I'm not sure it's required. If we make the change in this PR, we should also change codecs.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@sobolevn

sobolevn commented Nov 2, 2022

Copy link
Copy Markdown
Member Author

Yes, you are correct. Here's the full sequence:

(abstract)codecs.IncrementalDecoder -> (abstract)codecs.BufferedIncrementalDecoder -> encodings.*.IncrementalDecoder

Then io.IncrementalNewlineDecoder uses some of the concrete types from encodings.*.
Types in encodings.* already have ReadableBuffer set in this PR: #9043

So, I will create a new PR to fix codecs. types in a moment.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@sobolevn sobolevn mentioned this pull request Nov 2, 2022
codecs: improve bytes handling #9065
Closed
sobolevn added 2 commits November 2, 2022 11:03
@sobolevn
codecs: improve bytes handling
a674b55
@sobolevn
Merge branch 'bytes-io' of https://github.com/sobolevn/typeshed into …
27e9d72
…bytes-io
@github-actions

github-actions Bot commented Nov 2, 2022

Copy link
Copy Markdown
Contributor

According to mypy_primer, this change has no effect on the checked open source code. 🤖🎉

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@sobolevn

sobolevn commented Nov 2, 2022

Copy link
Copy Markdown
Member Author

There's an unrelated failure in the CI.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@JelleZijlstra
JelleZijlstra merged commit f972bdf into python:main Nov 2, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

3 participants

@sobolevn @srittau @JelleZijlstra
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.