GeezerVison is showing github.com — Getting a buffer from a Unicode array uses invalid format · Issue #57281 · python/cpython
Getting a buffer from a Unicode array uses invalid format #57281Description
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = 'https://github.com/vstinner'
closed_at = <Date 2012-08-24.18:22:37.401>
created_at = <Date 2011-09-30.00:09:50.065>
labels = ['type-bug', 'library', 'release-blocker']
title = 'Getting a buffer from a Unicode array uses invalid format'
updated_at = <Date 2012-08-24.18:22:37.400>
user = 'https://github.com/vstinner' bugs.python.org fields:
activity = <Date 2012-08-24.18:22:37.400>
actor = 'skrah'
assignee = 'vstinner'
closed = True
closed_date = <Date 2012-08-24.18:22:37.401>
closer = 'skrah'
components = ['Library (Lib)']
creation = <Date 2011-09-30.00:09:50.065>
creator = 'vstinner'
dependencies = []
files = ['26646', '26647', '26704', '26892']
hgrepos = []
issue_num = 13072
keywords = ['patch']
message_count = 45.0
messages = ['144658', '144812', '144814', '144817', '144818', '158381', '158892', '167091', '167109', '167112', '167119', '167122', '167165', '167173', '167520', '167521', '167522', '167540', '167545', '167546', '167547', '167549', '167551', '167561', '167566', '167571', '167673', '167702', '167703', '167708', '167732', '167936', '167947', '167997', '168005', '168369', '168373', '168558', '168561', '168567', '168571', '168575', '169026', '169063', '169065']
nosy_count = 10.0
nosy_names = ['loewis', 'georg.brandl', 'mark.dickinson', 'ncoghlan', 'pitrou', 'vstinner', 'Arfrever', 'skrah', 'meador.inge', 'python-dev']
pr_nums = []
priority = 'release blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue13072'
versions = ['Python 3.3'] Linked PRs
Activityvstinner commented on Sep 30, 2011 In Python 3.2, when you get a buffer from array.array('u'), "u" is used as buffer format. The format is supposed to be a format from the struct module, and "u" is an invalid struct format. "w" is used on wide mode.
I just upgraded the array module to use the new Unicode API (PEP-393). The array now uses a Py_UCS4 buffer. I used "I" or "L" format depending on the size of int and long C types.
It would be better to use a format for a Py_UCS4 string, but struct doesn't support such type.
For Python 2.7 and 3.2, I don't know if it is really a bug or not. skrah commented on Oct 3, 2011 The automatic conversion of 'u' to 'I' or 'L' causes test_buffer
(PEP-3118 repo) to fail:
# Not implemented formats. Ugly, but inevitable. This is the same as
# issue python/cpython#46783: equality is also used for membership testing and must
# return a result.
a = array.array('u', 'xyz')
v = memoryview(a)
self.assertNotEqual(v, a)
self.assertNotEqual(a, v)I don't have a better idea though what to do about 'u' except
officially implementing it for struct and memoryview as well. skrah commented on Oct 3, 2011
It would be better to use a format for a Py_UCS4 string, but struct doesn't support such type.
PEP-3118 suggests for the extended struct syntax:
'c' -> ucs-1 (latin-1) encoding
'u' -> ucs-2
'w' -> ucs-4 vstinner commented on Oct 3, 2011
The automatic conversion of 'u' to 'I' or 'L' causes test_buffer
(PEP-3118 repo) to fail:
Not implemented formats. Ugly, but inevitable. This is the same as
issue bpo-2531: equality is also used for membership testing and must
return a result.
a = array.array('u', 'xyz')
v = memoryview(a)
self.assertNotEqual(v, a)
self.assertNotEqual(a, v)
I don't understand: a buffer format is a format for the struct module,
or for the array module? skrah commented on Oct 3, 2011 STINNER Victor <[email protected]> wrote:
> # Not implemented formats. Ugly, but inevitable. This is the same as
> # issue bpo-2531: equality is also used for membership testing and must
> # return a result.
> a = array.array('u', 'xyz')
> v = memoryview(a)
> self.assertNotEqual(v, a)
> self.assertNotEqual(a, v)
I don't understand: a buffer format is a format for the struct module,
or for the array module?
It's like this: memoryview follows the current struct syntax, which
doesn't have 'u'. memory_richcompare() does not understand 'u', but
is required to return something for __eq__ and __ne__, so it returns
'not equal'.
This isn't so important, since I discovered (see my later post)
that 'u' and 'w' were scheduled for inclusion in the struct
module anyway.
So I think we should focus on whether the proposed 'c', 'u' and 'w'
format specifiers still make sense after the PEP-393 changes. vstinner commented on Apr 16, 2012 @Stefan: What is the status of this issue? skrah commented on Apr 20, 2012 vstinner commented on Aug 1, 2012 Should we do something before Python 3.3 final? skrah commented on Aug 1, 2012 Is it possible without too much effort to keep the old behavior
('u' -> Py_UNICODE)? Then I'd say that should go into 3.3.
The problem with the current behavior is that it's neither backwards
compatible nor PEP-3118 compliant.
If it is too much work to restore the status quo, we could leave this
change with the excuse that 'u' is probably not used very often. vstinner commented on Aug 1, 2012 Here is a patch reverting changes of the PEP-393, as suggested by Martin von Loewis. With the patch, array uses Py_UNICODE* type for the 'u' format. So array.array('u', '\u0010ffff')[0] should return '\uDBFF' on Windows. skrah commented on Aug 1, 2012 The diff between b9558df8cc58 and default with array_revert_pep393.patch
applied is small, but I noticed that in some places you switched back to
Py_UNICODE typecode and in others not. For instance, in struct arraydescr
typecode is still char.
I'm not sure why typecode was originally Py_UNICODE though. vstinner commented on Aug 1, 2012
The diff between b9558df8cc58 and default with array_revert_pep393.patch
applied is small, but I noticed that in some places you switched back to
Py_UNICODE typecode and in others not.
I just copied code from Python 3.2, I forgot to update typecode type
(Py_UNICODE => char). I attach a new patch which changes also the
documentation of the "u" format. skrah commented on Aug 1, 2012 array_revert_pep393-2.patch looks good (checked against 7042a83f37e
and all following commits that should be kept). vstinner commented on Aug 1, 2012 @georg: are you ok with this change? It reverts the behaviour of Python 3.2 and avoids to have to maintain an API that nobody wants to use ('u' format using Py_UCS4, 32 bits unsigned). loewis commented on Aug 24, 2012 Stefan, your patch array_deprecate_u.diff is fine. If you get to it, please also rephrase the clause "Python's unicode type"; not sure what the convention is to refer to Py_UNICODE now (perhaps "historical unicode type"). python-dev commented on Aug 24, 2012 skrah commented on Aug 24, 2012 Good, I think this can be closed then. StanFromIreland commented on Apr 28, 2025 This has been deprecated for a long time, I was unable to find any uses in the top 1000 pypi projects or on gh. This should be set for removal in 3.15. vstinner commented on Apr 28, 2025 Its removal is scheduled for Python 3.16, see the array C code:
if (PyErr_WarnEx(PyExc_DeprecationWarning,
"The 'u' type code is deprecated and "
"will be removed in Python 3.16",
1)) {
return NULL;
}
}
StanFromIreland commented on Apr 28, 2025 I see now why I was mislead, the note is under pending-removal-in-future-versions too. I sent a pr.
MetadataMetadataStatusShow more project fields
DevelopmentNo branches or pull requestsIssue actions
|