GeezerVisonThe World Wide Web, restored to 800 pixels of glory. |
| Home|About|Gaming cabinet|Archives|How it works|Starter sites |
GeezerVison is showing mail.python.org — Mailman 3 March 2014 - Python-Dev - python.org Python-Dev
Threads by month
March 2014
It has been a while since I posted a copy of PEP 1 to the mailing
lists and newsgroups. I've recently done some updating of a few
sections, so in the interest of gaining wider community participation
in the Python development process, I'm posting the latest revision of
PEP 1 here. A version of the PEP is always available on-line at
http://www.python.org/peps/pep-0001.html
Enjoy,
-Barry
-------------------- snip snip --------------------
PEP: 1
Title: PEP Purpose and Guidelines
Version: $Revision: 1.36 $
Last-Modified: $Date: 2002/07/29 18:34:59 $
Author: Barry A. Warsaw, Jeremy Hylton
Status: Active
Type: Informational
Created: 13-Jun-2000
Post-History: 21-Mar-2001, 29-Jul-2002
What is a PEP?
PEP stands for Python Enhancement Proposal. A PEP is a design
document providing information to the Python community, or
describing a new feature for Python. The PEP should provide a
concise technical specification of the feature and a rationale for
the feature.
We intend PEPs to be the primary mechanisms for proposing new
features, for collecting community input on an issue, and for
documenting the design decisions that have gone into Python. The
PEP author is responsible for building consensus within the
community and documenting dissenting opinions.
Because the PEPs are maintained as plain text files under CVS
control, their revision history is the historical record of the
feature proposal[1].
Kinds of PEPs
There are two kinds of PEPs. A standards track PEP describes a
new feature or implementation for Python. An informational PEP
describes a Python design issue, or provides general guidelines or
information to the Python community, but does not propose a new
feature. Informational PEPs do not necessarily represent a Python
community consensus or recommendation, so users and implementors
are free to ignore informational PEPs or follow their advice.
PEP Work Flow
The PEP editor, Barry Warsaw <peps(a)python.org>, assigns numbers
for each PEP and changes its status.
The PEP process begins with a new idea for Python. It is highly
recommended that a single PEP contain a single key proposal or new
idea. The more focussed the PEP, the more successfully it tends
to be. The PEP editor reserves the right to reject PEP proposals
if they appear too unfocussed or too broad. If in doubt, split
your PEP into several well-focussed ones.
Each PEP must have a champion -- someone who writes the PEP using
the style and format described below, shepherds the discussions in
the appropriate forums, and attempts to build community consensus
around the idea. The PEP champion (a.k.a. Author) should first
attempt to ascertain whether the idea is PEP-able. Small
enhancements or patches often don't need a PEP and can be injected
into the Python development work flow with a patch submission to
the SourceForge patch manager[2] or feature request tracker[3].
The PEP champion then emails the PEP editor <peps(a)python.org> with
a proposed title and a rough, but fleshed out, draft of the PEP.
This draft must be written in PEP style as described below.
If the PEP editor approves, he will assign the PEP a number, label
it as standards track or informational, give it status 'draft',
and create and check-in the initial draft of the PEP. The PEP
editor will not unreasonably deny a PEP. Reasons for denying PEP
status include duplication of effort, being technically unsound,
not providing proper motivation or addressing backwards
compatibility, or not in keeping with the Python philosophy. The
BDFL (Benevolent Dictator for Life, Guido van Rossum) can be
consulted during the approval phase, and is the final arbitrator
of the draft's PEP-ability.
If a pre-PEP is rejected, the author may elect to take the pre-PEP
to the comp.lang.python newsgroup (a.k.a. python-list(a)python.org
mailing list) to help flesh it out, gain feedback and consensus
from the community at large, and improve the PEP for
re-submission.
The author of the PEP is then responsible for posting the PEP to
the community forums, and marshaling community support for it. As
updates are necessary, the PEP author can check in new versions if
they have CVS commit permissions, or can email new PEP versions to
the PEP editor for committing.
Standards track PEPs consists of two parts, a design document and
a reference implementation. The PEP should be reviewed and
accepted before a reference implementation is begun, unless a
reference implementation will aid people in studying the PEP.
Standards Track PEPs must include an implementation - in the form
of code, patch, or URL to same - before it can be considered
Final.
PEP authors are responsible for collecting community feedback on a
PEP before submitting it for review. A PEP that has not been
discussed on python-list(a)python.org and/or python-dev(a)python.org
will not be accepted. However, wherever possible, long open-ended
discussions on public mailing lists should be avoided. Strategies
to keep the discussions efficient include, setting up a separate
SIG mailing list for the topic, having the PEP author accept
private comments in the early design phases, etc. PEP authors
should use their discretion here.
Once the authors have completed a PEP, they must inform the PEP
editor that it is ready for review. PEPs are reviewed by the BDFL
and his chosen consultants, who may accept or reject a PEP or send
it back to the author(s) for revision.
Once a PEP has been accepted, the reference implementation must be
completed. When the reference implementation is complete and
accepted by the BDFL, the status will be changed to `Final.'
A PEP can also be assigned status `Deferred.' The PEP author or
editor can assign the PEP this status when no progress is being
made on the PEP. Once a PEP is deferred, the PEP editor can
re-assign it to draft status.
A PEP can also be `Rejected'. Perhaps after all is said and done
it was not a good idea. It is still important to have a record of
this fact.
PEPs can also be replaced by a different PEP, rendering the
original obsolete. This is intended for Informational PEPs, where
version 2 of an API can replace version 1.
PEP work flow is as follows:
Draft -> Accepted -> Final -> Replaced
^
+----> Rejected
v
Deferred
Some informational PEPs may also have a status of `Active' if they
are never meant to be completed. E.g. PEP 1.
What belongs in a successful PEP?
Each PEP should have the following parts:
1. Preamble -- RFC822 style headers containing meta-data about the
PEP, including the PEP number, a short descriptive title
(limited to a maximum of 44 characters), the names, and
optionally the contact info for each author, etc.
2. Abstract -- a short (~200 word) description of the technical
issue being addressed.
3. Copyright/public domain -- Each PEP must either be explicitly
labelled as placed in the public domain (see this PEP as an
example) or licensed under the Open Publication License[4].
4. Specification -- The technical specification should describe
the syntax and semantics of any new language feature. The
specification should be detailed enough to allow competing,
interoperable implementations for any of the current Python
platforms (CPython, JPython, Python .NET).
5. Motivation -- The motivation is critical for PEPs that want to
change the Python language. It should clearly explain why the
existing language specification is inadequate to address the
problem that the PEP solves. PEP submissions without
sufficient motivation may be rejected outright.
6. Rationale -- The rationale fleshes out the specification by
describing what motivated the design and why particular design
decisions were made. It should describe alternate designs that
were considered and related work, e.g. how the feature is
supported in other languages.
The rationale should provide evidence of consensus within the
community and discuss important objections or concerns raised
during discussion.
7. Backwards Compatibility -- All PEPs that introduce backwards
incompatibilities must include a section describing these
incompatibilities and their severity. The PEP must explain how
the author proposes to deal with these incompatibilities. PEP
submissions without a sufficient backwards compatibility
treatise may be rejected outright.
8. Reference Implementation -- The reference implementation must
be completed before any PEP is given status 'Final,' but it
need not be completed before the PEP is accepted. It is better
to finish the specification and rationale first and reach
consensus on it before writing code.
The final implementation must include test code and
documentation appropriate for either the Python language
reference or the standard library reference.
PEP Template
PEPs are written in plain ASCII text, and should adhere to a
rigid style. There is a Python script that parses this style and
converts the plain text PEP to HTML for viewing on the web[5].
PEP 9 contains a boilerplate[7] template you can use to get
started writing your PEP.
Each PEP must begin with an RFC822 style header preamble. The
headers must appear in the following order. Headers marked with
`*' are optional and are described below. All other headers are
required.
PEP: <pep number>
Title: <pep title>
Version: <cvs version string>
Last-Modified: <cvs date string>
Author: <list of authors' real names and optionally, email addrs>
* Discussions-To: <email address>
Status: <Draft | Active | Accepted | Deferred | Final | Replaced>
Type: <Informational | Standards Track>
* Requires: <pep numbers>
Created: <date created on, in dd-mmm-yyyy format>
* Python-Version: <version number>
Post-History: <dates of postings to python-list and python-dev>
* Replaces: <pep number>
* Replaced-By: <pep number>
The Author: header lists the names and optionally, the email
addresses of all the authors/owners of the PEP. The format of the
author entry should be
address(a)dom.ain (Random J. User)
if the email address is included, and just
Random J. User
if the address is not given. If there are multiple authors, each
should be on a separate line following RFC 822 continuation line
conventions. Note that personal email addresses in PEPs will be
obscured as a defense against spam harvesters.
Standards track PEPs must have a Python-Version: header which
indicates the version of Python that the feature will be released
with. Informational PEPs do not need a Python-Version: header.
While a PEP is in private discussions (usually during the initial
Draft phase), a Discussions-To: header will indicate the mailing
list or URL where the PEP is being discussed. No Discussions-To:
header is necessary if the PEP is being discussed privately with
the author, or on the python-list or python-dev email mailing
lists. Note that email addresses in the Discussions-To: header
will not be obscured.
Created: records the date that the PEP was assigned a number,
while Post-History: is used to record the dates of when new
versions of the PEP are posted to python-list and/or python-dev.
Both headers should be in dd-mmm-yyyy format, e.g. 14-Aug-2001.
PEPs may have a Requires: header, indicating the PEP numbers that
this PEP depends on.
PEPs may also have a Replaced-By: header indicating that a PEP has
been rendered obsolete by a later document; the value is the
number of the PEP that replaces the current document. The newer
PEP must have a Replaces: header containing the number of the PEP
that it rendered obsolete.
PEP Formatting Requirements
PEP headings must begin in column zero and the initial letter of
each word must be capitalized as in book titles. Acronyms should
be in all capitals. The body of each section must be indented 4
spaces. Code samples inside body sections should be indented a
further 4 spaces, and other indentation can be used as required to
make the text readable. You must use two blank lines between the
last line of a section's body and the next section heading.
You must adhere to the Emacs convention of adding two spaces at
the end of every sentence. You should fill your paragraphs to
column 70, but under no circumstances should your lines extend
past column 79. If your code samples spill over column 79, you
should rewrite them.
Tab characters must never appear in the document at all. A PEP
should include the standard Emacs stanza included by example at
the bottom of this PEP.
A PEP must contain a Copyright section, and it is strongly
recommended to put the PEP in the public domain.
When referencing an external web page in the body of a PEP, you
should include the title of the page in the text, with a
footnote reference to the URL. Do not include the URL in the body
text of the PEP. E.g.
Refer to the Python Language web site [1] for more details.
...
[1] http://www.python.org
When referring to another PEP, include the PEP number in the body
text, such as "PEP 1". The title may optionally appear. Add a
footnote reference that includes the PEP's title and author. It
may optionally include the explicit URL on a separate line, but
only in the References section. Note that the pep2html.py script
will calculate URLs automatically, e.g.:
...
Refer to PEP 1 [7] for more information about PEP style
...
References
[7] PEP 1, PEP Purpose and Guidelines, Warsaw, Hylton
http://www.python.org/peps/pep-0001.html
If you decide to provide an explicit URL for a PEP, please use
this as the URL template:
http://www.python.org/peps/pep-xxxx.html
PEP numbers in URLs must be padded with zeros from the left, so as
to be exactly 4 characters wide, however PEP numbers in text are
never padded.
Reporting PEP Bugs, or Submitting PEP Updates
How you report a bug, or submit a PEP update depends on several
factors, such as the maturity of the PEP, the preferences of the
PEP author, and the nature of your comments. For the early draft
stages of the PEP, it's probably best to send your comments and
changes directly to the PEP author. For more mature, or finished
PEPs you may want to submit corrections to the SourceForge bug
manager[6] or better yet, the SourceForge patch manager[2] so that
your changes don't get lost. If the PEP author is a SF developer,
assign the bug/patch to him, otherwise assign it to the PEP
editor.
When in doubt about where to send your changes, please check first
with the PEP author and/or PEP editor.
PEP authors who are also SF committers, can update the PEPs
themselves by using "cvs commit" to commit their changes.
Remember to also push the formatted PEP text out to the web by
doing the following:
% python pep2html.py -i NUM
where NUM is the number of the PEP you want to push out. See
% python pep2html.py --help
for details.
Transferring PEP Ownership
It occasionally becomes necessary to transfer ownership of PEPs to
a new champion. In general, we'd like to retain the original
author as a co-author of the transferred PEP, but that's really up
to the original author. A good reason to transfer ownership is
because the original author no longer has the time or interest in
updating it or following through with the PEP process, or has
fallen off the face of the 'net (i.e. is unreachable or not
responding to email). A bad reason to transfer ownership is
because you don't agree with the direction of the PEP. We try to
build consensus around a PEP, but if that's not possible, you can
always submit a competing PEP.
If you are interested assuming ownership of a PEP, send a message
asking to take over, addressed to both the original author and the
PEP editor <peps(a)python.org>. If the original author doesn't
respond to email in a timely manner, the PEP editor will make a
unilateral decision (it's not like such decisions can be
reversed. :).
References and Footnotes
[1] This historical record is available by the normal CVS commands
for retrieving older revisions. For those without direct access
to the CVS tree, you can browse the current and past PEP revisions
via the SourceForge web site at
http://cvs.sourceforge.net/cgi-bin/cvsweb.cgi/python/nondist/peps/?cvsroot=…
[2] http://sourceforge.net/tracker/?group_id=5470&atid=305470
[3] http://sourceforge.net/tracker/?atid=355470&group_id=5470&func=browse
[4] http://www.opencontent.org/openpub/
[5] The script referred to here is pep2html.py, which lives in
the same directory in the CVS tree as the PEPs themselves.
Try "pep2html.py --help" for details.
The URL for viewing PEPs on the web is
http://www.python.org/peps/
[6] http://sourceforge.net/tracker/?group_id=5470&atid=305470
[7] PEP 9, Sample PEP Template
http://www.python.org/peps/pep-0009.html
Copyright
This document has been placed in the public domain.
Local Variables:
mode: indented-text
indent-tabs-mode: nil
sentence-end-double-space: t
fill-column: 70
End:
The current memory layout for dictionaries is
unnecessarily inefficient. It has a sparse table of
24-byte entries containing the hash value, key pointer,
and value pointer.
Instead, the 24-byte entries should be stored in a
dense table referenced by a sparse table of indices.
For example, the dictionary:
d = {'timmy': 'red', 'barry': 'green', 'guido': 'blue'}
is currently stored as:
entries = [['--', '--', '--'],
[-8522787127447073495, 'barry', 'green'],
['--', '--', '--'],
['--', '--', '--'],
['--', '--', '--'],
[-9092791511155847987, 'timmy', 'red'],
['--', '--', '--'],
[-6480567542315338377, 'guido', 'blue']]
Instead, the data should be organized as follows:
indices = [None, 1, None, None, None, 0, None, 2]
entries = [[-9092791511155847987, 'timmy', 'red'],
[-8522787127447073495, 'barry', 'green'],
[-6480567542315338377, 'guido', 'blue']]
Only the data layout needs to change. The hash table
algorithms would stay the same. All of the current
optimizations would be kept, including key-sharing
dicts and custom lookup functions for string-only
dicts. There is no change to the hash functions, the
table search order, or collision statistics.
The memory savings are significant (from 30% to 95%
compression depending on the how full the table is).
Small dicts (size 0, 1, or 2) get the most benefit.
For a sparse table of size t with n entries, the sizes are:
curr_size = 24 * t
new_size = 24 * n + sizeof(index) * t
In the above timmy/barry/guido example, the current
size is 192 bytes (eight 24-byte entries) and the new
size is 80 bytes (three 24-byte entries plus eight
1-byte indices). That gives 58% compression.
Note, the sizeof(index) can be as small as a single
byte for small dicts, two bytes for bigger dicts and
up to sizeof(Py_ssize_t) for huge dict.
In addition to space savings, the new memory layout
makes iteration faster. Currently, keys(), values, and
items() loop over the sparse table, skipping-over free
slots in the hash table. Now, keys/values/items can
loop directly over the dense table, using fewer memory
accesses.
Another benefit is that resizing is faster and
touches fewer pieces of memory. Currently, every
hash/key/value entry is moved or copied during a
resize. In the new layout, only the indices are
updated. For the most part, the hash/key/value entries
never move (except for an occasional swap to fill a
hole left by a deletion).
With the reduced memory footprint, we can also expect
better cache utilization.
For those wanting to experiment with the design,
there is a pure Python proof-of-concept here:
http://code.activestate.com/recipes/578375
YMMV: Keep in mind that the above size statics assume a
build with 64-bit Py_ssize_t and 64-bit pointers. The
space savings percentages are a bit different on other
builds. Also, note that in many applications, the size
of the data dominates the size of the container (i.e.
the weight of a bucket of water is mostly the water,
not the bucket).
Raymond
Semi-official read-only Github mirror of the CPython Mercurial repository
by Eli Bendersky Oct. 15, 2014 Oct. 15, 2014
Hi all,
https://github.com/python/cpython is now live as a semi-official, *read
only* Github mirror of the CPython Mercurial repository. Let me know if you
have any problems/concerns.
I still haven't decided how often to update it (considering either just N
times a day, or maybe use a Hg hook for batching). Suggestions are welcome.
The methodology I used to create it is via hg-fast-export. I also tried to
pack and gc the git repo as much as possible before the initial Github push
- it went down from almost ~2GB to ~200MB (so this is the size of a fresh
clone right now).
Eli
P.S. thanks Jesse for the keys to https://github.com/python
I've received some enthusiastic emails from someone who wants to
revive restricted mode. He started out with a bunch of patches to the
CPython runtime using ctypes, which he attached to an App Engine bug:
http://code.google.com/p/googleappengine/issues/detail?id=671
Based on his code (the file secure.py is all you need, included in
secure.tar.gz) it seems he believes the only security leaks are
__subclasses__, gi_frame and gi_code. (I have since convinced him that
if we add "restricted" guards to these attributes, he doesn't need the
functions added to sys.)
I don't recall the exploits that Samuele once posted that caused the
death of rexec.py -- does anyone recall, or have a pointer to the
threads?
--
--Guido van Rossum (home page: http://www.python.org/~guido/)
May 30, 2014
Hi,
On core-mentorship someone asked about PEP 3145 - Asynchronous I/O for
subprocess.popen. I answered that asyncio now has subprocess support
(including non-blocking I/O on the three standard stream pipes), so
it's not obvious anything else is needed.
Should we change the PEP's status to Rejected or Superseded?
Regards
Antoine.
ACTIVITY SUMMARY (2014-01-31 - 2014-02-07)
Python tracker at http://bugs.python.org/
To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.
Issues counts and deltas:
open 4501 (+17)
closed 27813 (+65)
total 32314 (+82)
Open issues with patches: 2045
Issues opened (65)
==================
#12915: Add inspect.locate and inspect.resolve
http://bugs.python.org/issue12915 reopened by vinay.sajip
#16779: Fix compiler warning when building extension modules on 64-bit
http://bugs.python.org/issue16779 reopened by skrah
#19363: Python 2.7's future_builtins.map is not compatible with Python
http://bugs.python.org/issue19363 reopened by Gareth.Rees
#20053: venv and ensurepip are affected by default pip config file
http://bugs.python.org/issue20053 reopened by ncoghlan
#20461: Argument Clinic included return converters hard code use of ``
http://bugs.python.org/issue20461 opened by brett.cannon
#20462: Python IDLE auto closed when creating a new file or open exist
http://bugs.python.org/issue20462 opened by Bishop
#20463: sqlite dumpiter dumps invalid script when virtual tables are u
http://bugs.python.org/issue20463 opened by Ronny.Pfannschmidt
#20464: Update distutils sample config file in Doc/install/index.rst
http://bugs.python.org/issue20464 opened by zach.ware
#20465: Upgrade SQLite to 3.8.3 with 3.4.0 Windows and OS X installers
http://bugs.python.org/issue20465 opened by ned.deily
#20466: Example in Doc/extending/embedding.rst fails to compile cleanl
http://bugs.python.org/issue20466 opened by zach.ware
#20467: Confusing wording about __init__
http://bugs.python.org/issue20467 opened by BreamoreBoy
#20468: resource module documentation is incorrect
http://bugs.python.org/issue20468 opened by Kurt.Rose
#20469: ssl.getpeercert() should include extensions
http://bugs.python.org/issue20469 opened by oninoshiko
#20475: pystone.py in 3.4 still uses time.clock(), even though it's ma
http://bugs.python.org/issue20475 opened by pfalcon
#20476: If new email policies are used, default message factory should
http://bugs.python.org/issue20476 opened by r.david.murray
#20478: Avoid inadvertently special casing Counter in statistics modul
http://bugs.python.org/issue20478 opened by ncoghlan
#20479: Efficiently support weight/frequency mappings in the statistic
http://bugs.python.org/issue20479 opened by ncoghlan
#20480: Add ipaddress property to get name of reverse DNS PTR record
http://bugs.python.org/issue20480 opened by leonn
#20481: Clarify type coercion rules in statistics module
http://bugs.python.org/issue20481 opened by ncoghlan
#20482: smtplib.SMTP.sendmail: improve exception message
http://bugs.python.org/issue20482 opened by yegle
#20483: Missing network resource checks in test_urllib2 & test_smtplib
http://bugs.python.org/issue20483 opened by ncoghlan
#20484: test_pydoc can alter execution environment causing subsequent
http://bugs.python.org/issue20484 opened by ned.deily
#20485: Enable non-ASCII extension module names
http://bugs.python.org/issue20485 opened by Suzumizaki
#20486: msilib: can't close opened database
http://bugs.python.org/issue20486 opened by gentoo90
#20487: Odd words in unittest.mock document.
http://bugs.python.org/issue20487 opened by naoki
#20490: Show clear error message on circular import
http://bugs.python.org/issue20490 opened by cool-RR
#20491: textwrap: Non-breaking space not honored
http://bugs.python.org/issue20491 opened by kunkku
#20493: asyncio: OverflowError('timeout is too large')
http://bugs.python.org/issue20493 opened by haypo
#20495: test_read_pty_output() hangs on FreeBSD 7.2 buildbot
http://bugs.python.org/issue20495 opened by haypo
#20496: function definition tutorial encourages bad practice
http://bugs.python.org/issue20496 opened by aisaac
#20497: Unclear word in socket document.
http://bugs.python.org/issue20497 opened by naoki
#20499: Rounding errors with statistics.variance
http://bugs.python.org/issue20499 opened by oscarbenjamin
#20500: assertion failed when passing an exception object to sys.exit
http://bugs.python.org/issue20500 opened by xdegaye
#20501: fileinput module will read whole file into memory when using f
http://bugs.python.org/issue20501 opened by gromgull
#20503: super behavioru and abstract base classes (either implementati
http://bugs.python.org/issue20503 opened by Gerrit.Holl
#20504: cgi.FieldStorage, multipart, missing Content-Length
http://bugs.python.org/issue20504 opened by srittau
#20505: Remove resolution from selectors and granularity from asyncio
http://bugs.python.org/issue20505 opened by haypo
#20506: Command to display all available Import Library
http://bugs.python.org/issue20506 opened by arupchakrav
#20507: TypeError from str.join has no message
http://bugs.python.org/issue20507 opened by Gareth.Rees
#20508: IndexError from ipaddress._BaseNetwork.__getitem__ has no mess
http://bugs.python.org/issue20508 opened by Gareth.Rees
#20510: Test cases in test_sys don't match the comments
http://bugs.python.org/issue20510 opened by Gareth.Rees
#20511: asyncio: StreamReader should use bytearray for its internal bu
http://bugs.python.org/issue20511 opened by yselivanov
#20513: Python 2.7. Script interruption on logoff from 0 session under
http://bugs.python.org/issue20513 opened by vScherba
#20515: Null pointer dereference in tkinter module
http://bugs.python.org/issue20515 opened by christian.heimes
#20516: Concurrent.futures base concurrency improvement (with patch)
http://bugs.python.org/issue20516 opened by glangford
#20517: Support errors with two filenames for errno exceptions
http://bugs.python.org/issue20517 opened by larry
#20518: multiple inheritance + C extension = possibly unexpected __bas
http://bugs.python.org/issue20518 opened by Martin.Teichmann
#20519: Replace uuid ctypes usage with an extension module.
http://bugs.python.org/issue20519 opened by gustavo
#20520: Readline test in test_codecs is broken
http://bugs.python.org/issue20520 opened by serhiy.storchaka
#20521: [PATCH] Cleanup for "dis" module documentation
http://bugs.python.org/issue20521 opened by svenberkvens
#20523: global .pdbrc on windows 7 not reachable out of the box
http://bugs.python.org/issue20523 opened by mbyt
#20524: format error messages should provide context information
http://bugs.python.org/issue20524 opened by r.david.murray
#20525: Got compiler warning when compiling readline module
http://bugs.python.org/issue20525 opened by vajrasky
#20526: python: Modules/gcmodule.c:379: visit_decref: Assertion `((gc)
http://bugs.python.org/issue20526 opened by haypo
#20529: Unittest displays ResourceWarning warnings when running tests,
http://bugs.python.org/issue20529 opened by Bernt.Røskar.Brenna
#20530: Change the text signature format (again) to be more robust
http://bugs.python.org/issue20530 opened by larry
#20531: TypeError in e-mail.parser when non-ASCII is present
http://bugs.python.org/issue20531 opened by jason.coombs
#20532: Mark all tests which use _testcapi as CPython only
http://bugs.python.org/issue20532 opened by serhiy.storchaka
#20534: Enum tests fail with pickle protocol 4
http://bugs.python.org/issue20534 opened by ethan.furman
#20535: run_tests.py: Work around issues 20355 and 20361
http://bugs.python.org/issue20535 opened by Arfrever
#20536: statistics._decimal_to_ratio() produces non-integer ratio
http://bugs.python.org/issue20536 opened by skrah
#20537: logging exc_info parameter should accept exception instances
http://bugs.python.org/issue20537 opened by yselivanov
#20538: Segfault in UTF-7 incremental decoder
http://bugs.python.org/issue20538 opened by serhiy.storchaka
#20539: math.factorial may throw OverflowError
http://bugs.python.org/issue20539 opened by ncoghlan
#20540: Python 3.3/3.4 regression in multiprocessing manager ?
http://bugs.python.org/issue20540 opened by Irvin.Probst
Most recent 15 issues with no replies (15)
==========================================
#20536: statistics._decimal_to_ratio() produces non-integer ratio
http://bugs.python.org/issue20536
#20535: run_tests.py: Work around issues 20355 and 20361
http://bugs.python.org/issue20535
#20526: python: Modules/gcmodule.c:379: visit_decref: Assertion `((gc)
http://bugs.python.org/issue20526
#20523: global .pdbrc on windows 7 not reachable out of the box
http://bugs.python.org/issue20523
#20521: [PATCH] Cleanup for "dis" module documentation
http://bugs.python.org/issue20521
#20516: Concurrent.futures base concurrency improvement (with patch)
http://bugs.python.org/issue20516
#20513: Python 2.7. Script interruption on logoff from 0 session under
http://bugs.python.org/issue20513
#20511: asyncio: StreamReader should use bytearray for its internal bu
http://bugs.python.org/issue20511
#20508: IndexError from ipaddress._BaseNetwork.__getitem__ has no mess
http://bugs.python.org/issue20508
#20504: cgi.FieldStorage, multipart, missing Content-Length
http://bugs.python.org/issue20504
#20503: super behavioru and abstract base classes (either implementati
http://bugs.python.org/issue20503
#20500: assertion failed when passing an exception object to sys.exit
http://bugs.python.org/issue20500
#20497: Unclear word in socket document.
http://bugs.python.org/issue20497
#20496: function definition tutorial encourages bad practice
http://bugs.python.org/issue20496
#20486: msilib: can't close opened database
http://bugs.python.org/issue20486
Most recent 15 issues waiting for review (15)
=============================================
#20540: Python 3.3/3.4 regression in multiprocessing manager ?
http://bugs.python.org/issue20540
#20537: logging exc_info parameter should accept exception instances
http://bugs.python.org/issue20537
#20535: run_tests.py: Work around issues 20355 and 20361
http://bugs.python.org/issue20535
#20534: Enum tests fail with pickle protocol 4
http://bugs.python.org/issue20534
#20532: Mark all tests which use _testcapi as CPython only
http://bugs.python.org/issue20532
#20530: Change the text signature format (again) to be more robust
http://bugs.python.org/issue20530
#20525: Got compiler warning when compiling readline module
http://bugs.python.org/issue20525
#20523: global .pdbrc on windows 7 not reachable out of the box
http://bugs.python.org/issue20523
#20521: [PATCH] Cleanup for "dis" module documentation
http://bugs.python.org/issue20521
#20520: Readline test in test_codecs is broken
http://bugs.python.org/issue20520
#20519: Replace uuid ctypes usage with an extension module.
http://bugs.python.org/issue20519
#20516: Concurrent.futures base concurrency improvement (with patch)
http://bugs.python.org/issue20516
#20515: Null pointer dereference in tkinter module
http://bugs.python.org/issue20515
#20513: Python 2.7. Script interruption on logoff from 0 session under
http://bugs.python.org/issue20513
#20511: asyncio: StreamReader should use bytearray for its internal bu
http://bugs.python.org/issue20511
Top 10 most discussed issues (10)
=================================
#20480: Add ipaddress property to get name of reverse DNS PTR record
http://bugs.python.org/issue20480 21 msgs
#20485: Enable non-ASCII extension module names
http://bugs.python.org/issue20485 17 msgs
#20481: Clarify type coercion rules in statistics module
http://bugs.python.org/issue20481 14 msgs
#20053: venv and ensurepip are affected by default pip config file
http://bugs.python.org/issue20053 13 msgs
#20185: Derby #17: Convert 49 sites to Argument Clinic across 13 files
http://bugs.python.org/issue20185 12 msgs
#20475: pystone.py in 3.4 still uses time.clock(), even though it's ma
http://bugs.python.org/issue20475 10 msgs
#15216: Support setting the encoding on a text stream after creation
http://bugs.python.org/issue15216 9 msgs
#20505: Remove resolution from selectors and granularity from asyncio
http://bugs.python.org/issue20505 9 msgs
#20520: Readline test in test_codecs is broken
http://bugs.python.org/issue20520 9 msgs
#16296: Patch to fix building on Win32/64 under VS 2010
http://bugs.python.org/issue16296 8 msgs
Issues closed (62)
==================
#2226: Small _abcoll Bugs / Oddities
http://bugs.python.org/issue2226 closed by ncoghlan
#7538: HP-UX 11.11 GCC build fails to build modules
http://bugs.python.org/issue7538 closed by Arfrever
#7717: Compilation fixes for IRIX
http://bugs.python.org/issue7717 closed by berker.peksag
#11117: Implementing Async IO
http://bugs.python.org/issue11117 closed by yselivanov
#12187: subprocess.wait() with a timeout uses polling on POSIX
http://bugs.python.org/issue12187 closed by haypo
#14515: tempfile.TemporaryDirectory documented as returning object but
http://bugs.python.org/issue14515 closed by r.david.murray
#14665: faulthandler prints tracebacks in reverse order
http://bugs.python.org/issue14665 closed by ncoghlan
#14854: faulthandler: fatal error with "SystemError: null argument to
http://bugs.python.org/issue14854 closed by haypo
#17159: Remove explicit type check from inspect.Signature.from_functio
http://bugs.python.org/issue17159 closed by yselivanov
#17162: Py_LIMITED_API needs a PyType_GenericDealloc
http://bugs.python.org/issue17162 closed by loewis
#18801: inspect.classify_class_attrs() misclassifies object.__new__()
http://bugs.python.org/issue18801 closed by yselivanov
#18891: Master patch for content manager addition to email package
http://bugs.python.org/issue18891 closed by r.david.murray
#19186: expat symbols should be namespaced in pyexpat again
http://bugs.python.org/issue19186 closed by python-dev
#19320: Tkinter tests ran with wantobjects is false
http://bugs.python.org/issue19320 closed by serhiy.storchaka
#19334: test_asyncio hanging for 1 hour (non-AIX version)
http://bugs.python.org/issue19334 closed by haypo
#19652: test_asyncio: test_subprocess_send_signal() hangs on buildbot
http://bugs.python.org/issue19652 closed by haypo
#19761: test_tk fails on OS X with multiple test case failures with bo
http://bugs.python.org/issue19761 closed by serhiy.storchaka
#19863: Missing function attributes in 2.7 docs.
http://bugs.python.org/issue19863 closed by mark.dickinson
#19920: TarFile.list() fails on some files
http://bugs.python.org/issue19920 closed by serhiy.storchaka
#20029: asyncio.SubprocessProtocol is missing
http://bugs.python.org/issue20029 closed by haypo
#20102: shutil._make_zipfile possible resource leak
http://bugs.python.org/issue20102 closed by python-dev
#20136: Logging: StreamHandler does not use OS line separator.
http://bugs.python.org/issue20136 closed by vinay.sajip
#20137: Logging: RotatingFileHandler computes string length instead of
http://bugs.python.org/issue20137 closed by vinay.sajip
#20162: Test test_hash_distribution fails on RHEL 6.5 / ppc64
http://bugs.python.org/issue20162 closed by haypo
#20311: epoll.poll(timeout) and PollSelector.select(timeout) must roun
http://bugs.python.org/issue20311 closed by haypo
#20319: concurrent.futures.wait() can block forever even if Futures ha
http://bugs.python.org/issue20319 closed by bquinlan
#20354: tracemalloc causes segfault in "make profile-opt"
http://bugs.python.org/issue20354 closed by haypo
#20363: BytesWarnings triggerred by test suite
http://bugs.python.org/issue20363 closed by serhiy.storchaka
#20368: Tkinter: handle the null character
http://bugs.python.org/issue20368 closed by serhiy.storchaka
#20400: Add create_read_pipe_protocol/create_write_pipe_protocol to as
http://bugs.python.org/issue20400 closed by haypo
#20404: Delayed exception using non-text encodings with TextIOWrapper
http://bugs.python.org/issue20404 closed by python-dev
#20412: Enum and IntEnum classes are not defined in the documentation
http://bugs.python.org/issue20412 closed by python-dev
#20423: io.StringIO newline param has wrong default
http://bugs.python.org/issue20423 closed by pitrou
#20426: Compiling a regex with re.DEBUG should force a recompile
http://bugs.python.org/issue20426 closed by pitrou
#20435: Discrepancy between io.StringIO and _pyio.StringIO with univer
http://bugs.python.org/issue20435 closed by pitrou
#20445: HAVE_BROKEN_NICE detected incorrectly due to configure.ac typo
http://bugs.python.org/issue20445 closed by George.Kouryachy
#20452: test_timeout_rounding() of test_asyncio fails on "x86 Ubuntu S
http://bugs.python.org/issue20452 closed by haypo
#20455: test_asyncio hangs on Windows
http://bugs.python.org/issue20455 closed by haypo
#20456: Argument Clinic rollup patch, 2014/01/31
http://bugs.python.org/issue20456 closed by larry
#20460: Wrong markup in c-api/arg.rst
http://bugs.python.org/issue20460 closed by zach.ware
#20470: "Fatal Python error: Bus error" on the SPARC Solaris 10 buildb
http://bugs.python.org/issue20470 closed by haypo
#20471: test_signature_on_class() of test_inspect fails on "AMD64 Free
http://bugs.python.org/issue20471 closed by yselivanov
#20472: test_write_pty() of test_asyncio fails on "x86 Tiger 3.x" buil
http://bugs.python.org/issue20472 closed by haypo
#20473: inspect.Signature no longer handles builtin classes correctly
http://bugs.python.org/issue20473 closed by yselivanov
#20474: test_socket failures on OS X due to fixed "expected failures"
http://bugs.python.org/issue20474 closed by ned.deily
#20477: Add examples of using the new contentmanager api to the email
http://bugs.python.org/issue20477 closed by r.david.murray
#20488: importlib title and introduction are out of date
http://bugs.python.org/issue20488 closed by brett.cannon
#20489: help() fails for zlib Compress and Decompress objects
http://bugs.python.org/issue20489 closed by serhiy.storchaka
#20492: test_tcl is failing on windows 2.7
http://bugs.python.org/issue20492 closed by serhiy.storchaka
#20494: Backport of http://bugs.python.org/issue11849 to Python 2.7 (M
http://bugs.python.org/issue20494 closed by python-dev
#20498: Update StringIO newline tests
http://bugs.python.org/issue20498 closed by serhiy.storchaka
#20502: Context.create_decimal_from_float() inconsistent precision for
http://bugs.python.org/issue20502 closed by mark.dickinson
#20509: logging.config.fileConfig() docs could link to the config file
http://bugs.python.org/issue20509 closed by python-dev
#20512: Python3.3 segfaults when using big5hkscs encoding
http://bugs.python.org/issue20512 closed by ned.deily
#20514: Errno 10013 - ipython notebook
http://bugs.python.org/issue20514 closed by ned.deily
#20522: pandas version 0.13.0 does not give warning when summing float
http://bugs.python.org/issue20522 closed by mark.dickinson
#20527: multiprocessing.Queue deadlocks after “reader” process dea
http://bugs.python.org/issue20527 closed by sbt
#20528: fileinput module will read whole file into memory when using f
http://bugs.python.org/issue20528 closed by r.david.murray
#20533: bug in float arithmetic?
http://bugs.python.org/issue20533 closed by zach.ware
#20541: os.path.exists() gives wrong answer for Windows special files
http://bugs.python.org/issue20541 closed by ncoghlan
#20542: Assertion failure in test_readline
http://bugs.python.org/issue20542 closed by ncoghlan
#1028088: Cookies without values are silently ignored (by design?)
http://bugs.python.org/issue1028088 closed by berker.peksag
April 19, 2014
Several significant changes in this revision:
- scope narrowed to just Python 2.7 plus permission for commercial
redistributors to use the same strategy in their long term support
releases
- far more explicit that this is about inviting potential corporate
contributors to address the situation for the benefit of the overall
Python ecosystem, not offering to fix it for them for free
- clarified that third party integration testing services would need
to be updated to support testing against multiple Python 2.7 minor
releases
- explicit sections on why I don't think the status quo is
sustainable, why I don't think Python 2.8 would actually solve the
problem, and why I think a PyPI based solution not only wouldn't solve
the problem, but would be rather difficult to get working in the first
place
- be completely explicit that I am *not* speaking on behalf of Red Hat
at this point and have no authority to make commitments on their
behalf. Instead, I'm looking for upstream consensus that 1) this is a
genuine problem that needs to be solved; 2) we're open to corporate
assistance in solving it; and 3) we have a pretty good idea what help
we actually want. If all that happens, *then* I can take up the issue
internally to try to get us some help in maintaining the proposed
solution (hopefully other folks with corporate influence can do the
same, and we may actually get some ongoing assistance with upstream
maintenance out of this, rather than having our downstream
redistributors continue to take us for granted).
Diff: http://hg.python.org/peps/rev/2e82209dda21
Updated web version: http://www.python.org/dev/peps/pep-0466/
Advance warning: while I was able to get this revision turned around
pretty quickly, future revisions are likely to take a fair bit longer.
It was already a rather busy month before I decided to start this
discussion on top of everything else :)
Cheers,
Nick.
PEP: 466
Title: Network Security Enhancement Exception for Python 2.7
Version: $Revision$
Last-Modified: $Date$
Author: Nick Coghlan <ncoghlan(a)gmail.com>,
Status: Draft
Type: Informational
Content-Type: text/x-rst
Created: 23-Mar-2014
Post-History: 23-Mar-2014
Abstract
========
Most CPython tracker issues are classified as errors in behaviour or
proposed enhancements. Most patches to fix behavioural errors are
applied to all active maintenance branches. Enhancement patches are
restricted to the default branch that becomes the next Python version.
This cadence works reasonably well during Python's normal 18-24 month
feature release cycle, which is still applicable to the Python 3 series.
However, the age of the standard library in Python 2 has now reached a point
where it is sufficiently far behind the state of the art in network security
protocols for it to be causing real problems in commercial use cases
where upgrading to Python 3 in the near term may not be practical.
Accordingly, this PEP relaxes the normal restrictions by allowing
enhancements to be applied in Python 2.7 maintenance releases for standard
library components that have implications for the overall security of the
internet. In particular, the exception will apply to:
* the ``ssl`` module
* the ``hashlib`` module
* the ``hmac`` module
* the ``sha`` module (Python 2 only)
* the components of other networking modules that make use of these modules
* the components of the ``random`` and ``os`` modules that are relevant to
cryptographic applications
* the version of OpenSSL bundled with the binary installers
Proposed backports for these modules will still need to undergo normal
backwards compatibility assessments, but new features will be permitted where
appropriate, making it easier to implement secure networked software in
Python, even for software that needs to remain compatible with older feature
releases of Python.
While this PEP does not make any changes to the core development team's
handling of security-fix-only branches that are no longer in active
maintenance, it *does* recommend that commercial redistributors providing
extended support periods for the Python standard library either adopt a
similar approach to ensuring that the secure networking infrastructure
keeps pace with the evolution of the internet, or else disclaim support
for the use of older versions in roles that involving connecting
directly to the public internet.
Exemption Policy
================
Under this policy, the following network security related modules are
granted a blanket exemption to the restriction against adding new features
in maintenance releases, for the purpose of keeping their APIs aligned with
their counterparts in the latest feature release of Python 3:
* the ``ssl`` module
* the ``hashlib`` module
* the ``hmac`` module
* the ``sha`` module (Python 2 only)
This exemption applies to *all* proposals to backport backwards compatible
changes in these modules to Python 2.7 maintenance releases. This choice is
made deliberately to ensure that the "feature or fix?" argument isn't simply
replaced by a "security related or not?" argument. These particular modules
are inherently security related, and all enhancements to them improve
Python's capabilities as a platform for development of secure networked
software.
As part of this policy, permission is also granted to upgrade to newer
feature releases of OpenSSL when preparing the binary installers
for new maintenance releases of CPython.
In addition to the above blanket exemption, a conditional exemption is
granted for these modules that may include some network security related
features:
* the ``os`` module (specifically ``os.urandom``)
* the ``random`` module
* networking related modules that depend on one or more of the network
security related modules listed above
This more limited exemption for these modules requires that the *specific*
enhancement being proposed for backporting needs to be justified as being
network security related. If the enhancement under discussion is designed
to take advantage of a new feature in one of the network security related
modules, then that will be taken as implying that the enhancement is
security related.
Backwards Compatibility Considerations
======================================
This PEP does *not* grant any general exemptions to the usual backwards
compatibility policy for maintenance releases. Instead, by explicitly
encouraging the use of feature based checks and explicitly opting in to
less secure configurations, it is designed to make it easier to provide
more "secure by default" behaviour in future feature releases, while still
limiting the risk of breaking currently working software when upgrading to
a new maintenance release.
In *all* cases where this policy is applied to backport enhancements to
maintenance releases, it MUST be possible to write cross-version compatible
code that operates by "feature detection" (for example, checking for
particular attributes in the module), without needing to explicitly check
the Python version.
It is then up to library and framework code to provide an appropriate warning
and fallback behaviour if a desired feature is found to be missing. While
some especially security sensitive software MAY fail outright if a desired
security feature is unavailable, most software SHOULD instead continue
operating using a slightly degraded security configuration.
Affected APIs SHOULD be designed to allow library and application code to
perform the following actions after detecting the presence of a relevant
network security related feature:
* explicitly opt in to more secure settings (to allow the use of enhanced
security features in older maintenance releases of Python)
* explicitly opt in to less secure settings (to allow the use of newer Python
feature releases in lower security environments)
* determine the default setting for the feature (this MAY require explicit
Python version checks to determine the Python feature release, but MUST
NOT require checking for a specific maintenance release)
Security related changes to other modules (such as data format processing
libraries) will continue to be made available as backports and new modules
on the Python Package Index, as independent distribution remains the
preferred approach to handling software that needs to evolve faster than
the standard library. Refer to the `Motivation and Rationale`_ section for
a review of the characteristics that make the secure networking
infrastructure worthy of special consideration.
Other Considerations
====================
Maintainability
---------------
This policy does NOT represent a commitment by volunteer contributors to
actually backport network security related changes from the Python 3 series
to the Python 2 series. Rather, it is intended to send a clear signal to
potential corporate contributors that the core development team are willing
to review and merge corporate contributions that put this policy into
effect.
Backporting security related fixes and enhancements to earlier versions is
a common service for commercial redistributors to offer to their customers.
This policy represents an explicit invitation to contribute some of those
changes back to the upstream community in cases where they are likely to
have a broad impact that helps improve the security of the internet as a
whole.
Documentation
-------------
All modules that take advantage of this policy to backport network
security related enhancements to earlier Python versions MUST include
a "Security Considerations" section in their documentation.
In addition to any other module specific contents, this section MUST
enumerate key security enhancements and fixes (with CVE identifiers where
applicable), along with the feature and maintenance releases that first
included them.
Security releases
-----------------
This PEP does not propose any changes to the handling of security
releases - those will continue to be source only releases that
include only critical security fixes.
However, the recommendations for library and application developers are
deliberately designed to accommodate commercial redistributors applying
this policy to any Python release series that is either in security
fix only mode, or has been declared "end of life" by the core development
team.
Whether or not redistributors choose to exercise that option will be up
to the redistributor.
Integration testing
-------------------
Third party integration testing services would likely need to start
offering users a choice of multiple Python 2.7.x versions to test against,
to ensure that the application is correctly degrading gracefully if it
attempts to use newer networking features on maintenance releases that
are too old to provide them.
Evolution of this Policy
========================
The key requirement for a module to be considered for inclusion in this
policy (whether under a blanket or conditional exemption) is that it must
have security implications *beyond* the specific application that is written
in Python and the system that application is running on. Thus the focus on
network security protocols and related cryptographic infrastructure - Python
is a popular choice for the development of web services and clients, and
thus the capabilities of widely used Python versions have implications for
the security design of other services that may be using newer versions of
Python or other development languages.
The intent behind this requirement is to minimise any impact that the
introduction of this policy may have on the stability and compatibility of
maintenance releases. It would be thoroughly counterproductive if end
users became as cautious about updating to new Python maintenance releases
as they are about updating to new feature releases.
Motivation and Rationale
========================
This PEP can be seen as a more targeted version of the "faster standard
library release cycle" proposals discussed in PEP 407 and PEP 413,
focusing specifically on those areas which have implications beyond the
Python community.
Background
----------
The creation of this PEP was prompted primarily by the aging SSL support in
the Python 2 series. As of March 2014, the Python 2.7 SSL module is
approaching four years of age, and the SSL support in the still popular
Python 2.6 release had its feature set locked six years ago.
These are simply too old to provide a foundation that can be recommended
in good conscience for secure networking software that operates over the
public internet, especially in an era where it is becoming quite clearly
evident that advanced persistent security threats are even more widespread
and more indiscriminate in their targeting than had previously been
understood. While they represented reasonable security infrastructure in
their time, the state of the art has moved on, and we need to investigate
mechanisms for effectively providing more up to date network security
infrastructure for users that, for whatever reason, are not currently in
a position to migrate to Python 3.
While the use of the system OpenSSL installation addresses many of these
concerns on Linux platforms, it doesn't address all of them, and in the
case of the binary installers for Windows and Mac OS X that are published
on python.org, the version of OpenSSL used is entirely within the control
of the Python core development team, and currently limited to OpenSSL
maintenance releases for the version initially shipped with the corresponding
Python feature release.
With increased popularity comes increased responsibility, and this policy
aims to acknowledge the fact that Python's popularity and adoption has now
reached a level where some of our design and policy decisions have
significant implications beyond the Python development community.
As one example, the Python 2 ``ssl`` module does not support the Server
Name Identification standard. While it is possible to obtain SNI support
by using the third party ``requests`` client library, actually doing so
currently requires using not only ``requests`` and its embedded dependencies,
but also half a dozen or more additional libraries. The lack of support
in the Python 2 series thus serves as an impediment to making effective
use of SNI on servers, as Python 2 clients will frequently fail to handle
it correctly.
Another more critical example is the lack of SSL hostname matching in the
Python 2 standard library - it is currently necessary to rely on a third
party library, such as ``requests`` or ``backports.ssl_match_hostname`` to
obtain that functionality in Python 2.
The Python 2 series also remains more vulnerable to remote timing attacks
on security sensitive comparisons than the Python 3 series, as it lacks a
standard library equivalent to the timing attack resistant
``hmac.compare_digest()`` function. While appropriate secure comparison
functions can be implemented in third party extensions, may users don't
even consider the problem and use ordinary equality comparisons instead
- while a standard library solution doesn't automatically fix that problem,
it *does* make the barrier to resolution much lower once the problem is
pointed out.
My position on the ongoing transition from Python 2 to Python 3 has long
been that Python 2 remains a supported platform for the core development
team, and that commercial support will remain available well after upstream
maintenance ends. However, in the absence of this network security
enhancement policy, that position is difficult to justify when it comes to
software that operates over the public internet. Just as many developers
consider it too difficult to develop truly secure modern networked software
in C/C++ (largely due to the challenges associated with manual
memory management), I anticipate that in the not too distant future, it
will be considered too difficult to develop truly secure modern networked
software using the Python 2 series (some developers would argue that we
have already reached that point).
Alternative: advise developers of networked software to migrate to Python 3
---------------------------------------------------------------------------
This alternative represents the status quo. Unfortunately, it has proven
to be unworkable in practice, as the backwards compatibility implications
mean that this is a non-trivial migration process for large applications
and integration projects.
Now that we're fully aware of the impact the limitations in Python 2 may be
having on the evolution of internet security standards, I no longer believe
that it is reasonable to expect platform and application developers to
resolve all of the latent defects in an application's Unicode correctness
solely in order to gain access to the network security enhancements
available in Python 3.
While (as far as I am aware) Ubuntu has successfully switched to Python 3.4
as its main Python interpreter for its 14.04 LTS release, Fedora still
has a lot of work to do to migrate, and it will take a non-trivial amount
of time to migrate the relevant infrastructure components. While Red Hat
are also actively working to make it easier for users to use more recent
versions of Python on our stable platforms, it's going to take time for
those efforts to start having an impact on end users' choice of version,
and those changes won't affect the core tools regardless.
The OpenStack migration to Python 3 is also still in its infancy, and even
though that's a project with an extensive and relatively robust automated
test suite, it's large enough that it is going to take quite some time
to migrate.
And that's just three of the highest profile open source projects that
make heavy use of Python. Given the likely existence of large amounts of
legacy code that lacks the kind of automated regression test suite needed
to help support a migration from Python 2 to Python 3. The key point of
this PEP is that those situations affect more people than just the
developers and users of the affected application: their existence becomes
something that developers of secure networked services need to take into
account as part of their security design.
As Terry Reedy noted, if we try to persist with the status quo, the likely
outcome is that commercial redistributors will attempt to do something
like this on behalf of their customers *anyway*, but in a potentially
inconsistent and ad hoc manner. By drawing the scope definition process
into the upstream project we are in a better position to influence the
approach taken to address the situation and to help ensure some consistency
across redistributors.
The problem is real, so *something* needs to change, and this PEP describes
my currently preferred approach to addressing the situation.
Alternative: create and release Python 2.8
------------------------------------------
With sufficient corporate support, it likely *would* be possible to create
and release Python 2.8 (it's highly unlikely such a project would garner
enough interest to be achievable with only volunteers). However, this
wouldn't actually solve the problem, as the aim is to provide a *relatively
low impact* way to incorporate enhanced security features into integrated
products and deployments that make use of Python 2. Upgrading to a new
Python feature release would mean both more work for the core development
team, as well as a more disruptive update that most potential end users
would likely just skip entirely.
Attempting to create a Python 2.8 release would also bring in suggestions
to backport many additional features from Python 3 (such as ``tracemalloc``
and the improved coroutine support).
This is not a recommended approach, as it would involve substantial
additional work for a result that is actually less effective as a solution
to the original problem (the widespread use of the aging network security
infrastructure in Python 2).
Alternative: distribute the security enhancements via PyPI
----------------------------------------------------------
While it initially appears to be an attractive and easier to manage
approach, there are actually several significant problems with this
idea.
Firstly, this PEP encompasses a non-trivial portion of the standard library.
It's not just the underlying SSL support, but also the libraries for other
network protocols like HTTP, FTP, IMAP, and POP3 that integrate with the
SSL infrastructure to provide secure links, and that's just the protocols
in the standard library. Even if an API compatible ``ssl2`` module was
made available, it would need to be imported and injected
into ``sys.modules`` as ``ssl`` before importing any other module that
needed it.
Secondly, this is complex, low level, cross-platform code that integrates
with the underlying operating system across a variety of POSIX platforms
(including Mac OS X) and Windows. The CPython BuildBot fleet is already set
up to handle continuous integration in that context, but most of the
freely available continuous integration services just offer Linux, and
perhaps paid access to Windows. Those services work reasonably well for
software that largely runs on the abstraction layers offered by Python and
other dynamic languages, but won't suffice for the kind of code involved
here.
The OpenSSL dependency for the network security support also qualifies as
the kind of "complex binary dependency" that isn't yet handled well by the
``pip`` based software distribution ecosystem. Relying on a binary
dependency also creates potential compatibility problems for ``pip`` when
running on other interpreters like ``PyPy``.
Another practical problem with the idea is the fact that ``pip`` itself
relies on the ``ssl`` support in the standard library (with some additional
support from a bundled copy of ``requests``, which in turn bundles
``backport.ssl_match_hostname``), and hence would require any replacement
module to also be bundled within ``pip``. This wouldn't pose any
insurmountable difficulties (it's just another dependency to vendor), but
it *would* mean yet another copy of OpenSSL to keep up to
date.
This approach also has the same flaw as all other "improve security by
renaming things" approaches: they completely miss the users who most need
help, and raise significant barriers against being able to encourage users
to do the right thing when their infrastructure supports it (since
"use this other module" is a much higher impact change than "turn on this
higher security setting"). Deprecating the aging SSL infrastructure in the
standard library in favour of an external module would be even more user
hostile than taking the risk of trying to upgrade it in place.
Last, but certainly not least, this approach suffers from the same problem
as the idea of doing a Python 2.8 release: likely not solving the actual
problem. Commercial redistributors of Python are set up to redistribute
*Python*, and a pre-existing set of additional packages. Getting new
packages added to the pre-existing set *can* be done, but means going
around to each and every redistributor and asking them to update their
repackaging process accordingly. By contrast, the approach described in
this PEP would require redistributors to *opt out* of the security
enhancements, which most of them are unlikely to do.
Open Questions
==============
* What are the risks associated with allowing OpenSSL to be updated to
new feature versions in the Windows and Mac OS X binary installers for
maintenance releases? Currently we just upgrade to the appropriate
OpenSSL maintenance releases, rather than switching to the latest
feature release. In particular, is it possible Windows C extensions may
be linking against the Python provided OpenSSL module?
* Are there any other security relevant modules that should be covered
by either a blanket or conditional exemption?
Disclosure of Interest
======================
The author of this PEP currently works for Red Hat on test automation tools.
If this proposal is accepted, I will be strongly encouraging Red Hat to take
advantage of the resulting opportunity to help improve the overall security
of the Python ecosystem. However, I do not speak for Red Hat in this matter,
and cannot make any commitments on Red Hat's behalf.
Acknowledgements
================
Thanks to Christian Heimes for his recent efforts on greatly improving
Python's SSL support in the Python 3 series, and a variety of members of
the Python community for helping me to better understand the implications
of the default settings we provide in our SSL modules, and the impact that
tolerating the use of SSL infrastructure that was defined in 2010
(Python 2.7) or even 2008 (Python 2.6) potentially has for the security
of the web as a whole.
Christian and Donald Stufft also provided valuable feedback on a preliminary
draft of this proposal.
Thanks also to participants in the python-dev mailing list thread [1]_
References
==========
.. [1] https://mail.python.org/pipermail/python-dev/2014-March/133334.html
Copyright
=========
This document has been placed in the public domain.
--
Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia
April 12, 2014
Okay, I included that last round of comments (from late February).
Barring typos, this should be the final version.
Final comments?
-----------------------------------------------------------------------------
PEP: 461
Title: Adding % formatting to bytes and bytearray
Version: $Revision$
Last-Modified: $Date$
Author: Ethan Furman <ethan(a)stoneleaf.us>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 2014-01-13
Python-Version: 3.5
Post-History: 2014-01-14, 2014-01-15, 2014-01-17, 2014-02-22, 2014-03-25
Resolution:
Abstract
========
This PEP proposes adding % formatting operations similar to Python 2's ``str``
type to ``bytes`` and ``bytearray`` [1]_ [2]_.
Rationale
=========
While interpolation is usually thought of as a string operation, there are
cases where interpolation on ``bytes`` or ``bytearrays`` make sense, and the
work needed to make up for this missing functionality detracts from the overall
readability of the code.
Motivation
==========
With Python 3 and the split between ``str`` and ``bytes``, one small but
important area of programming became slightly more difficult, and much more
painful -- wire format protocols [3]_.
This area of programming is characterized by a mixture of binary data and
ASCII compatible segments of text (aka ASCII-encoded text). Bringing back a
restricted %-interpolation for ``bytes`` and ``bytearray`` will aid both in
writing new wire format code, and in porting Python 2 wire format code.
Common use-cases include ``dbf`` and ``pdf`` file formats, ``email``
formats, and ``FTP`` and ``HTTP`` communications, among many others.
Proposed semantics for ``bytes`` and ``bytearray`` formatting
=============================================================
%-interpolation
---------------
All the numeric formatting codes (such as ``%x``, ``%o``, ``%e``, ``%f``,
``%g``, etc.) will be supported, and will work as they do for str, including
the padding, justification and other related modifiers. The only difference
will be that the results from these codes will be ASCII-encoded text, not
unicode. In other words, for any numeric formatting code `%x`::
b"%x" % val
is equivalent to
("%x" % val).encode("ascii")
Examples::
>>> b'%4x' % 10
b' a'
>>> b'%#4x' % 10
' 0xa'
>>> b'%04X' % 10
'000A'
``%c`` will insert a single byte, either from an ``int`` in range(256), or from
a ``bytes`` argument of length 1, not from a ``str``.
Examples::
>>> b'%c' % 48
b'0'
>>> b'%c' % b'a'
b'a'
``%s`` is included for two reasons: 1) `b` is already a format code for
``format`` numerics (binary), and 2) it will make 2/3 code easier as Python 2.x
code uses ``%s``; however, it is restricted in what it will accept::
- input type supports ``Py_buffer`` [6]_?
use it to collect the necessary bytes
- input type is something else?
use its ``__bytes__`` method [7]_ ; if there isn't one, raise a ``TypeError``
In particular, ``%s`` will not accept numbers (use a numeric format code for
that), nor ``str`` (encode it to ``bytes``).
Examples::
>>> b'%s' % b'abc'
b'abc'
>>> b'%s' % 'some string'.encode('utf8')
b'some string'
>>> b'%s' % 3.14
Traceback (most recent call last):
...
TypeError: b'%s' does not accept numbers, use a numeric code instead
>>> b'%s' % 'hello world!'
Traceback (most recent call last):
...
TypeError: b'%s' does not accept 'str', it must be encoded to `bytes`
``%a`` will call ``ascii()`` on the interpolated value. This is intended
as a debugging aid, rather than something that should be used in production.
Non-ASCII values will be encoded to either ``\xnn`` or ``\unnnn``
representation. Use cases include developing a new protocol and writing
landmarks into the stream; debugging data going into an existing protocol
to see if the problem is the protocol itself or bad data; a fall-back for a
serialization format; or even a rudimentary serialization format when
defining ``__bytes__`` would not be appropriate [8].
.. note::
If a ``str`` is passed into ``%a``, it will be surrounded by quotes.
Unsupported codes
-----------------
``%r`` (which calls ``__repr__`` and returns a ``str``) is not supported.
Proposed variations
===================
It was suggested to let ``%s`` accept numbers, but since numbers have their own
format codes this idea was discarded.
It has been suggested to use ``%b`` for bytes as well as ``%s``. This was
rejected as not adding any value either in clarity or simplicity.
It has been proposed to automatically use ``.encode('ascii','strict')`` for
``str`` arguments to ``%s``.
- Rejected as this would lead to intermittent failures. Better to have the
operation always fail so the trouble-spot can be correctly fixed.
It has been proposed to have ``%s`` return the ascii-encoded repr when the
value is a ``str`` (b'%s' % 'abc' --> b"'abc'").
- Rejected as this would lead to hard to debug failures far from the problem
site. Better to have the operation always fail so the trouble-spot can be
easily fixed.
Originally this PEP also proposed adding format-style formatting, but it was
decided that format and its related machinery were all strictly text (aka
``str``) based, and it was dropped.
Various new special methods were proposed, such as ``__ascii__``,
``__format_bytes__``, etc.; such methods are not needed at this time, but can
be visited again later if real-world use shows deficiencies with this solution.
Objections
==========
The objections raised against this PEP were mainly variations on two themes::
- the ``bytes`` and ``bytearray`` types are for pure binary data, with no
assumptions about encodings
- offering %-interpolation that assumes an ASCII encoding will be an
attractive nuisance and lead us back to the problems of the Python 2
``str``/``unicode`` text model
As was seen during the discussion, ``bytes`` and ``bytearray`` are also used
for mixed binary data and ASCII-compatible segments: file formats such as
``dbf`` and ``pdf``, network protocols such as ``ftp`` and ``email``, etc.
``bytes`` and ``bytearray`` already have several methods which assume an ASCII
compatible encoding. ``upper()``, ``isalpha()``, and ``expandtabs()`` to name
just a few. %-interpolation, with its very restricted mini-language, will not
be any more of a nuisance than the already existing methods.
Some have objected to allowing the full range of numeric formatting codes with
the claim that decimal alone would be sufficient. However, at least two
formats (dbf and pdf) make use of non-decimal numbers.
Footnotes
=========
.. [1] http://docs.python.org/2/library/stdtypes.html#string-formatting
.. [2] neither string.Template, format, nor str.format are under consideration
.. [3] https://mail.python.org/pipermail/python-dev/2014-January/131518.html
.. [4] to use a str object in a bytes interpolation, encode it first
.. [5] %c is not an exception as neither of its possible arguments are str
.. [6] http://docs.python.org/3/c-api/buffer.html
examples: ``memoryview``, ``array.array``, ``bytearray``, ``bytes``
.. [7] http://docs.python.org/3/reference/datamodel.html#object.__bytes__
.. [8] https://mail.python.org/pipermail/python-dev/2014-February/132750.html
Copyright
=========
This document has been placed in the public domain.
..
Local Variables:
mode: indented-text
indent-tabs-mode: nil
sentence-end-double-space: t
fill-column: 70
coding: utf-8
End:
Hi,
The libpython library has been added to the ABI tracker:
http://upstream-tracker.org/versions/python.html
The page lists library versions and changes in API/ABI.
--
Andrey Ponomarenko, NTC IT ROSA.
There's a furious discussion going on at the python-list mailing list,
about negative timedelta strings:
py> str(timedelta(0, -1))
'-1 day, 23:59:59'
This is documented. It's even documented as being "somewhat unusual". I
found a tracker item for it, back in July 2010:
http://bugs.python.org/issue9430
and that refers back to a discussion on #python-dev, but (as far as I
know) that's not archived anywhere. I had a look on the python-dev
mailing list around that time, and couldn't find anything.
Does anyone remember the rationale for this behaviour? Is it open to
debate or is it now cast in stone? (Personally, I'm not looking to
reopen the debate, and if I did, it would be on python-ideas.) Any links
to the earlier discussion would be helpful.
Thanks,
--
Steve
|
| You are visitor 01402503 — GeezerVison does not store the pages you look up. Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required. |
