GeezerVison is showing github.com — C extension naming doesn't take bitness into account · Issue #67169 · python/cpython
C extension naming doesn't take bitness into account #67169Description
| BPO |
22980 |
| Nosy |
@malemburg, @warsaw, @doko42, @ncoghlan, @pitrou, @larryhastings, @tjguk, @ned-deily, @ericsnowcurrently, @berkerpeksag, @zware, @1st1, @zooba, @dstufft |
| Files |
22980_windows.patch22980_2.patch: Patch with version in tagabi_bitness.patchma.diff: alternate patchma2.difflarry.whatsnew35.ext.module.suffix.diff.1.txt |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2015-09-12.16:13:19.053>
created_at = <Date 2014-12-02.14:35:25.510>
labels = ['deferred-blocker', 'type-feature']
title = "C extension naming doesn't take bitness into account"
updated_at = <Date 2017-01-14.07:25:16.112>
user = 'https://github.com/pitrou' bugs.python.org fields:
activity = <Date 2017-01-14.07:25:16.112>
actor = 'python-dev'
assignee = 'none'
closed = True
closed_date = <Date 2015-09-12.16:13:19.053>
closer = 'larry'
components = []
creation = <Date 2014-12-02.14:35:25.510>
creator = 'pitrou'
dependencies = []
files = ['37379', '37384', '38155', '38964', '39027', '40420']
hgrepos = []
issue_num = 22980
keywords = ['patch']
message_count = 96.0
messages = ['232000', '232001', '232002', '232003', '232008', '232014', '232016', '232026', '232030', '232031', '232033', '232034', '232036', '232037', '232038', '232039', '232040', '232041', '232042', '232043', '232044', '232045', '232047', '232052', '232063', '232065', '232251', '232256', '232266', '232268', '232270', '232272', '232277', '232297', '232710', '232711', '232729', '232740', '232759', '232760', '232764', '232765', '232807', '233822', '236109', '236111', '237558', '237559', '240516', '240548', '240746', '240748', '240750', '240761', '240762', '240769', '240775', '241065', '241127', '241139', '241141', '241159', '241167', '241183', '241211', '241223', '241228', '241230', '241231', '241236', '241242', '241248', '241249', '241250', '241251', '241252', '241253', '241254', '241283', '241285', '241287', '241291', '241512', '241513', '246247', '249725', '250045', '250299', '250302', '250308', '250341', '250424', '250425', '250431', '250524', '285464']
nosy_count = 16.0
nosy_names = ['lemburg', 'barry', 'doko', 'ncoghlan', 'pitrou', 'larry', 'tim.golden', 'ned.deily', 'Arfrever', 'python-dev', 'eric.snow', 'berker.peksag', 'zach.ware', 'yselivanov', 'steve.dower', 'dstufft']
pr_nums = []
priority = 'deferred blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue22980'
versions = ['Python 3.5'] Activitypitrou commented on Dec 2, 2014 Currently, C extensions are named something like "_helperlib.cpython-34dm.so". This doesn't take into account the bitness of the interpreter (32- vs. 64-bit), which makes it awkward to use the same working copy with two different interpreters (you have to rebuild everything each time you switch bitnesses).
Worse, under Windows it seems ABI tags aren't even used, giving generic names such as "_helperlib.pyd". Why is that? vstinner commented on Dec 2, 2014 pitrou commented on Dec 2, 2014 PEP-3149 says """It is not currently clear that the facilities in this PEP are even useful for Windows""". Well, it seems I have found a use for it :-) pitrou commented on Dec 2, 2014 Ideally, we would use distutils.util.get_platform(). However, there are two cases where it relies on other modules:
- the re module under CygWin
- the sysconfig and _osx_support under OS X
Of course, ideally we should be able to hardcode this into the compiled CPython executable... pitrou commented on Dec 2, 2014 As a side-note, it is interesting to note that Python currently wrongly identifies 32-bit builds under 64-bit Linux:
Python 3.5.0a0 (default:64a54f0c87d7, Nov 2 2014, 17:18:13)
[GCC 4.9.1] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys, os, sysconfig
>>> sys.maxsize
2147483647
>>> os.uname()
posix.uname_result(sysname='Linux', nodename='fsol', release='3.16.0-25-generic', version='#33-Ubuntu SMP Tue Nov 4 12:06:54 UTC 2014', machine='x86_64')
>>> sysconfig.get_platform()
'linux-x86_64' AFAIU, sysconfig.get_platform() (or its sibling distutils.util.get_platform()) is used for the naming of binary distributions... pitrou commented on Dec 2, 2014 The MULTIARCH variable can help at least under Linux:
>>> import sysconfig
>>> sysconfig.get_platform()
'linux-x86_64'
>>> sysconfig.get_config_var('MULTIARCH')
'i386-linux-gnu'vstinner commented on Dec 2, 2014 There is also platform.architecture(). I don't like its implementation, it relies on the external file program :-( zooba commented on Dec 2, 2014 I'm very much in favor of adding this for .pyds on Windows.
I assume the hard part will be getting the details for Linux (doesn't bitness have to be compiled in there? For Windows it can be determined at compile-time...), but preferring an extension with the ABI tag and falling back on one without seems easy enough.
(Would/could this also work for .py files? So a 2.7/3.x or Jython/CPython/IronPython package could include tags in pure-Python code files?) malemburg commented on Dec 2, 2014 Note that there's a difference between the platform's architecture (which is what get_platform() returns) and the pointer size of the currently running Python executable.
On 64-bit Linux, it's rather rare to have an application built as 32-bit executable. On 64-bit Windows, it's rather common to have 32-bit applications running.
The best way to determine 32-bit vs. 64-bit is by using the struct module:
# Determine bitsize used by Python (not necessarily the same as
# the one used by the platform)
import struct
bits = struct.calcsize('P') * 8This should be portable across all platforms and will always refer to the pointer size of the currently running Python executable. pitrou commented on Dec 2, 2014
Note that there's a difference between the platform's architecture
Yes, that's pointed out above. pitrou commented on Dec 2, 2014 Nothing new should be necessary for pyc files under Windows:
Python 3.4.2 |Continuum Analytics, Inc.| (default, Oct 22 2014, 11:51:45) [MSC v.1600 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.implementation.cache_tag
'cpython-34' The problem is with C extensions:
>>> import _imp
>>> _imp.extension_suffixes()
['.pyd'] Compare with Linux:
>>> import _imp
>>> _imp.extension_suffixes()
['.cpython-35dm.so', '.abi3.so', '.so'] pitrou commented on Dec 2, 2014
I assume the hard part will be getting the details for Linux (doesn't bitness
have to be compiled in there? For Windows it can be determined at compile-
time...), but preferring an extension with the ABI tag and falling back on
one without seems easy enough.
Sticking to bitness should be easy (although I wonder if it would be desirable for platforms with fat binaries - Ned?). If we can go the extra mile and include platform identification all the better, of course. zooba commented on Dec 2, 2014 I was more interested in source file resolution than bytecode caching. If Python 3.5 would prefer "spam.cpython-35.py" or "spam.cpython-3.py" over "spam.py" and Python 2 preferred "spam.py", then I can more easily separate the code that won't parse in the alternative.
Happy to be told it's unrelated and I should raise it separately, but from my POV resolving .pyd filenames looks very similar to resolving .py files. malemburg commented on Dec 2, 2014 On 02.12.2014 19:02, Antoine Pitrou wrote:
Sticking to bitness should be easy (although I wonder if it would be desirable for platforms with fat binaries - Ned?). If we can go the extra mile and include platform identification all the better, of course.
I hear the "can of worms" alarm ringing :-)
Seriously, I think that putting platform infos into the file name
is bound to cause more trouble than it tries to solve. Fat builds
leave the decision to the linker, which is a good method and avoids
the file name clashes.
I think we should only focus on platforms where fat builds are
uncommon, while at the same time you do have to support multiple
architectures, like e.g. Windows:
http://en.wikipedia.org/wiki/Fat_binary
Note that on Linux, 32-bit and 64-bit versions are typically placed
into different directory trees:
http://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard
so I'm not sure whether it's a real problem on Linux. ned-deily commented on Jul 4, 2015 I think we should add something to the 3.5 "What's New" document about these changes and which platforms are affected. Otherwise is there anything left to do before closing? larryhastings commented on Sep 4, 2015 larryhastings commented on Sep 7, 2015 I'm leaving this open just because we're apparently waiting on some "What's New" docs. larryhastings commented on Sep 9, 2015 Here's an attempt at a What's New section for this change. I expect it's wrong! Maybe someone can fix it. Maybe it's actually better than not having one at all.
Can we maybe get a round or two of edits on this and get something in for 3.5 final? berkerpeksag commented on Sep 9, 2015 Adding Yury since he and Elvis are working on Doc/whatsnew/3.5.rst and they might want to take a look at the latest patch. zooba commented on Sep 9, 2015 Only thing I'd add is that the extra tag is optional (on Windows at least), and Python will happily import extensions without it. But extensions with a mismatched tag won't be loaded. doko42 commented on Sep 9, 2015 thanks for the draft!
I'm not sure how to describe this properly. The extension names are derived from https://wiki.debian.org/Multiarch/Tuples
and this again is derived from the GNU triplets/quadruplets.
there is no "cpu" and "os" part, depending on the architecture some ABI parts are either encoded in the "cpu" part or the "os" part.
So what about just enumerating the most common cases (i386-linux-gnu, x86_64-linux-gnu, arm-linux-gnueabi (still used for the old Raspberry Pi), arm-linux-gnueabihf), and then point to the "spec"? The above examples have some irregular cases, most other cases just follow the triplets.
I wouldn't mention x86_64-linux-gnux32 explicitly. Until now there are only unreleased or experimental distros.
Not sure if it is worth mentioning that this would allow distributing "fat" wheels. python-dev commented on Sep 10, 2015 1st1 commented on Sep 10, 2015 Larry, Matthias, Steve, Berker - I've mentioned this issue in the whatsnew (applied Larry's patch with some modifications to address comments from Steve and Matthias). Please review. zooba commented on Sep 11, 2015 There's no dot before the debug marker on Windows. Otherwise, looks good to me. Thanks for writing this up. larryhastings commented on Sep 12, 2015 It's fixed! So it's finally closed. python-dev commented on Jan 14, 2017
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|