GeezerVison is showing github.com — ndbm can't iterate through values on OS X · Issue #74573 · python/cpython
ndbm can't iterate through values on OS X #74573Description
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2017-05-17.16:03:55.042>
labels = ['OS-mac', 'type-bug', 'library']
title = "ndbm can't iterate through values on OS X"
updated_at = <Date 2020-11-27.11:32:30.539>
user = 'https://bugs.python.org/ForestGregg' bugs.python.org fields:
activity = <Date 2020-11-27.11:32:30.539>
actor = 'ronaldoussoren'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)', 'macOS']
creation = <Date 2017-05-17.16:03:55.042>
creator = 'Forest Gregg'
dependencies = []
files = ['46872', '49627']
hgrepos = []
issue_num = 30388
keywords = []
message_count = 11.0
messages = ['293858', '293934', '293974', '293975', '294040', '295189', '295215', '295282', '295512', '380963', '381944']
nosy_count = 5.0
nosy_names = ['terry.reedy', 'ronaldoussoren', 'ned.deily', 'emilyemorehouse', 'Forest Gregg']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue30388'
versions = ['Python 3.5'] Linked PRs
ActivityForestGregg commented on May 17, 2017 MannequinAuthor More actions On Mac OS 10.12.4, a large shelve, backed by ndbm, can be created. But when I attempt to iterate through the values of the shelve it raises this exception:
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/dedupe/api.py", line 281, in _blockData
for block in viewvalues(blocks):
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/collections_abc.py", line 693, in __iter_
for key in self._mapping:
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/shelve.py", line 95, in __iter__
for k in self.dict.keys():
SystemError: Negative size passed to PyBytes_FromStringAndSize
I've confirmed that this works with Python 3.6.1. All the Python versions were installed from homebrew.
I cannot reproduce on linux on windows machines. I've attached a zip file of the ndbm files that python created. ForestGregg commented on May 18, 2017 MannequinAuthor More actions The ndbm db's two files (in the attachment) have the following size
tmp___otctx 0 bytes
tmp___otctx.db 12857344 bytes terryjreedy commented on May 19, 2017 Do you mean that on OSX, you code works with 3.6.1 and fails with 3.5.3? ForestGregg commented on May 19, 2017 MannequinAuthor More actions Very sorry for the ambiguity.
The bug appears for both python 3.5 and python 3.6.1 on OS X. I have not tried other versions of python on OS X. ned-deily commented on May 20, 2017 Can you provide a complete example of how to reproduce the behavior you are seeing? Ideally, it would include producing the shelve object. Also, please provide the output of this command:
otool -L $(python3.6 -c 'import _dbm;print(dbm.__file_)') ForestGregg commented on Jun 5, 2017 MannequinAuthor More actions ned-deily commented on Jun 5, 2017 It would be helpful if you could provide the output of this command:
otool -L $(python3.6 -c 'import _dbm;print(dbm.__file_)')
if necessary, substituting whatever command name(s) you are using to invoke python when failing. ForestGregg commented on Jun 6, 2017 MannequinAuthor More actions From one user who had problems under both 3.5 and 3.6
otool -L $(python3.5 -c 'import _dbm;print(dbm.__file_)')
/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/lib-dynload/_dbm.cpython-35m-darwin.so:
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0,
current version 125.2.0)
otool -L $(python3.6 -c 'import _dbm;print(dbm.__file_)')
/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/lib-dynload/_
dbm.cpython-36m-darwin.so:
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version
125.2.0) ForestGregg commented on Jun 9, 2017 MannequinAuthor More actions A different user:
otool -L $(python3.6 -c 'import _dbm;print(_dbm.file)')
/usr/local/var/pyenv/versions/3.6.1/lib/python3.6/lib-dynload/_dbm.cpython-36m-darwin.so:
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1238.50.2) ronaldoussoren commented on Nov 14, 2020 ronaldoussoren commented on Nov 27, 2020 I've attached a C program that demonstrates the problem, and have filed an issue with Apple about this: FB8919203
It is not clear to me what we can do about this, other than not building the ndbm binding on macOS. The manpages for the ndbm APIs do not mention limitations on item sizes, and the dbopen manpage mention that there a no limitations on item sizes). ronaldoussoren commented on Dec 21, 2023 The best we can do is document the issue. added a commit that references this issue on Dec 23, 2023gh-74573: document that ndbm can silently corrupt databases on macOS (#… ronaldoussoren commented on Dec 23, 2023 The just merged PRs document the problem in main, 3.12 and 3.11. As we cannot fix Apple's code that's the best we can do.
MetadataMetadataStatusShow more project fields
DevelopmentNo branches or pull requestsIssue actions
|