GeezerVison is showing github.com — Difference in behavior between pure Python pickle and C extension _pickle when unpickling instance of instance of metaclass · Issue #105250 · python/cpython
Difference in behavior between pure Python pickle and C extension _pickle when unpickling instance of instance of metaclass #105250DescriptionBug report
When pickling and unpickling an instance of an instance of a metaclass, _pickle reaches into the class object directly and calls tp_new from the C side, whereas pickle uses cls.__new__, which triggers a custom __getattribute__ if defined. Whether this is a bug or just an intentional optimization I'm not sure but at the very least one there is an observable difference in behavior in that unpickling from a distribution where _pickle is available does not call __getattribute__ but unpickling from a distribution where _pickle is not available does call __getattribute__.
MWE
(Works on all available CPythons on Compiler Explorer, 3.5 - 3.11)
https://godbolt.org/z/x76bs8dzv
To simulate different distributions having _pickle available or not, we can use a meta path entry to cause it to fail to import, causing pickle to fall back to the pure Python version.
import importlib.abc
import sys
class HideModuleFinder(importlib.abc.MetaPathFinder):
def __init__(self, hidden):
self.hidden = set(hidden)
def find_spec(self, fullname, path, target=None):
if fullname in self.hidden:
raise ImportError("Module is hidden")
return None # let next finder try
def install(hidden):
sys.meta_path.insert(0, HideModuleFinder(hidden))
hide_pickle = False # change me for testing different behavior
if hide_pickle:
install({"_pickle"})
import pickle # must be done after meta path hook
class Meta(type):
def __getattribute__(self, item):
print("__getattribute__ called with", item)
return type.__getattribute__(self, item)
MyClass = Meta("MyClass", (), {})
obj = MyClass()
print("PICKLING")
obj_str = pickle.dumps(obj)
print("UNPICKLING")
new_obj = pickle.loads(obj_str)Output with hide_pickle = True
PICKLING
__getattribute__ called with __reduce__
__getattribute__ called with __dict__
__getattribute__ called with __slots__
__getattribute__ called with __new__
__getattribute__ called with __qualname__
__getattribute__ called with __module__
UNPICKLING
__getattribute__ called with __new__
Output with hide_pickle = False
PICKLING
__getattribute__ called with __reduce__
__getattribute__ called with __dict__
__getattribute__ called with __slots__
__getattribute__ called with __qualname__
__getattribute__ called with __module__
UNPICKLING
Linked PRs
Activitychanged the title [-]Difference in behavior between pure Python pickle and C extension _pickle when unpickling class instances.[/-] [+]Difference in behavior between pure Python pickle and C extension _pickle when unpickling instance of instance of metaclass[/+] on Jun 2, 2023chaerrypick01 commented on Aug 16, 2026 Hi @corona10 @hugovk, I'd like to work on this issue as part of the PyCon KR 2026 sprint.
I confirmed it still reproduces on current main (70fdc96).
One subtlety I noticed while investigating: the C unpickler is not always "metaclass-free". load_newobj in Modules/_pickle.c calls tp_new directly (even though the comment above the call says the intent is to call cls.__new__(cls, *args, **kwargs)), but when the class defines __new__ in Python, slot_tp_new itself looks __new__ up on the type, so the metaclass hook does fire even under _pickle. So whether the hook runs currently depends on whether the class being unpickled happens to define __new__ in Python.
Reproducer and output on main
from test.support.import_helper import import_fresh_module
c_pickle = import_fresh_module("pickle", fresh=["_pickle"])
py_pickle = import_fresh_module("pickle", blocked=["_pickle"])
LOG = []
class Meta(type):
def __getattribute__(cls, name):
LOG.append(name)
return super().__getattribute__(name)
class InheritsNew(metaclass=Meta): # tp_new is object.__new__ (a real C slot)
def __init__(self, value):
self.value = value
class DefinesNew(metaclass=Meta): # tp_new is slot_tp_new
def __new__(cls, *args):
return super().__new__(cls)
def __init__(self, value):
self.value = value
def lookups(mod, obj, protocol):
"""Attribute lookups the metaclass observes while unpickling."""
data = mod.dumps(obj, protocol=protocol)
LOG.clear()
mod.loads(data)
return sorted(set(LOG))
for cls in (InheritsNew, DefinesNew):
for protocol in (2, 5):
c = lookups(c_pickle, cls(42), protocol)
py = lookups(py_pickle, cls(42), protocol)
print(f"{cls.__name__} / protocol {protocol}: C={c!s:<13} "
f"py={py!s:<13} {'DIVERGES' if c != py else 'same'}")InheritsNew / protocol 2: C=[] py=['__new__'] DIVERGES
InheritsNew / protocol 5: C=[] py=['__new__'] DIVERGES
DefinesNew / protocol 2: C=['__new__'] py=['__new__'] same
DefinesNew / protocol 5: C=['__new__'] py=['__new__'] same
I'm planning to keep the fix as minimal as possible (mirroring the pure-Python cls.__new__ lookup on the _pickle.c side). If you think this issue isn't a good fit for the sprint, please let me know and I'll pick another one. Reacted by Donghee Na, Hugo van Kemenade and James Murphy corona10 commented on Aug 17, 2026 Reacted by Elly Chaewon Kim Reacted by Elly Chaewon Kim, Hugo van Kemenade and James Murphy
MetadataMetadataStatusShow more project fields
DevelopmentNo branches or pull requestsIssue actions
|