Abstract

This PEP proposes a redesign of the way in which built-in and extension modules interact with the import machinery. This was last revised for Python 3.0 in PEP 3121, but did not solve all problems at the time. The goal is to solve import-related problems by bringing extension modules closer to the way Python modules behave; specifically to hook into the ModuleSpec-based loading mechanism introduced in PEP 451.

This proposal draws inspiration from PyType_Spec of PEP 384 to allow extension authors to only define features they need, and to allow future additions to extension module declarations.

Extensions modules are created in a two-step process, fitting better into the ModuleSpec architecture, with parallels to __new__ and __init__ of classes.

Extension modules can safely store arbitrary C-level per-module state in the module that is covered by normal garbage collection and supports reloading and sub-interpreters. Extension authors are encouraged to take these issues into account when using the new API.

The proposal also allows extension modules with non-ASCII names.

Not all problems tackled in PEP 3121 are solved in this proposal. In particular, problems with run-time module lookup (PyState_FindModule) are left to a future PEP.

Motivation

Python modules and extension modules are not being set up in the same way. For Python modules, the module object is created and set up first, then the module code is being executed (PEP 302). A ModuleSpec object (PEP 451) is used to hold information about the module, and passed to the relevant hooks.

For extensions (i.e. shared libraries) and built-in modules, the module init function is executed straight away and does both the creation and initialization. The initialization function is not passed the ModuleSpec, or any information it contains, such as the __file__ or fully-qualified name. This hinders relative imports and resource loading.

In Py3, modules are also not being added to sys.modules, which means that a (potentially transitive) re-import of the module will really try to re-import it and thus run into an infinite loop when it executes the module init function again. Without access to the fully-qualified module name, it is not trivial to correctly add the module to sys.modules either. This is specifically a problem for Cython generated modules, for which it’s not uncommon that the module init code has the same level of complexity as that of any ‘regular’ Python module. Also, the lack of __file__ and __name__ information hinders the compilation of “__init__.py” modules, i.e. packages, especially when relative imports are being used at module init time.

Furthermore, the majority of currently existing extension modules has problems with sub-interpreter support and/or interpreter reloading, and, while it is possible with the current infrastructure to support these features, it is neither easy nor efficient. Addressing these issues was the goal of PEP 3121, but many extensions, including some in the standard library, took the least-effort approach to porting to Python 3, leaving these issues unresolved. This PEP keeps backwards compatibility, which should reduce pressure and give extension authors adequate time to consider these issues when porting.

The current process

Currently, extension and built-in modules export an initialization function named “PyInit_modulename”, named after the file name of the shared library. This function is executed by the import machinery and must return a fully initialized module object. The function receives no arguments, so it has no way of knowing about its import context.

During its execution, the module init function creates a module object based on a PyModuleDef object. It then continues to initialize it by adding attributes to the module dict, creating types, etc.

In the back, the shared library loader keeps a note of the fully qualified module name of the last module that it loaded, and when a module gets created that has a matching name, this global variable is used to determine the fully qualified name of the module object. This is not entirely safe as it relies on the module init function creating its own module object first, but this assumption usually holds in practice.

The proposal

The initialization function (PyInit_modulename) will be allowed to return a pointer to a PyModuleDef object. The import machinery will be in charge of constructing the module object, calling hooks provided in the PyModuleDef in the relevant phases of initialization (as described below).

This multi-phase initialization is an additional possibility. Single-phase initialization, the current practice of returning a fully initialized module object, will still be accepted, so existing code will work unchanged, including binary compatibility.

The PyModuleDef structure will be changed to contain a list of slots, similarly to PEP 384’s PyType_Spec for types. To keep binary compatibility, and avoid needing to introduce a new structure (which would introduce additional supporting functions and per-module storage), the currently unused m_reload pointer of PyModuleDef will be changed to hold the slots. The structures are defined as:

typedef struct {
    int slot;
    void *value;
} PyModuleDef_Slot;

typedef struct PyModuleDef {
    PyModuleDef_Base m_base;
    const char* m_name;
    const char* m_doc;
    Py_ssize_t m_size;
    PyMethodDef *m_methods;
    PyModuleDef_Slot *m_slots;  /* changed from `inquiry m_reload;` */
    traverseproc m_traverse;
    inquiry m_clear;
    freefunc m_free;
} PyModuleDef;

The m_slots member must be either NULL, or point to an array of PyModuleDef_Slot structures, terminated by a slot with id set to 0 (i.e. {0, NULL}).

To specify a slot, a unique slot ID must be provided. New Python versions may introduce new slot IDs, but slot IDs will never be recycled. Slots may get deprecated, but will continue to be supported throughout Python 3.x.

A slot’s value pointer may not be NULL, unless specified otherwise in the slot’s documentation.

The following slots are currently available, and described later:

• Py_mod_create
• Py_mod_exec

Unknown slot IDs will cause the import to fail with SystemError.

When using multi-phase initialization, the m_name field of PyModuleDef will not be used during importing; the module name will be taken from the ModuleSpec.

Before it is returned from PyInit_*, the PyModuleDef object must be initialized using the newly added PyModuleDef_Init function. This sets the object type (which cannot be done statically on certain compilers), refcount, and internal bookkeeping data (m_index). For example, an extension module “example” would be exported as:

static PyModuleDef example_def = {...}

PyMODINIT_FUNC
PyInit_example(void)
{
    return PyModuleDef_Init(&example_def);
}

The PyModuleDef object must be available for the lifetime of the module created from it – usually, it will be declared statically.

class BuiltinImporter:
    def create_module(self, spec):
        module = _imp.create_builtin(spec)

    def exec_module(self, module):
        _imp.exec_dynamic(module)

    def load_module(self, name):
        # use a backwards compatibility shim
        _load_module_shim(self, name)

class ExtensionFileLoader:
    def create_module(self, spec):
        module = _imp.create_dynamic(spec)

    def exec_module(self, module):
        _imp.exec_dynamic(module)

    def load_module(self, name):
        # use a backwards compatibility shim
        _load_module_shim(self, name)

def create_dynamic(spec):
    name = spec.name
    path = spec.origin

    # Find an already loaded module that used single-phase init.
    # For multi-phase initialization, mod is NULL, so a new module
    # is always created.
    mod = _PyImport_FindExtensionObject(name, name)
    if mod:
        return mod

    return _PyImport_LoadDynamicModuleWithSpec(spec)

def exec_dynamic(module):
    if not isinstance(module, types.ModuleType):
        # non-modules are skipped -- PyModule_GetDef fails on them
        return

    def = PyModule_GetDef(module)
    state = PyModule_GetState(module)
    if state is NULL:
        PyModule_ExecDef(module, def)

def create_builtin(spec):
    name = spec.name

    # Find an already loaded module that used single-phase init.
    # For multi-phase initialization, mod is NULL, so a new module
    # is always created.
    mod = _PyImport_FindExtensionObject(name, name)
    if mod:
        return mod

    for initname, initfunc in PyImport_Inittab:
        if name == initname:
            m = initfunc()
            if isinstance(m, PyModuleDef):
                def = m
                return PyModule_FromDefAndSpec(def, spec)
            else:
                # fall back to single-phase initialization
                module = m
                _PyImport_FixupExtensionObject(module, name, name)
                return module

def _PyImport_LoadDynamicModuleWithSpec(spec):
    path = spec.origin
    package, dot, name = spec.name.rpartition('.')

    # see the "Non-ASCII module names" section for export_hook_name
    hook_name = export_hook_name(name)

    # call platform-specific function for loading exported function
    # from shared library
    exportfunc = _find_shared_funcptr(hook_name, path)

    m = exportfunc()
    if isinstance(m, PyModuleDef):
        def = m
        return PyModule_FromDefAndSpec(def, spec)

    module = m

    # fall back to single-phase initialization
    ....

def PyModule_FromDefAndSpec(def, spec):
    name = spec.name
    create = None
    for slot, value in def.m_slots:
        if slot == Py_mod_create:
            create = value
    if create:
        m = create(spec, def)
    else:
        m = PyModule_New(name)

    if isinstance(m, types.ModuleType):
        m.md_state = None
        m.md_def = def

    if def.m_methods:
        PyModule_AddFunctions(m, def.m_methods)
    if def.m_doc:
        PyModule_SetDocString(m, def.m_doc)

def PyModule_ExecDef(module, def):
    if isinstance(module, types.module_type):
        if module.md_state is NULL:
            # allocate a block of zeroed-out memory
            module.md_state = _alloc(module.md_size)

    if def.m_slots is NULL:
        return

    for slot, value in def.m_slots:
        if slot == Py_mod_exec:
            value(module)

PyObject* (*PyModuleCreateFunction)(PyObject *spec, PyModuleDef *def)

int (*PyModuleExecFunction)(PyObject* module)

#include <Python.h>

static int spam_exec(PyObject *module) {
    PyModule_AddStringConstant(module, "food", "spam");
    return 0;
}

#ifdef Py_mod_exec
static PyModuleDef_Slot spam_slots[] = {
    {Py_mod_exec, spam_exec},
    {0, NULL}
};
#endif

static PyModuleDef spam_def = {
    PyModuleDef_HEAD_INIT,                      /* m_base */
    "spam",                                     /* m_name */
    PyDoc_STR("Utilities for cooking spam"),    /* m_doc */
    0,                                          /* m_size */
    NULL,                                       /* m_methods */
#ifdef Py_mod_exec
    spam_slots,                                 /* m_slots */
#else
    NULL,
#endif
    NULL,                                       /* m_traverse */
    NULL,                                       /* m_clear */
    NULL,                                       /* m_free */
};

PyMODINIT_FUNC
PyInit_spam(void) {
#ifdef Py_mod_exec
    return PyModuleDef_Init(&spam_def);
#else
    PyObject *module;
    module = PyModule_Create(&spam_def);
    if (module == NULL) return NULL;
    if (spam_exec(module) != 0) {
        Py_DECREF(module);
        return NULL;
    }
    return module;
#endif
}

PyObject * PyModule_FromDefAndSpec(PyModuleDef *def, PyObject *spec)
PyObject * PyModule_FromDefAndSpec2(PyModuleDef *def, PyObject *spec,
                                    int module_api_version)

PyAPI_FUNC(int) PyModule_ExecDef(PyObject *module, PyModuleDef *def)

PyObject * PyModuleDef_Init(PyModuleDef *);

int PyModule_SetDocString(PyObject *, const char *)
int PyModule_AddFunctions(PyObject *, PyMethodDef *)

def export_hook_name(name):
    try:
        suffix = b'_' + name.encode('ascii')
    except UnicodeEncodeError:
        suffix = b'U_' + name.encode('punycode').replace(b'-', b'_')
    return b'PyInit' + suffix

import importlib.machinery
import importlib.util
loader = importlib.machinery.ExtensionFileLoader(name, path)
spec = importlib.util.spec_from_loader(name, loader)
module = importlib.util.module_from_spec(spec)
loader.exec_module(module)
return module

Summary of API Changes and Additions

New functions:

• PyModule_FromDefAndSpec (macro)
• PyModule_FromDefAndSpec2
• PyModule_ExecDef
• PyModule_SetDocString
• PyModule_AddFunctions
• PyModuleDef_Init

New macros:

• Py_mod_create
• Py_mod_exec

New types:

• PyModuleDef_Type will be exposed

New structures:

• PyModuleDef_Slot

Other changes:

PyModuleDef.m_reload changes to PyModuleDef.m_slots.

BuiltinImporter and ExtensionFileLoader will now implement create_module and exec_module.

The internal _imp module will have backwards incompatible changes: create_builtin, create_dynamic, and exec_dynamic will be added; init_builtin, load_dynamic will be removed.

The undocumented functions imp.load_dynamic and imp.init_builtin will be replaced by backwards-compatible shims.

Possible Future Extensions

The slots mechanism, inspired by PyType_Slot from PEP 384, allows later extensions.

Some extension modules exports many constants; for example _ssl has a long list of calls in the form:

PyModule_AddIntConstant(m, "SSL_ERROR_ZERO_RETURN",
                        PY_SSL_ERROR_ZERO_RETURN);

Converting this to a declarative list, similar to PyMethodDef, would reduce boilerplate, and provide free error-checking which is often missing.

String constants and types can be handled similarly. (Note that non-default bases for types cannot be portably specified statically; this case would need a Py_mod_exec function that runs before the slots are added. The free error-checking would still be beneficial, though.)

Another possibility is providing a “main” function that would be run when the module is given to Python’s -m switch. For this to work, the runpy module will need to be modified to take advantage of ModuleSpec-based loading introduced in PEP 451. Also, it will be necessary to add a mechanism for setting up a module according to slots it wasn’t originally defined with.

Implementation

Work-in-progress implementation is available in a GitHub repository [3]; a patchset is at [4].

Previous Approaches

Stefan Behnel’s initial proto-PEP [1] had a “PyInit_modulename” hook that would create a module class, whose __init__ would be then called to create the module. This proposal did not correspond to the (then nonexistent) PEP 451, where module creation and initialization is broken into distinct steps. It also did not support loading an extension into pre-existing module objects.

Alyssa (Nick) Coghlan proposed “Create” and “Exec” hooks, and wrote a prototype implementation [2]. At this time PEP 451 was still not implemented, so the prototype does not use ModuleSpec.

The original version of this PEP used Create and Exec hooks, and allowed loading into arbitrary pre-constructed objects with Exec hook. The proposal made extension module initialization closer to how Python modules are initialized, but it was later recognized that this isn’t an important goal. The current PEP describes a simpler solution.

A further iteration used a “PyModuleExport” hook as an alternative to PyInit, where PyInit was used for existing scheme, and PyModuleExport for multi-phase. However, not being able to determine the hook name based on module name complicated automatic generation of PyImport_Inittab by tools like freeze. Keeping only the PyInit hook name, even if it’s not entirely appropriate for exporting a definition, yielded a much simpler solution.

References

Copyright

This document has been placed in the public domain.