Does the pickling work? I've been trying several things on pickable arguments, but I keep getting concurrent.interpreters.NotShareableError: args not shareable. My first thought was this:
# /// script
# dependencies = ["numpy"]
# requires-python = ">=3.14"
# ///
import numpy as np
from concurrent.futures import InterpreterPoolExecutor
def identity(*args):
return args[0]
if __name__ == "__main__":
x = np.random.rand(10003)
xs = np.array_split(x, 4)
with InterpreterPoolExecutor(max_workers=4) as pool:
results = pool.map(identity, xs)
final = np.concat(results)
assert np.all(x == final)But I get that error. Then I tried explicitly pickling:
# /// script
# dependencies = ["numpy"]
# requires-python = ">=3.14"
# ///
import numpy as np
import pickle
from concurrent.futures import InterpreterPoolExecutor
def identity(*args):
return pickle.dumps(pickle.loads(args[0]))
if __name__ == "__main__":
x = np.random.rand(10003)
xs = np.array_split(x, 4)
xsp = (pickle.dumps(a) for a in xs)
with InterpreterPoolExecutor(max_workers=4) as pool:
results = pool.map(identity, xsp)
final = np.concat([pickle.loads(r) for r in results])
assert np.all(x == final)But I still get this error:
Exception: AttributeError: module '__main__' has no attribute 'identity'
The above exception was the direct cause of the following exception:
concurrent.interpreters.NotShareableError: object could not be unpickled
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/henryschreiner/git/scikit-hep/boost-histogram/example.py", line 21, in <module>
final = np.concat([pickle.loads(r) for r in results])
^^^^^^^
File "/Users/henryschreiner/.local/share/uv/python/cpython-3.14.0rc1+freethreaded-macos-x86_64-none/lib/python3.14t/concurrent/futures/_base.py", line 639, in result_iterator
yield _result_or_cancel(fs.pop())
~~~~~~~~~~~~~~~~~^^^^^^^^^^
File "/Users/henryschreiner/.local/share/uv/python/cpython-3.14.0rc1+freethreaded-macos-x86_64-none/lib/python3.14t/concurrent/futures/_base.py", line 311, in _result_or_cancel
return fut.result(timeout)
~~~~~~~~~~^^^^^^^^^
File "/Users/henryschreiner/.local/share/uv/python/cpython-3.14.0rc1+freethreaded-macos-x86_64-none/lib/python3.14t/concurrent/futures/_base.py", line 450, in result
return self.__get_result()
~~~~~~~~~~~~~~~~~^^
File "/Users/henryschreiner/.local/share/uv/python/cpython-3.14.0rc1+freethreaded-macos-x86_64-none/lib/python3.14t/concurrent/futures/_base.py", line 395, in __get_result
raise self._exception
File "/Users/henryschreiner/.local/share/uv/python/cpython-3.14.0rc1+freethreaded-macos-x86_64-none/lib/python3.14t/concurrent/futures/thread.py", line 86, in run
result = ctx.run(self.task)
File "/Users/henryschreiner/.local/share/uv/python/cpython-3.14.0rc1+freethreaded-macos-x86_64-none/lib/python3.14t/concurrent/futures/interpreter.py", line 85, in run
return self.interp.call(do_call, self.results, *task)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/henryschreiner/.local/share/uv/python/cpython-3.14.0rc1+freethreaded-macos-x86_64-none/lib/python3.14t/concurrent/interpreters/__init__.py", line 233, in call
return self._call(callable, args, kwargs)
~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/henryschreiner/.local/share/uv/python/cpython-3.14.0rc1+freethreaded-macos-x86_64-none/lib/python3.14t/concurrent/interpreters/__init__.py", line 215, in _call
res, excinfo = _interpreters.call(self._id, callable, args, kwargs, restrict=True)
~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
concurrent.interpreters.NotShareableError: args not shareable
Also, I'm not sure why the Exception: AttributeError: module '__main__' has no attribute 'identity' without a traceback is at the top. Other executors are fine with code.
I was actually trying to use this in a test for boost-histogram, trying to pass a numpy array in and a boost-histogram object back out, but it fails even with byte strings from pickle above.
Edit: I do think it's an issue with the import like seen at the top, and the traceback is a red herring. The tests version of this list Exception: ModuleNotFoundError: No module named 'tests' at the top. So it's not able to import the function to run. I thought __main__ support was fixed?
It seems the subinterpreter's don't inherit the current sys.path? Using pytest 7's pythonpath setting doesn't work, they aren't picking up the added test paths.
If I put the function inside my package, I get ImportError: NumPy: dtype is already registered which I think that's occurring when pybind11 is registering my dtype. I've opened an issue pybind/pybind11#5781; if I manually skip over that, it looks like numpy doesn't support subinterpreters yet, so I'd not really be able to do much with it yet.