#155619

Reproduction

When threads race, the watcher registry hands out duplicate IDs and accepts duplicate clears. On current main the new tests fail on a free-threaded debug build:

The same binary with PYTHON_GIL=1 passes. (The double-clear test skips itself there, since the calls can't overlap.)

gh-159739 already replaced the assert(cb != NULL) in notify_context_watchers() with a NULL check, so a clear racing with a context switch no longer crashes. What's left is that the registry is read and written with plain accesses from several threads, and an add or clear isn't atomic as a whole.

Fix

PyContext_AddWatcher() and PyContext_ClearWatcher() now run under Py_BEGIN_CRITICAL_SECTION_MUTEX() on a new PyMutex in the interpreter state, the same pattern as TYPE_LOCK. They run about once per extension, so the lock costs nothing that matters.

notify_context_watchers() runs on every context switch and still takes no lock. It loads each callback with an acquire load, paired with the release store in PyContext_AddWatcher(), and keeps gh-159739's NULL check for a watcher cleared by an earlier callback or by another thread. I kept the bitmask rather than scanning all eight slots because it makes the common zero-watcher case one load and a not-taken branch.

Docs

Doc/c-api/contextvars.rst had no threading guidance at all, unlike dict and type watchers, so I added some.

Testing

The new tests pass 20/20 runs. test_free_threading, test_context, test_capi, test_contextlib and test_asyncio pass on a free-threaded debug build (5,149 tests), and so does the same binary with PYTHON_GIL=1 (1,690 tests).

test_concurrent_clear_same_watcher releases its threads with a busy-wait instead of a Barrier. A barrier wakes them too far apart to overlap and found nothing in 20,000 rounds, while the busy-wait hits the race in about a quarter of rounds.

No ThreadSanitizer run from me. It wouldn't catch the double clear anyway: in the previous revision every access was already atomic, so there was no data race to report, just clears that weren't atomic as a whole.

Benchmarks

pyperformance async_tree, async_tree_io, async_generators (asyncio runs each task step through ctx.run()), plus nbody as a control that never enters a context. Free-threaded build, baseline vs. this change:

These were measured on the previous revision. Notification, the only path they exercise, hasn't changed since; add and clear now take a lock, but they aren't on that path.

The default build was the same story, except async_generators came out 1.08x slower there. I think that one is noise: its spread is about 12% of the mean, t=-2.54, and the same benchmark went faster on the free-threaded build. These were plain -O3 builds with --fast on a laptop, so they answer "does this cost anything" and not much else; happy to redo them properly if it matters.

• Issue: PyContext watcher registry has data races in free-threaded builds #155619