GeezerVison is showing github.com — Move _Py_RefTotal to PyInterpreterState · Issue #102304 · python/cpython
Move _Py_RefTotal to PyInterpreterState #102304Description(See gh-100227.)
_Py_RefTotal holds the current global total number of refcounts. It only exists if Py_REF_DEBUG is defined (implied by Py_DEBUG). It is exposed by sys.gettotalrefcount() and set by Py_INCREF(), Py_DECREF(), etc. and _Py_NewReference().
Modications to _Py_RefTotal are currently protected by the GIL so it should be moved to PyInterpreterState. For various aspects of compatibility, it makes sense to keep the _Py_RefTotal symbol around (and correct) and keep returning the global total from sys.gettotalrefcount().
Also, _Py_RefTotal is used by stable ABI extensions only where Py_REF_DEBUG is defined (unlikely) and only where built against 3.9 or earlier. Just in case, though, we must still keep the global variable around, so any solution here must respect that.
Linked PRs
Activityericsnowcurrently commented on Feb 27, 2023 From #100227 (comment):
FYI, I have a plan for _Py_RefTotal that should preserve stable ABI compatibility without breaking interpreter isolation (and without relying on granular locks). Basically:
- deprecate
_Py_Reftotal but keep exporting the symbol (for stable ABI compatibility)
- add
_PyRuntimeState.object_state.last_legacy_reftotal
- add
_PyRuntimeState.object_state.reftotal
- update everywhere to use the
reftotal field (for simplicity, add #define _Py_RefTotal _PyRuntime.object_state.reftotal)
We would update _Py_GetRefTotal() to something like this:
Py_ssize_t
_Py_GetRefTotal(void)
{
// _Py_GetLegacyRefTotal() returns the value of the actual `_Py_RefTotal` global.
Py_ssize_t legacy = _Py_GetLegacyRefTotal();
_PyRuntimeState.object_state.reftotal += legacy - _PyRuntimeState.object_state.last_legacy_reftotal;
_PyRuntimeState.object_state.last_legacy_reftotal = legacy;
return _PyRuntimeState.object_state.reftotal;
}All this assumes that folks are not using _Py_RefTotal directly.
For a per-interpreter GIL, I see two options:
- add a granular lock around modifying
_PyRuntimeState.object_state.reftotal, which would hurt the performance of incref/decref, but only on debug builds
- switch to
PyInterpreterState.object_state.reftotal and aggregate the values from all interpreters in _Py_GetRefTotal()
ericsnowcurrently commented on Feb 27, 2023 Alternately, from #100227 (comment):
_Py_RefTotal is only defined if Py_REF_DEBUG is defined.
All updates to _Py_RefTotal can be made atomic (which will have terrible performance with multiple parallel interpreters) to preserve the API.
kumaraditya303 commented on Mar 3, 2023 I vote to make it atomic instead of per interpreter. I use this while debugging refleaks and making it per interp will effectively make it impossible to use. This is only used in debug builds so I am not too much concerned about performance.
MetadataMetadataStatusShow more project fields
DevelopmentNo branches or pull requestsIssue actions
|