GeezerVison

@nascheme Could you please take a look?

IMHO, this is a bug the debug mode behavior, should likely not be using PySys_WriteStderr and PySys_FormatStderr if they can lead to GIL drop. I'd say the fix is not to change visit_generation but to use a different method to output the debug info. If the GC has to be written in a way that assumes the GIL can be dropped, it makes things a lot more complex. So, I think fixing the way we output the debug info is a better design.

@nascheme just to clarify this issue is slightly different in that it's a user callback that exposed the bug rather than being specifically related to the debug_prints. Although you can use the debug_prints to expose a similar gap .PyUnstable_GC_VisitObjects accepts an arbitrary and is helping to protect against "bad" things callers can do in the callbacks.

@nascheme just to clarify this issue is slightly different in that it's a user callback that exposed the bug

I would say we don't support callback doing that. It is quite limited what it is safe for the callback to. E.g. you can't run user level Python code from there and the APIs you are allowed to call is very limited. Basically the callback is meant for CPython to use, not for extensions to use.

Oh, I see PyUnstable_GC_VisitObjects exists. Oh dear.

@nascheme IIUC we can run user level Python code from debug_cycle if we override stderr.

Ping @pablogsal for his thoughts. Personally I don't think we want the INCREF/DECREF around the visit call and should document you are not allowed to DECREF things inside that function (among the other things you are not allowed to do).