GeezerVison

I've implemented proof of concept out of process async stack reconstruction here: 1st1@7185f8d

@mpage

Thanks for doing this! Do you know how much overhead the awaiter tracking adds?

I haven't measured, but I don't expect the overhead to be detectable at all. In the most common scenario the tracking is just assigning / resetting a pointer in Task/Future C structs.

@mpage

Thanks for doing this! Do you know how much overhead the awaiter tracking adds?

I haven't measured, but I don't expect the overhead to be detectable at all. In the most common scenario the tracking is just assigning / resetting a pointer in Task/Future C structs.

I ran some async benchmarks from pyperformance and a small echo tcp server and the Perf impact is below the noise.

@1st1 @ambv I fixed the crashes, refleaks and I did one of the most painful rebases ever, so please let get this merged as soon as possible to avoid collisions with the other work going on :)

There have been some changes in asyncio which are relevant for this:

• The C implementation is thread safe so new code should ideally be thread safe as well, thread safety is superficial here as tasks aren't thread safe really, it just shouldn't crash, mostly adding @critical_section to getter and methods would make it work.
• Add object locked assertion to any internal method if necessary
• There have been discussions about moving current task to per loop for free-threading, that might break the external introspection in future.

I haven't followed on this PR so not sure how many of those points matter here, but still I wrote it incase anyone else missed them.

• There have been discussions about moving current task to per loop for free-threading, that might break the external introspection in future.

Then we should find some solution that doesn't break the external introspection in the future ;)

• The C implementation is thread safe so new code should ideally be thread safe as well, thread safety is superficial here as tasks aren't thread safe really, it just shouldn't crash, mostly adding @critical_section to getter and methods would make it work.
• Add object locked assertion to any internal method if necessary

This is slightly tricky and if you recall we removed the locks we added at your request. This PR is already unwieldy complicated so I would prefer if we work together in a separate PR to ensure the lock safety is addressed as you would like it

This is slightly tricky and if you recall we removed the locks we added at your request. This PR is already unwieldy complicated so I would prefer if we work together in a separate PR to ensure the lock safety is addressed as you would like it

Yes, I do remember but it was long time ago 2-3 months, I have made significant changes related to free-threading since that time. It's fine by me to work on free-threading later and merge this first "as-is".

Thank you, Kumar. I landed this feature. We've received a ton of review here, definitely more than most changes. We addressed all feedback on design, correctness, and performance. The long-lived branch is hard to keep alive for longer. Thank you to everyone involved, let's evolve this forward from the main branch.

We'll help with #128869 and future free-threading compatibility to make sure external introspection keeps working. As part of the free-threading compatibility of this particular feature, we'll address this in a subsequent PR soon, where we will also add support for eager task external introspection. The new tests added by Kumar in January show some interesting behavior of eager tasks, I'm looking into that. Will provide more information on the subsequent PR.

Not sure if this is directly related, but we're seeing build bot failures now: https://buildbot.python.org/#/builders/125/builds/6894

test_async_gather_remote_stack_trace (test.test_external_inspection.TestGetStackTrace.test_async_gather_remote_stack_trace) ... Fatal Python error: _Py_CheckFunctionResult: a function returned NULL without setting an exception

Not sure if this is directly related, but we're seeing build bot failures now: https://buildbot.python.org/#/builders/125/builds/6894

test_async_gather_remote_stack_trace (test.test_external_inspection.TestGetStackTrace.test_async_gather_remote_stack_trace) ... Fatal Python error: _Py_CheckFunctionResult: a function returned NULL without setting an exception

On it

See also my comment #129189 (comment)

@pablogsal @ambv, this broke several tier-1 buildbots for more than a day so, by the book, it should be reverted. But, I also know there are issues with a buildbot update currently, and reverting & reapplying would be a lot churn in this case.

I'll leave it to you to handle this one.

FWIW, each working day I check there are no other buildbot failures, but I'll be offline during the weekend.

I wrote a PR to fix the test: #129262