GeezerVison

Is the goal for this benchmark to guide our future improvements, or just to highlight the improvement already made in the linked issue?

We should really avoid the latter... the last thing I want is to add synthetic benchmarks just to boost our 3.12 numbers.

Is the goal for this benchmark to guide our future improvements, or just to highlight the improvement already made in the linked issue?

It is to guide for future improvements and more importantly avoid regressions. As it currently stands there are no benchmarks for asyncio thus it is hard to evaluate the impact of any bug fixes on performance, having this is pyperformance will help in tracking this. (recent python/cpython#100053)

We should really avoid the latter... the last thing I want is to add synthetic benchmarks just to boost our 3.12 numbers.

That's true though this isn't any different from other benchmark, even if there is improvement in pyperformance it doesn't mean the user's application will be that much faster or faster at all.

I am fine with keeping this in my own fork but I would prefer to have it in pyperformance so it can be tracked using speed.python.org.

FYI if this is acceptable then I have other benchmarks to add for asyncio notably udp and ssl performance.

Comment which does not judge this PR:
I think that adding a new benchmark will consume the cost of running time for measuring.
Since we can not add every single benchmark for each dimension, each benchmark should cover a lot of dimensions of performance if possible.

From this view,

• asyncio-based http benchmark will be better than asyncio-based TCP benchmark.
• asyncio-based https benchmark will be better than asyncio-based SSL benchmark.
• asyncio-based http3 benchmark will be better than asyncio-based UDP benchmark.

I think that adding a new benchmark will consume the cost of running time for measuring.

I was under the impression that cost of running the benchmark is not an issue. If it is an issue then it isn't worth it and I can run it myself when necessary so no need for this in pyperformance.

I think that adding a new benchmark will consume the cost of running time for measuring.
Since we can not add every single benchmark for each dimension, each benchmark should cover a lot of dimensions of performance if possible.

If running time is a concern, we can restrict the set of benchmarks that are run by default. That is easily done in the manifest file. Then we can do longer runs whenever we feel comfortable with the extra time by passing the --benchmarks all (or maybe it is ""?) option to pyperformance.

In the long term, ideally we would have a solid set of workload-oriented ("macro") benchmarks that measure real-world performance (close enough, at least). This is what we would use to measure performance effectively and those results are what we would communicate to users. However, we still need the wide variety of feature-oriented ("micro") benchmarks, to help us avoid regressions and narrow down causes when we have regressions, as well as point to where possible improvements might be. See https://github.com/faster-cpython/ideas/wiki/All-About-Python-Benchmarking.

Currently, pretty much all of the pyperformance benchmarks are feature-oriented ("micro"). So we have plenty of work ahead of us. (Note that the pyston benchmarks are much closer to workload-oriented and we have been incorporating them into the faster-cpython results as much as possible.)

If running time is a concern, we can restrict the set of benchmarks that are run by default.

Ah okay, in that case, I am okay to add it.
I just worried about all various of bench_asyncio_xxxx.py.

Final benchmark result:

I plan to merge this by weekend if there are no objections.

@mdboom What do you think?

I'll merge this now as python/cpython#31871 is almost done, awaiting final review and merge. We can sort out any issues with this later down the road.