GeezerVison is showing github.com — Move Benchmarks From Tools/ to the pyperformance Repo · Issue #97680 · python/cpython
Move Benchmarks From Tools/ to the pyperformance Repo #97680Activityarhadthedev commented on Oct 6, 2022
maybe others?
Tool/scripts/var_access_benchmark.py Reacted by Eric Snow and Donghee Na Reacted by Raymond Hettinger rhettinger commented on Oct 10, 2022 Why put them somewhere else? These are used during CPython development to quickly test the effect of changes. If you put them in another repo, they just get harder to access — I would likely never use them again or would have to keep a local copy.
IIRC the original purpose of the Tools directly was as a convenient place for us to keep scripts that we use during development. gvanrossum commented on Oct 10, 2022 I created the “faster Python” project and we definitely don’t use these to assess performance improvements. ericsnowcurrently commented on Oct 10, 2022 Regardless of the value of keeping these in the Tools directory, adding these micro benchmarks to pyperformance is still likely worth doing.
(FWIW, I have no real interest either way in the ongoing Tools/ cleanup.) corona10 commented on Oct 11, 2022 @ericsnowcurrently
I concur with Eric's comment.
As the #98152 author (even if it was duplicated).
The Faster CPython team measures performance regression for every iteration so to make their task more effective we should centralize benchmark suites as possible.
And also nowadays most of performance people are looking pyperformance benchmarks instead of running the manual bench suites under Tools directory.
If we port them into pyperformance benchmarks we can get the diff result for every iteration from https://github.com/faster-cpython/ideas/tree/main/benchmark-results.
Yeah, accessibility from some people can be degraded but I think that overall gain will be positive if we move them into pyperformance projects and I believe that we should reduce fragmentation as possible. gvanrossum commented on Oct 11, 2022 If someone submits a PR to pyperformance can they please link to it here? vstinner commented on Oct 12, 2022 I proposed #98214 to just remove Tools/scripts/startuptime.py: there are other existing tools to measure Python startup time. vstinner commented on Oct 12, 2022
Tool/scripts/var_access_benchmark.py
That's a micro-benchmark. I don't think that it belongs to pyperformance which tries to be a collection of macro benchmarks. Well, if someone wants to move it there, go ahead.
In the past, I created https://github.com/vstinner/pymicrobench which is a collection of CPython micro-benchmarks. If someone is interested by microbenchmarks, maybe it should be moved somewhere else. Usually, I only run a micro-benchmark once to prove that a change makes a function faster. But later, I don't use them anymore.
If possible, I would prefer to move var_access_benchmark.py outside CPython source code, since I'm trying to remove Tools/scripts/: see issue #97669 :-) vstinner commented on Oct 12, 2022 stringbench is also documented as being a microbenchmark:
Various microbenchmarks comparing unicode and byte string performance
I'm not sure if pyperformance is a good fit for it. Sophist-UK commented on Feb 18, 2024 I would have commented sooner had I known that this was about to happen. It would have been helpful if someone had commented in my PR to ccbench.py that this was being considered so that I would have been notified and might have had a chance to comment before it happened.
I believe it was clear from the extensive discussion about the shortcomings of the current GIL in faster-cpython that the ccbench script is still useful - because it clearly identifies an ongoing performance issue when multithreading with one or more CPU-heavy threads.
That said, I do understand that performance scripts do not really have a place in the cpython source code repo, but unfortunately I do not have the time to rework the updated ccbench script for the pyperformance suite, so I guess it is going to just disappear.
@vstinner Victor : I am sorry to hear that your own micro-benchmarking scripts are also languishing. Despite an overall good focus on cpython performance improvements, unfortunately from the evidence of the above referenced discussion, the lack of response to your appeal for someone to take on your own code and this issue/PR removing performance scripts do suggest that the focus on cpython performance improvements is somewhat narrow and clique-esque. Ah well - we can but try, hugovk commented on Feb 18, 2024 Note: the above is reference to #101853 that I just merged (after being open for a year) which removed Tools/{io,cc,string}bench.
I would have commented sooner had I known that this was about to happen. It would have been helpful if someone had commented in my PR to ccbench.py that this was being considered so that I would have been notified and might have had a chance to comment before it happened.
Apologies, I'll ping next time this sort of thing comes up. (btw we wouldn't be allowed to merge your PR because it looks like the CLA isn't signed.)
That said, I do understand that performance scripts do not really have a place in the cpython source code repo, but unfortunately I do not have the time to rework the updated ccbench script for the pyperformance suite, so I guess it is going to just disappear.
Yep, in general, benchmarks are much more useful in pyperformance, because they're run often and people pay attention to their results.
The code hasn't disappeared, it's in the 3.12 and older branches, and in Git history, so can still be referred to when/if adding to pyperformance.
And of course, if there's consensus we can easily revert/re-add bits of #101853.
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|