GeezerVison

I thought there had been a decision not to add hypothesis tests to any module not under active development. During development, hypothesis is good at teasing out special cases that need to be fixed and have an explicit targeted test. Once in production, hypothesis doesn't do much for us and leaves it unclear whether the search space for a property has actually been covered.

AFAICT this PR doesn't add any value, it just eats clock cycles and adds another tool that a person has to learn before they can read the tests. If our current tests are inadequate in any way, we should add specifically targeted tests to fill the gaps.

We should differentiate a goal we care about, "thoroughly understand and test our code" from the more dubious goal, "Let's use this tool every we can." ISTM that this PR only does the latter and doesn't help base64 in any way.

Also note, that we've already had a number of people do the exercise of trying hypothesis on many standard library modules. We were able to do that without actually checking in the tests into main distribution. FWIW, the main result of the exercise was identifying bugs in the tester's understanding of the module (i.e. an expectation that colorsys conversions would round-trip without understanding gamut limits and published specifications that did not round-trip).

I think these concerns are addressed in python/steering-council#65 and #107862; the stub implementation we added in #22863 has very low overhead.

We're starting with simpler cases rather than e.g. #83134, because it's easier to shake out config issues and get used to the workflow when we don't also need to work out how we'll handle e.g. the strategy to generate arbitrary Python source code.

I thought there had been a decision not to add hypothesis tests to any module not under active development.

I couldn't find that decision, do you have a link to it?
I'm happy to ask the SC for a decision if that's what we need.

I agree that the issues found by Hypothesis should be added to the tests that always run, though it might be enough to add them as @example.
Gaps in understanding can usually be used improve documentation -- see #119409 for two non-obvious things I found here.

Checking this into the repo means it stays in sync with stdlib.

Focusing on this particular PR as an example of what is to come, is anyone claiming that it adds any value at all? Will users be any better-off? I don't think so.

AFAICT only the time and datetime module maintainer has said they wanted this (because it is part of their development process). No other module maintainer has asked for this. I expect that every such PR in the future will come from devs who have not previously shown any interest in the module being tested. Their goal isn't to improve the module. Their goal is just to use hypothesis. IMO that is a false goal. I don't think it should be done.

The goal is to use property-based testing in general -- i.e. using a tool to find edge cases that humans are bad at finding.

You're right that this is meant for tests in general: rather than base64 itself I'm interested in hypothesis/regrtest integration, the stubs (see the second commit here), and finding usability issues in running these tests. I'd love to get to the point where people who are actively developing a module have a bunch of examples and institutional knowledge ready to help them find new edge cases.

FWIW, we already test properties (invariants) almost everywhere they are applicable (even in base64). You don't need hypothesis for that.

Hypothesis doesn't have a monopoly on properties or invariants (pretty much every existing assertion does that). The superpower of hypothesis is generating semi-random input patterns to find problems and then simplify to a minimal reproducer. That is helpful during development but not really something we want in the CI. Instead, a core-dev is expected to produce a comprehensive and deterministic sets of tests that reflect a full understanding of the code (both black-box and white-box).

I do not see how “not really something we want in the CI” follows from what was said before.

If this stuff is helpful during development, it should also be helpful to testing that the hypothesis/regrtest integration works as intended, and to have a bunch of worked examples around, so that getting it to run when you need it isn't an uphill battle.

Yes, a core dev is expected to produce good tests. Property-based testing is a tool to help with that. It also encodes the expected invariants for future generations, in a more useful format than only a set of examples.

I don't think we need to re-litigate the question of using property-based testing in every thread that adds some tests. I don't see any indication of a decision to keep these tests to a minimum or to give them any special treatment at all. @encukou found actual concrete issues as part of making these tests, and @Zac-HD and I both agree that these are a useful addition.

Since the only objection to this particular PR is that it adds property-based (hypothesis) tests and none of us can point to any policy against doing such a thing or any kind of conditions attached when the testing framework was first added, I'm going to go ahead and merge this.