GeezerVison is showing github.com — Decide what should we do with doctests in CI · Issue #111704 · python/cpython
Decide what should we do with doctests in CI #111704DescriptionBug report
Initial PR and discussion: #111682
So, here's the problem:
Example 1::
>>> 1 + 2
3
Example 2:
.. doctest::
>>> 1 + 2
3 We have two tools: doctest (builtin module) and sphinx (3rd party) that can run these samples as doctests. The problem is that they define "doctest" term differently:
doctest module says that both of these examples are doctestssphinx says that only example with .. doctest:: directive is a doctest
This is problematic, because not all example are actually executed in CI, right now we have a CI job that uses Sphinx. So, in the result of this we have broken doctest (1st meaning) examples in the docs.
Examples:
What can we do?
Options:
- Add
.. doctest:: directives everywhere, but I think it is too much work and people will forget to do this
- Try to use
doctest_test_doctest_blocks, see docs https://www.sphinx-doc.org/en/master/usage/extensions/doctest.html This is what I plan to do locally and I will send a PR with the results. I fear that quite a lot of doctests might be broken
- Add doctests to
test_* files using something like
def load_tests(loader, tests, pattern):
tests.addTests(doctest.DocFileSuite(
os.path.join(REPO_ROOT, 'Doc/library/LIBNAME.rst'),
module_relative=False,
))
return testsThe last option is also problematic, because right now we can only run doctests in CI, when PR only touches Doc/ folder, I am not sure that it will be easy with this setup.
CC @AlexWaygood @AA-Turner
Linked PRs
Activitychanged the title [-]Decide what should we do with doctests[/-] [+]Decide what should we do with doctests in CI[/+] on Nov 3, 2023sobolevn commented on Nov 3, 2023 I hope that I will send the first PR today / tomorrow, but I am a bit sick :( Reacted by Alex Waygood and Terry Jan Reedy AlexWaygood commented on Nov 3, 2023 Option (2) is definitely attractive. Just two concerns from me:
- How long will this make the Doctest CI job? It already takes 10 minutes to run all CPython's doctests via Sphinx; I worry that this option could make it significantly slower, by dramatically increasing the number of doctests run in CI. This is hypothetical, though; I might be wrong!
- I agree with you that there'll probably be quite a few that fail at first. If we could figure out a way to tackle the errors incrementally, like what we've been doing with the Sphinx nitpicks, that would be ideal. Would it be possible to apply
doctest_test_doctest_blocks only to certain files? (Cc. @AA-Turner again as the Sphinx expert here!)
AlexWaygood commented on Nov 3, 2023 Also, rest up and get well soon! hugovk commented on Nov 3, 2023 Option 4: Modify Sphinx to run both types?
But I'm also concerned about the doctest taking longer. Of the three docs jobs, doctest is the longest at 10 minutes (= 3m setup/building + 7m doctest) compared to 2.5 minutes each for the other two:
Can we do them in parallel instead of sequential? Can the doctest module run them quicker? sobolevn commented on Nov 3, 2023
Modify Sphinx to run both types?
This is option 2 :) Reacted by Hugo van Kemenade sobolevn commented on Nov 7, 2023 After working with make doctest for a while, here are my thoughts:
- It does not allow to even run a single file (or maybe I cannot find the docs about it). This causes a lot of problems when debugging tests
- It uses stuff that is fundamentally incompatible with
-m doctest, like testsetup
- It is hard to clean things up after usage (see gh-111726: Explicitly close database connections in sqlite3 doctests #111730), because you have to separate cleanup from the test itself (to not show it to user):
.. testcleanup::
import os
os.remove("tutorial.db")
- If you want to skip some file, you have to:
I didn't like this tool at all.
It might be useful for small projects, but not for big ones. AlexWaygood commented on Nov 7, 2023
- Add
:skipIf: TO ALL DOCTESTS in a file (see
Can't you just not add .. doctest to the REPL snippets in the documentation that you don't want to be treated as doctests? (Though I know we'd need to change that if we were to switch to a system where sphinx automatically treated all REPL snippets as doctests, regardless of whether they have the .. doctest directive above them.) sobolevn commented on Nov 7, 2023
Can't you just not add .. doctest
Turns out that the answer is "no".
See #111735
It is a bit of a mess. sobolevn commented on Jul 10, 2025 I propose adding more text files to doctests, we can start with important ones and move slowly to increase the coverage.
There are two options:
.. doctest:: directive in the file itself. This allows us to add only some parts of the file to the doctest. Like we do here:
|
.. doctest:: |
|
|
|
>>> from typing import Annotated, get_type_hints |
|
>>> def func(x: Annotated[int, "metadata"]) -> None: pass |
|
... |
|
>>> get_type_hints(func) |
|
{'x': <class 'int'>, 'return': <class 'NoneType'>} |
|
>>> get_type_hints(func, include_extras=True) |
|
{'x': typing.Annotated[int, 'metadata'], 'return': <class 'NoneType'>} |
This way we can also create test setups and teardowns- Using
def load_test in some related test file with passing filename to load. Like we do here:
|
def load_tests(loader, tests, ignore): |
|
tests.addTests(doctest.DocTestSuite(enum)) |
|
|
|
lib_tests = os.path.join(REPO_ROOT, 'Doc/library/enum.rst') |
|
if os.path.exists(lib_tests): |
|
tests.addTests(doctest.DocFileSuite( |
|
lib_tests, |
|
module_relative=False, |
|
optionflags=doctest.ELLIPSIS|doctest.NORMALIZE_WHITESPACE, |
|
)) |
|
howto_tests = os.path.join(REPO_ROOT, 'Doc/howto/enum.rst') |
|
if os.path.exists(howto_tests) and sys.float_repr_style == 'short': |
|
tests.addTests(doctest.DocFileSuite( |
|
howto_tests, |
|
module_relative=False, |
|
optionflags=doctest.ELLIPSIS|doctest.NORMALIZE_WHITESPACE, |
|
)) |
|
return tests |
Important things to keep in mind:
- There are builds that do not have docs nearby
- There are different
float formatting options
- There are different optional dependencies
- There are different OSes
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|
