I'd estimate that 90% of the times that I introduce a new directory to the Python project, I forget that Python requires a two-phase operation to add new directories:

• Add the directory and files to the Git repo.
• Add a reference to the new directory to the Makefile.

If one completes only the first step, the tests will pass locally and in CI, but will fail in buildbots after merging. See #102010 for the most recent example.

Ideally, the build system should be reconfigured such that only one phase is required to add a new directory to the build.

Short of that, the CI builds should somehow check that the introduction of a new directory either triggers the buildbot "installed" builds or otherwise checks that the new directories are included in any source bundle.

It should not be a common occurrence to introduce a directory, pass CI, and then fail buildbots.