GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-62432: unittest runner: Exit code 5 if no tests were run by stefanor · Pull Request #102051 · python/cpython

Conversation

@stefanor

@stefanor stefanor commented Feb 19, 2023 •
edited by bedevere-bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

As discussed in https://discuss.python.org/t/unittest-fail-if-zero-tests-were-discovered/21498/7

It is common for test runner misconfiguration to fail to find any tests,
this should be an error.

Fixes: #62432

  • Issue: running a suite with no tests is not an error #62432

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
stefanor added 3 commits February 19, 2023 11:41
@stefanor
Refactor test_program to allow for multiple test suites
0a2007d
@stefanor
Refactor test_program to use assertRaises as a context manager
6532200
@stefanor
unittest: Print an error message if no tests ran
ebba337
@bedevere-bot bedevere-bot added the awaiting review label Feb 19, 2023
@bedevere-bot bedevere-bot mentioned this pull request Feb 19, 2023
running a suite with no tests is not an error #62432
Closed
@stefanor
stefanor force-pushed the fail-zero-tests branch from eab4b21 to d8b54a3 Compare February 19, 2023 16:37
stefanor added 3 commits February 19, 2023 12:41
@stefanor
unittest runner: Exit code 5 if no tests were run
5e1a88d 
As discussed in https://discuss.python.org/t/unittest-fail-if-zero-tests-were-discovered/21498/7

It is common for test runner misconfiguration to fail to find any tests,
this should be an error.

Fixes: python#62432
@stefanor
Count the test run in our stub test so that the suite succeeds
fb67714
@stefanor
Update ACKs
0687ca8
@stefanor
stefanor force-pushed the fail-zero-tests branch from d8b54a3 to 0687ca8 Compare February 19, 2023 16:41
@terryjreedy
terryjreedy requested review from ezio-melotti and gpshead February 20, 2023 03:13
@terryjreedy terryjreedy added type-feature A feature request or enhancement tests Tests in the Lib/test dir labels Feb 20, 2023
@terryjreedy

terryjreedy commented Feb 20, 2023

Copy link
Copy Markdown
Member

Tests/macOS is routinely failing; ignore currently.

This is a design bug fix, hence enhancement (feature change), as much as I would like to pretend otherwise.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
hugovk
hugovk reviewed Feb 20, 2023
View reviewed changes
Comment thread Lib/unittest/main.py Resolved

Uh oh!

There was an error while loading. Please reload this page.
Comment thread Lib/unittest/runner.py Resolved

Uh oh!

There was an error while loading. Please reload this page.
Comment thread Lib/test/test_unittest/test_program.py Resolved

Uh oh!

There was an error while loading. Please reload this page.
Comment thread Lib/test/test_unittest/test_runner.py Resolved

Uh oh!

There was an error while loading. Please reload this page.
Comment thread Misc/NEWS.d/next/Library/2023-02-19-12-37-08.gh-issue-62432.GnBFIB.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.
Comment thread Lib/test/test_unittest/test_program.py Resolved

Uh oh!

There was an error while loading. Please reload this page.
stefanor added 4 commits February 20, 2023 07:45
@stefanor
Put success first in the if ladder
3ac2e71
@stefanor
NO TESTS RAN, in past tense
7470779
@stefanor
The parameter name was testcase
ecb363c
@stefanor
Drop a comma
7d6c322
@stefanor
stefanor requested review from hugovk and removed request for ezio-melotti and gpshead February 20, 2023 11:51
@stefanor

stefanor commented Feb 20, 2023

Copy link
Copy Markdown
Contributor Author

@stefanor stefanor requested review from hugovk and removed request for gpshead and ezio-melotti

Urgh, that removal wasn't intentional, I just hit the re-request review button.
hugovk reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@hugovk
hugovk requested review from ezio-melotti and gpshead February 20, 2023 12:42
@hauntsaninja
Merge branch 'main' into fail-zero-tests
058e22c
@gpshead gpshead self-assigned this Feb 20, 2023
@stefanor

stefanor commented Apr 26, 2023

Copy link
Copy Markdown
Contributor Author

@gpshead: Ping? (it's been 2 months)
gpshead reacted with heart emoji
All reactions
  • ❤️ 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@AlexWaygood
Merge branch 'main' into fail-zero-tests
455ca8b
@gpshead
gpshead enabled auto-merge (squash) April 27, 2023 01:04
@gpshead
gpshead merged commit 76632b8 into python:main Apr 27, 2023
@bedevere-bot bedevere-bot removed the awaiting merge label Apr 27, 2023
@stefanor
stefanor deleted the fail-zero-tests branch April 27, 2023 11:49
@stefanor

stefanor commented Apr 27, 2023

Copy link
Copy Markdown
Contributor Author

Thank you!
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@gpshead gpshead added the 3.12 only security fixes label Jun 13, 2023
copybara-service Bot pushed a commit to abseil/abseil-py that referenced this pull request Sep 14, 2023
@yilei @copybara-github
absltest: do not fail tests on Python 3.12+ when no tests ran AND:
b274e14 
- Either test filtering is used
- Sharding is used and the shard index > 0, i.e. only the first shard will fail when no tests ran on Python 3.12+.

Context: Python 3.12 unittest will now fail when no tests ran after the change from python/cpython#102051. Since `absltest` is built on top of `unittest`, it will follow this behavior change in Python 3.12.

However, when test filtering is used in `absltest`, often used via `bazel test --test_filter=<my_filter>`, the current user expectation is the `bazel test` command should NOT fail is at least one test ran. Since the test runner here has no visibility of the overall `bazel` invocation, we'll make the test not fail when test filtering is used. This is the existing behavior before Python 3.12.

Also test absl-py on Python 3.12.

PiperOrigin-RevId: 565190992
@copybara-service copybara-service Bot mentioned this pull request Sep 14, 2023
absltest: do not fail tests on Python 3.12+ when no tests ran and: abseil/abseil-py#253
Merged
copybara-service Bot pushed a commit to abseil/abseil-py that referenced this pull request Sep 14, 2023
@yilei @copybara-github
absltest: do not fail tests on Python 3.12+ when no tests ran AND:
16bcbeb 
- Either test filtering is used
- Sharding is used and the shard index > 0, i.e. only the first shard will fail when no tests ran on Python 3.12+.

Context: Python 3.12 unittest will now fail when no tests ran after the change from python/cpython#102051. Since `absltest` is built on top of `unittest`, it will follow this behavior change in Python 3.12.

However, when test filtering is used in `absltest`, often used via `bazel test --test_filter=<my_filter>`, the current user expectation is the `bazel test` command should NOT fail is at least one test ran. Since the test runner here has no visibility of the overall `bazel` invocation, we'll make the test not fail when test filtering is used. This is the existing behavior before Python 3.12.

Also test absl-py on Python 3.12.

PiperOrigin-RevId: 565190992
copybara-service Bot pushed a commit to abseil/abseil-py that referenced this pull request Sep 14, 2023
@yilei @copybara-github
absltest: do not fail tests on Python 3.12+ when no tests ran AND:
4d8eba2 
- Either test filtering is used
- Sharding is used and the shard index > 0, i.e. only the first shard will fail when no tests ran on Python 3.12+.

Context: Python 3.12 unittest will now fail when no tests ran after the change from python/cpython#102051. Since `absltest` is built on top of `unittest`, it will follow this behavior change in Python 3.12.

However, when test filtering is used in `absltest`, often used via `bazel test --test_filter=<my_filter>`, the current user expectation is the `bazel test` command should NOT fail is at least one test ran. Since the test runner here has no visibility of the overall `bazel` invocation, we'll make the test not fail when test filtering is used. This is the existing behavior before Python 3.12.

Also test absl-py on Python 3.12.

PiperOrigin-RevId: 565190992
copybara-service Bot pushed a commit to abseil/abseil-py that referenced this pull request Sep 14, 2023
@yilei @copybara-github
absltest: do not fail tests on Python 3.12+ when no tests ran AND:
6c95d4d 
- Either test filtering is used
- Sharding is used and the shard index > 0, i.e. only the first shard will fail when no tests ran on Python 3.12+.

Context: Python 3.12 unittest will now fail when no tests ran after the change from python/cpython#102051. Since `absltest` is built on top of `unittest`, it will follow this behavior change in Python 3.12.

However, when test filtering is used in `absltest`, often used via `bazel test --test_filter=<my_filter>`, the current user expectation is the `bazel test` command should NOT fail is at least one test ran. Since the test runner here has no visibility of the overall `bazel` invocation, we'll make the test not fail when test filtering is used via bazel's environment variable. This is the existing behavior before Python 3.12.

Also test absl-py on Python 3.12.

PiperOrigin-RevId: 565190992
copybara-service Bot pushed a commit to abseil/abseil-py that referenced this pull request Sep 14, 2023
@yilei @copybara-github
absltest: do not fail tests on Python 3.12+ when no tests ran AND:
9ff9bfa 
- Either test filtering is used
- Sharding is used and the shard index > 0, i.e. only the first shard will fail when no tests ran on Python 3.12+.

Context: Python 3.12 unittest will now fail when no tests ran after the change from python/cpython#102051. Since `absltest` is built on top of `unittest`, it will follow this behavior change in Python 3.12.

However, when test filtering is used in `absltest`, often used via `bazel test --test_filter=<my_filter>`, the current user expectation is the `bazel test` command should NOT fail is at least one test ran. Since the test runner here has no visibility of the overall `bazel` invocation, we'll make the test not fail when test filtering is used via bazel's environment variable. This is the existing behavior before Python 3.12.

Also test absl-py on Python 3.12.

PiperOrigin-RevId: 565190992
copybara-service Bot pushed a commit to abseil/abseil-py that referenced this pull request Sep 14, 2023
@yilei @copybara-github
absltest: do not fail tests on Python 3.12+ when no tests ran and:
8d1a40c 
- Either test filtering is used
- Or sharding is used and the shard index > 0, i.e. only the first shard will fail when no tests ran on Python 3.12+.

Context: Python 3.12 unittest will now fail when no tests ran after the change from python/cpython#102051. Since `absltest` is built on top of `unittest`, it will follow this behavior change in Python 3.12.

However, when test filtering is used in `absltest`, often used via `bazel test --test_filter=<my_filter>`, the current user expectation is the `bazel test` command should NOT fail is at least one test ran. Since the test runner here has no visibility of the overall `bazel` invocation, we'll make the test not fail when test filtering is used via bazel's environment variable. This is the existing behavior before Python 3.12.

Also test absl-py on Python 3.12.

PiperOrigin-RevId: 565190992
copybara-service Bot pushed a commit to abseil/abseil-py that referenced this pull request Sep 14, 2023
@yilei @copybara-github
absltest: do not fail tests on Python 3.12+ when no tests ran and:
bbd341c 
- Either test filtering is used
- Or sharding is used and the shard index > 0, i.e. only the first shard will fail when no tests ran on Python 3.12+.

Context: Python 3.12 unittest will now fail when no tests ran after the change from python/cpython#102051. Since `absltest` is built on top of `unittest`, it will follow this behavior change in Python 3.12.

However, when test filtering is used in `absltest`, often used via `bazel test --test_filter=<my_filter>`, the current user expectation is the `bazel test` command should NOT fail is at least one test ran. Since the test runner here has no visibility of the overall `bazel` invocation, we'll make the test not fail when test filtering is used via bazel's environment variable. This is the existing behavior before Python 3.12.

Also test absl-py on Python 3.12.

PiperOrigin-RevId: 565190992
copybara-service Bot pushed a commit to abseil/abseil-py that referenced this pull request Sep 15, 2023
@yilei @copybara-github
absltest: do not fail tests on Python 3.12+ when no tests ran and:
a4db4a8 
- Either test filtering is used
- Or sharding is used and the shard index > 0, i.e. only the first shard will fail when no tests ran on Python 3.12+.

Context: Python 3.12 unittest will now fail when no tests ran after the change from python/cpython#102051. Since `absltest` is built on top of `unittest`, it will follow this behavior change in Python 3.12.

However, when test filtering is used in `absltest`, often used via `bazel test --test_filter=<my_filter>`, the current user expectation is the `bazel test` command should NOT fail is at least one test ran. Since the test runner here has no visibility of the overall `bazel` invocation, we'll make the test not fail when test filtering is used via bazel's environment variable. This is the existing behavior before Python 3.12.

Also test absl-py on Python 3.12.

PiperOrigin-RevId: 565190992
copybara-service Bot pushed a commit to abseil/abseil-py that referenced this pull request Sep 15, 2023
@yilei @copybara-github
absltest: do not fail tests on Python 3.12+ when no tests ran and:
14f99e8 
- Either test filtering is used
- Or sharding is used and the shard index > 0, i.e. only the first shard will fail when no tests ran on Python 3.12+.

Context: Python 3.12 unittest will now fail when no tests ran after the change from python/cpython#102051. Since `absltest` is built on top of `unittest`, it will follow this behavior change in Python 3.12.

However, when test filtering is used in `absltest`, often used via `bazel test --test_filter=<my_filter>`, the current user expectation is the `bazel test` command should NOT fail is at least one test ran. Since the test runner here has no visibility of the overall `bazel` invocation, we'll make the test not fail when test filtering is used via bazel's environment variable. This is the existing behavior before Python 3.12.

Also test absl-py on Python 3.12.

PiperOrigin-RevId: 565190992
copybara-service Bot pushed a commit to abseil/abseil-py that referenced this pull request Sep 15, 2023
@yilei @copybara-github
absltest: do not fail tests on Python 3.12+ when no tests ran and:
609aa55 
- Either test filtering is used
- Or sharding is used and the shard index > 0, i.e. only the first shard will fail when no tests ran on Python 3.12+.

Context: Python 3.12 unittest will now fail when no tests ran after the change from python/cpython#102051. Since `absltest` is built on top of `unittest`, it will follow this behavior change in Python 3.12.

However, when test filtering is used in `absltest`, often used via `bazel test --test_filter=<my_filter>`, the current user expectation is the `bazel test` command should NOT fail is at least one test ran. Since the test runner here has no visibility of the overall `bazel` invocation, we'll make the test not fail when test filtering is used via bazel's environment variable. This is the existing behavior before Python 3.12.

Also test absl-py on Python 3.12.

PiperOrigin-RevId: 565190992
copybara-service Bot pushed a commit to abseil/abseil-py that referenced this pull request Sep 15, 2023
@yilei @copybara-github
absltest: do not fail tests on Python 3.12+ when no tests ran and:
1edf2ab 
- Either test filtering is used
- Or sharding is used and the shard index > 0, i.e. only the first shard will fail when no tests ran on Python 3.12+.

Context: Python 3.12 unittest will now fail when no tests ran after the change from python/cpython#102051. Since `absltest` is built on top of `unittest`, it will follow this behavior change in Python 3.12.

However, when test filtering is used in `absltest`, often used via `bazel test --test_filter=<my_filter>`, the current user expectation is the `bazel test` command should NOT fail is at least one test ran. Since the test runner here has no visibility of the overall `bazel` invocation, we'll make the test not fail when test filtering is used via bazel's environment variable. This is the existing behavior before Python 3.12.

Also test absl-py on Python 3.12.

PiperOrigin-RevId: 565771935
@zware zware mentioned this pull request Jan 3, 2024
3.12 starts to return an error when all tests in a test file are skipped #113661
Closed
@vstinner

vstinner commented Jan 8, 2024

Copy link
Copy Markdown
Member

The new Python 3.12 message surprises users when all tests are skipped: see issue #113661.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@fishy

fishy commented Jan 8, 2024

Copy link
Copy Markdown

The PR description says:

It is common for test runner misconfiguration to fail to find any tests

Which seems to suggest that it should not fail test if all tests are skipped, as skipped tests are still "found", just intentionally skipped from running.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@stefanor

stefanor commented Jan 9, 2024

Copy link
Copy Markdown
Contributor Author

Which seems to suggest that it should not fail test if all tests are skipped, as skipped tests are still "found", just intentionally skipped from running.

Yeah, I'd agree with that.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@cottsay cottsay mentioned this pull request Nov 25, 2024
NO TESTS RAN failure on Noble for an ament_python package colcon/colcon-core#678
Open
texhnolyze added a commit to bit-bots/bitbots_main that referenced this pull request Dec 12, 2024
@texhnolyze
test(colcon): configure our pkgs to use pytest
ef65537 
to ensure that the CI `colcon test` run works, because with a change to
python 3.12 the `unittest` standard library used by default with colcon
now exits with an error code of 5 for an empty test suite.

See: colcon/colcon-core#678
See: python/cpython#102051
texhnolyze added a commit to bit-bots/udp_bridge that referenced this pull request Dec 16, 2024
@texhnolyze
test(colcon): configure our pkgs to use pytest
d74e356 
to ensure that the CI `colcon test` run works, because with a change to
python 3.12 the `unittest` standard library used by default with colcon
now exits with an error code of 5 for an empty test suite.

Additionally removed old ros 1 integration tests.

See: colcon/colcon-core#678
See: python/cpython#102051
@texhnolyze texhnolyze mentioned this pull request Dec 16, 2024
Move to pytest testing bit-bots/udp_bridge#39
Merged
3 of 6 tasks
texhnolyze added a commit to bit-bots/dynamic_stack_decider that referenced this pull request Dec 16, 2024
@texhnolyze
test(colcon): configure our pkgs to use pytest
d42a46a 
to ensure that the CI `colcon test` run works, because with a change to
python 3.12 the `unittest` standard library used by default with colcon
now exits with an error code of 5 for an empty test suite.

See: colcon/colcon-core#678
See: python/cpython#102051
@texhnolyze texhnolyze mentioned this pull request Dec 16, 2024
Configure pytest testing for vizualization bit-bots/dynamic_stack_decider#111
Merged
3 of 6 tasks
texhnolyze added a commit to bit-bots/udp_bridge that referenced this pull request Dec 16, 2024
@texhnolyze
test(colcon): configure our pkgs to use pytest
87adf72 
to ensure that the CI `colcon test` run works, because with a change to
python 3.12 the `unittest` standard library used by default with colcon
now exits with an error code of 5 for an empty test suite.

Additionally removed old ros 1 integration tests.

See: colcon/colcon-core#678
See: python/cpython#102051
mrg123 added a commit to mrg123/nanojev-mlx that referenced this pull request Sep 19, 2026
@mrg123
Harden CI against a silent zero-test pass
ae1ea82 
Python 3.10 is in the test matrix, and on 3.10 unittest still exits 0 when it
discovers no tests at all -- exit code 5 for "NO TESTS RAN" only landed in 3.12
(python/cpython#102051). So if tests/ were ever renamed or moved, the 3.10 leg
would report success while running nothing at all.

Capture the runner output and fail explicitly when zero tests were collected.
Once tests are actually discovered the new branch cannot trigger.

Verified both paths locally under `bash -eo pipefail`, the shell GitHub Actions
uses: a simulated zero-test run (exit 0, "Ran 0 tests") is rejected with exit 1,
and a normal 19-test run passes through untouched.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gpshead gpshead gpshead approved these changes
@hugovk hugovk Awaiting requested review from hugovk
@ezio-melotti ezio-melotti Awaiting requested review from ezio-melotti

Uh oh!

There was an error while loading. Please reload this page.

Assignees

@gpshead gpshead

Labels

3.12 only security fixes tests Tests in the Lib/test dir type-feature A feature request or enhancement

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

running a suite with no tests is not an error

Uh oh!

There was an error while loading. Please reload this page.

9 participants

@stefanor @terryjreedy @vstinner @fishy @gpshead @hugovk @bedevere-bot @hauntsaninja @AlexWaygood
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.