GeezerVison

The World Wide Web, restored to 800 pixels of glory.

GeezerVison is showing github.com — gh-127221: Add colour to unittest output by hugovk · Pull Request #127223 · python/cpython

Conversation

hugovk commented Nov 24, 2024 •

edited by bedevere-app Bot

Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link

Copy Markdown

Member

Use the existing _colorize module already used for tracebacks and doctest so output can be controlled with the PYTHON_COLORS, NO_COLOR and FORCE_COLOR environment variables.

This PR does a couple of things to add colour:

Enable colour for tracebacks by passing in colorize to msgLines = list(tb_e.format(colorize=can_colorize()))
Add colour for pass/fail/errors, using pytest as a guide
Plus some unrelated tidy-up (use more specific test methods, new-style objects, underscore variables)

Tested with some small demo scripts:

No tests ran

run-unittests-none.py

import unittest

if __name__ == "__main__":
    unittest.main()

Before	After	pytest

Passed

run-unittests-pass.py

Mode	Before	After	pytest
Regular
Verbose

Failing

run-unittests.py

import unittest


class TestThings(unittest.TestCase):
    def test_pass(self):
        self.assertTrue(True)

    def test_fail(self):
        self.assertTrue(False)

    def test_error(self):
        raise Exception("Manually raised exception")

    def test_pass2(self):
        self.assertEqual(1, 1)

    @unittest.skip("demonstrating skipping")
    def test_skip(self):
        self.fail("shouldn't happen")


class ExpectedFailureTestCase(unittest.TestCase):
    @unittest.expectedFailure
    def test_fail(self):
        self.assertEqual(1, 0, "broken")

    @unittest.expectedFailure
    def test_pass(self):
        self.assertEqual(1, 1, "not broken")


if __name__ == "__main__":
    unittest.main()

Mode	Before	After	pytest
Regular
Verbose

Subtests

run-unittests-subtests.py

import unittest


class DoSubTest(unittest.TestCase):
    def test_subtest(self):
        """Test with subtest"""
        for i in range(0, 4):
            with self.subTest(i=i):
                if i == 2:
                    raise Exception("Manually raised exception")
                self.assertEqual(i % 2, 0)


if __name__ == "__main__":
    unittest.main()

Mode	Before	After	pytest
Regular
Verbose

📚 Documentation preview 📚: https://cpython-previews--127223.org.readthedocs.build/

Issue: Add colour to unittest output #127221

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

tomasr8, nineteendo, AlexWaygood, cmaloney, and scastlara reacted with thumbs up emoji

hroncok and ambv reacted with heart emoji

All reactions

👍 5 reactions
❤️ 2 reactions

hugovk added 3 commits

November 24, 2024 14:59


          Add colour to unittest output

8cd1932


          Refactor repeated RST into epilog


          Refactor: use underscores, add missing spaces

d01272f

hugovk added type-feature

A feature request or enhancement

stdlib Standard Library Python modules in the Lib/ directory labels

Nov 24, 2024

bedevere-app Bot added the awaiting core review label

Nov 24, 2024

hugovk changed the title ~~Add colour to unittest output~~ gh-127221: Add colour to unittest output

Nov 24, 2024

bedevere-app Bot mentioned this pull request

Nov 24, 2024

Add colour to unittest output #127221

Closed

nineteendo commented Nov 24, 2024

Copy link

Copy Markdown

Contributor

Looks good, I see you're taking some inspiration from pytest, but I had to zoom in all the way to be able to read the text. :)

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

hugovk commented Nov 24, 2024

Copy link

Copy Markdown

Member Author

You can click the images to open them full size.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

nineteendo commented Nov 24, 2024 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link

Copy Markdown

Contributor

Well, I wanted to compare the images so I had to zoom in to see the difference. Maybe we could only show the comparison between pytest? The uncoloured images don't add much value.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

AlexWaygood reviewed

Nov 24, 2024

View reviewed changes

Comment thread

Lib/test/test_unittest/support.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

AlexWaygood reviewed

Nov 24, 2024

View reviewed changes

Comment thread

Lib/test/test_unittest/test_discovery.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

AlexWaygood reviewed

Nov 24, 2024

View reviewed changes

AlexWaygood left a comment

Copy link

Copy Markdown

Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is great!! With the coloured tracebacks as well, though, I sort-of feel like there might be a bit too much colour now when tests fail. It feels slightly overwhelming!

What about making only the "ERROR"/"FAIL"/"UNEXPECTED SUCCESS" lines coloured red, and not the ======= line above or the ------- below?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

sobolevn reacted with thumbs up emoji

All reactions

👍 1 reaction

hugovk force-pushed the 3.14-unittest-colour branch from e18da3e to d01272f Compare

November 24, 2024 16:32

AlexWaygood reviewed

Nov 24, 2024

View reviewed changes

Comment thread

Lib/test/test_unittest/test_runner.py

		self.assertTrue(test.doCleanups())
		self.assertEqual(cleanups, [(2, (), {}), (1, (1, 2, 3), dict(four='hello', five='goodbye'))])

		@force_not_colorized

AlexWaygood Nov 24, 2024

Copy link

Copy Markdown

Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we have to apply this decorator to so many methods that I almost wonder if it's worth having a custom metaclass that automatically adds it to each method on the class... or we could not use the @force_not_colorized decorator, and instead duplicate the logic in setUp and tearDown methods on the TestCleanup class here.

Neither feels ideal; it might be that what you have now is in fact best!

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

saakshaat-singh reacted with thumbs up emoji

All reactions

👍 1 reaction


          Don't colour the separators

06f1318

hugovk commented Nov 24, 2024

Copy link

Copy Markdown

Member Author

This is great!! With the coloured tracebacks as well, though, I sort-of feel like there might be a bit too much colour now when tests fail. It feels slightly overwhelming!

What about making only the "ERROR"/"FAIL"/"UNEXPECTED SUCCESS" lines coloured red, and not the ======= line above or the ------- below?

Thanks! Here's without coloured separators:

And verbose:

What do you think?

Also these demo scripts are a bit contrived; hopefully most of the time, most of your tests pass and just one or two fail. But then again, sometimes a lot do fail!

StanFromIreland reacted with thumbs up emoji

All reactions

👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

AlexWaygood commented Nov 24, 2024

Copy link

Copy Markdown

Member

Nice, that looks great to me now!

hugovk reacted with thumbs up emoji

All reactions

👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

nineteendo commented Nov 24, 2024 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link

Copy Markdown

Contributor

How about swapping the rows and columns in the table? It makes the text better readable:

show table

Mode	Regular	Verbose
Before
After
pytest

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

AlexWaygood commented Nov 24, 2024

Copy link

Copy Markdown

Member

How about swapping the rows and columns in the table? It makes the text better readable:

To be clear @nineteendo, this is a comment on the table in Hugo's PR description rather than the code he's changing in this PR, correct?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

nineteendo commented Nov 24, 2024

Copy link

Copy Markdown

Contributor

Correct, with 3 images on the same row I can't read the text without zooming in.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

AlexWaygood commented Nov 24, 2024

Copy link

Copy Markdown

Member

I see.

@hugovk, do you think we should add a link to the using-on-controlling-color somewhere in the unittest docs? I see we didn't when we added color support to doctest, but maybe we should have done

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.


          Mention colour control in docs

5a1de7c

hugovk commented Nov 24, 2024

Copy link

Copy Markdown

Member Author

@hugovk, do you think we should add a link to the using-on-controlling-color somewhere in the unittest docs? I see we didn't when we added color support to doctest, but maybe we should have done

Yeah, good idea. Something like 5a1de7c?

Let's also add it to doctest and traceback.

AlexWaygood reacted with heart emoji

All reactions

❤️ 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

AlexWaygood approved these changes

Nov 24, 2024

View reviewed changes

AlexWaygood left a comment

Copy link

Copy Markdown

Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

Comment thread

Doc/library/unittest.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

bedevere-app Bot added awaiting merge and removed awaiting core review labels

Nov 24, 2024

sobolevn reviewed

Nov 24, 2024

View reviewed changes

sobolevn left a comment

Copy link

Copy Markdown

Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you! I like the idea. But, I personally feel like it is still a bit too colorful. Right now almost nothing is white.

One more idea: can you please test this with the white theme as well?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions


          Mention colour control in docs for doctest, traceback and unittest

d2b0e29

hugovk commented Nov 24, 2024

Copy link

Copy Markdown

Member Author

Thank you! I like the idea. But, I personally feel like it is still a bit too colorful. Right now almost nothing is white.

Almost nothing is white because these demo scripts are intentionally crafted to generate lots of failures :) It's very similar to pytest.

One more idea: can you please test this with the white theme as well?

Sure:

unittest	pytest

Note I've not adjusted my local colours for this light theme. If I was using it normally, I may have adjusted it already for pytest.

nineteendo, sobolevn, scastlara, and Eclips4 reacted with heart emoji

All reactions

❤️ 4 reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

Eclips4 reviewed

Nov 24, 2024

View reviewed changes

Comment thread

Misc/NEWS.d/next/Library/2024-11-23-00-17-29.gh-issue-127221.OSXdFE.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.


          Link to unittest module

7226b77

Co-authored-by: Kirill Podoprigora <[email protected]>

hugovk merged commit 23f2e8f into python:main

Dec 5, 2024

hugovk deleted the 3.14-unittest-colour branch

December 5, 2024 19:10

bedevere-app Bot removed the awaiting merge label

Dec 5, 2024

hugovk mentioned this pull request

Dec 12, 2024

gh-127873: Only check sys.flags.ignore_environment for PYTHON* env vars #127877

Merged

srinivasreddy pushed a commit to srinivasreddy/cpython that referenced this pull request

Jan 8, 2025


          pythongh-127221: Add colour to unittest output (python#127223)

7dc6aa0

Co-authored-by: Kirill Podoprigora <[email protected]>

hugovk mentioned this pull request

Jan 22, 2025

Add colour to GitHub Actions output #129196

Merged

atrigent mentioned this pull request

Oct 18, 2026

The traceback module does not colorize by default, contrary to its documentation #159661

Closed

Reviewers

sobolevn

sobolevn left review comments

AlexWaygood

AlexWaygood approved these changes

Eclips4

Eclips4 left review comments

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

stdlib

Standard Library Python modules in the Lib/ directory

type-feature

A feature request or enhancement

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

5 participants

Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.

GeezerVison

Conversation

hugovk commented Nov 24, 2024 • edited by bedevere-app Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

No tests ran

Passed

Failing

Subtests

Uh oh!

nineteendo commented Nov 24, 2024

Uh oh!

hugovk commented Nov 24, 2024

Uh oh!

nineteendo commented Nov 24, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

AlexWaygood left a comment

Choose a reason for hiding this comment

Uh oh!

AlexWaygood Nov 24, 2024

Choose a reason for hiding this comment

Uh oh!

hugovk commented Nov 24, 2024

Uh oh!

AlexWaygood commented Nov 24, 2024

Uh oh!

nineteendo commented Nov 24, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AlexWaygood commented Nov 24, 2024

Uh oh!

nineteendo commented Nov 24, 2024

Uh oh!

AlexWaygood commented Nov 24, 2024

Uh oh!

hugovk commented Nov 24, 2024

Uh oh!

AlexWaygood left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sobolevn left a comment

Choose a reason for hiding this comment

Uh oh!

hugovk commented Nov 24, 2024

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

hugovk commented Nov 24, 2024 •

edited by bedevere-app Bot

Loading

Uh oh!

There was an error while loading. Please reload this page.

nineteendo commented Nov 24, 2024 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.

nineteendo commented Nov 24, 2024 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.