GeezerVison

The World Wide Web, restored to 800 pixels of glory.

GeezerVison is showing github.com — tools: add more options to track flaky tests · nodejs/node@bfe0ac6

File tree

Expand file treeCollapse file tree

Open diff view settings

Filter options

.github/workflows
tools
- test.py

Expand file treeCollapse file tree

Open diff view settings

Collapse file

`‎.github/workflows/build-tarball.yml‎`

Copy file name to clipboardExpand all lines: .github/workflows/build-tarball.yml

+2-2Lines changed: 2 additions & 2 deletions

Original file line number	Diff line number	Diff line change
`@@ -29,7 +29,7 @@ concurrency:`
`29`	`29`
`30`	`30`	`env:`
`31`	`31`	`PYTHON_VERSION: '3.10'`
`32`		`- FLAKY_TESTS: dontcare`
	`32`	`+ FLAKY_TESTS: keep_retrying`
`33`	`33`
`34`	`34`	`permissions:`
`35`	`35`	`contents: read`
`@@ -94,4 +94,4 @@ jobs:`
`94`	`94`	`- name: Test`
`95`	`95`	`run: \|`
`96`	`96`	`cd $TAR_DIR`
`97`		`- make run-ci -j2 V=1 TEST_CI_ARGS="-p dots"`
	`97`	`+ make run-ci -j2 V=1 TEST_CI_ARGS="-p dots --measure-flakiness 9"`

Collapse file

`‎.github/workflows/build-windows.yml‎`

Copy file name to clipboardExpand all lines: .github/workflows/build-windows.yml

+1-1Lines changed: 1 addition & 1 deletion

Original file line number	Diff line number	Diff line change
`@@ -24,7 +24,7 @@ concurrency:`
`24`	`24`
`25`	`25`	`env:`
`26`	`26`	`PYTHON_VERSION: '3.10'`
`27`		`- FLAKY_TESTS: dontcare`
	`27`	`+ FLAKY_TESTS: keep_retrying`
`28`	`28`
`29`	`29`	`permissions:`
`30`	`30`	`contents: read`

Collapse file

`‎.github/workflows/coverage-linux.yml‎`

Copy file name to clipboardExpand all lines: .github/workflows/coverage-linux.yml

+2-2Lines changed: 2 additions & 2 deletions

Original file line number	Diff line number	Diff line change
`@@ -27,7 +27,7 @@ concurrency:`
`27`	`27`
`28`	`28`	`env:`
`29`	`29`	`PYTHON_VERSION: '3.10'`
`30`		`- FLAKY_TESTS: dontcare`
	`30`	`+ FLAKY_TESTS: keep_retrying`
`31`	`31`
`32`	`32`	`permissions:`
`33`	`33`	`contents: read`
`@@ -53,7 +53,7 @@ jobs:`
`53`	`53`	`# TODO(bcoe): fix the couple tests that fail with the inspector enabled.`
`54`	`54`	`# The cause is most likely coverage's use of the inspector.`
`55`	`55`	`- name: Test`
`56`		`- run: NODE_V8_COVERAGE=coverage/tmp make test-cov -j2 V=1 TEST_CI_ARGS="-p dots" \|\| exit 0`
	`56`	`+ run: NODE_V8_COVERAGE=coverage/tmp make test-cov -j2 V=1 TEST_CI_ARGS="-p dots --measure-flakiness 9" \|\| exit 0`
`57`	`57`	`- name: Report JS`
`58`	`58`	`run: npx c8 report --check-coverage`
`59`	`59`	`env:`

Collapse file

`‎.github/workflows/coverage-windows.yml‎`

Copy file name to clipboardExpand all lines: .github/workflows/coverage-windows.yml

+1-1Lines changed: 1 addition & 1 deletion

Original file line number	Diff line number	Diff line change
`@@ -29,7 +29,7 @@ concurrency:`
`29`	`29`
`30`	`30`	`env:`
`31`	`31`	`PYTHON_VERSION: '3.10'`
`32`		`- FLAKY_TESTS: dontcare`
	`32`	`+ FLAKY_TESTS: keep_retrying`
`33`	`33`
`34`	`34`	`permissions:`
`35`	`35`	`contents: read`

Collapse file

`‎.github/workflows/doc.yml‎`

Copy file name to clipboardExpand all lines: .github/workflows/doc.yml

+1-1Lines changed: 1 addition & 1 deletion

Original file line number	Diff line number	Diff line change
`@@ -40,4 +40,4 @@ jobs:`
`40`	`40`	`name: docs`
`41`	`41`	`path: out/doc`
`42`	`42`	`- name: Test`
`43`		`- run: NODE=$(command -v node) make test-doc-ci TEST_CI_ARGS="-p actions"`
	`43`	`+ run: NODE=$(command -v node) make test-doc-ci TEST_CI_ARGS="-p actions --measure-flakiness 9"`

Collapse file

`‎.github/workflows/test-asan.yml‎`

Copy file name to clipboardExpand all lines: .github/workflows/test-asan.yml

+2-2Lines changed: 2 additions & 2 deletions

Original file line number	Diff line number	Diff line change
`@@ -31,7 +31,7 @@ concurrency:`
`31`	`31`	`env:`
`32`	`32`	`ASAN_OPTIONS: intercept_tls_get_addr=0`
`33`	`33`	`PYTHON_VERSION: '3.10'`
`34`		`- FLAKY_TESTS: dontcare`
	`34`	`+ FLAKY_TESTS: keep_retrying`
`35`	`35`
`36`	`36`	`permissions:`
`37`	`37`	`contents: read`
`@@ -58,4 +58,4 @@ jobs:`
`58`	`58`	`- name: Build`
`59`	`59`	`run: make build-ci -j2 V=1`
`60`	`60`	`- name: Test`
`61`		`- run: make run-ci -j2 V=1 TEST_CI_ARGS="-p actions -t 300"`
	`61`	`+ run: make run-ci -j2 V=1 TEST_CI_ARGS="-p actions -t 300 --measure-flakiness 9"`

Collapse file

`‎.github/workflows/test-internet.yml‎`

Copy file name to clipboardExpand all lines: .github/workflows/test-internet.yml

+1-1Lines changed: 1 addition & 1 deletion

Original file line number	Diff line number	Diff line change
`@@ -22,7 +22,7 @@ concurrency:`
`22`	`22`
`23`	`23`	`env:`
`24`	`24`	`PYTHON_VERSION: '3.10'`
`25`		`- FLAKY_TESTS: dontcare`
	`25`	`+ FLAKY_TESTS: keep_retrying`
`26`	`26`
`27`	`27`	`permissions:`
`28`	`28`	`contents: read`

Collapse file

`‎.github/workflows/test-linux.yml‎`

Copy file name to clipboardExpand all lines: .github/workflows/test-linux.yml

+2-2Lines changed: 2 additions & 2 deletions

Original file line number	Diff line number	Diff line change
`@@ -24,7 +24,7 @@ concurrency:`
`24`	`24`
`25`	`25`	`env:`
`26`	`26`	`PYTHON_VERSION: '3.10'`
`27`		`- FLAKY_TESTS: dontcare`
	`27`	`+ FLAKY_TESTS: keep_retrying`
`28`	`28`
`29`	`29`	`permissions:`
`30`	`30`	`contents: read`
`@@ -46,4 +46,4 @@ jobs:`
`46`	`46`	`- name: Build`
`47`	`47`	`run: make build-ci -j2 V=1 CONFIG_FLAGS="--error-on-warn"`
`48`	`48`	`- name: Test`
`49`		`- run: make run-ci -j2 V=1 TEST_CI_ARGS="-p actions"`
	`49`	`+ run: make run-ci -j2 V=1 TEST_CI_ARGS="-p actions --measure-flakiness 9"`

Collapse file

`‎.github/workflows/test-macos.yml‎`

Copy file name to clipboardExpand all lines: .github/workflows/test-macos.yml

+2-2Lines changed: 2 additions & 2 deletions

Original file line number	Diff line number	Diff line change
`@@ -30,7 +30,7 @@ concurrency:`
`30`	`30`
`31`	`31`	`env:`
`32`	`32`	`PYTHON_VERSION: '3.10'`
`33`		`- FLAKY_TESTS: dontcare`
	`33`	`+ FLAKY_TESTS: keep_retrying`
`34`	`34`
`35`	`35`	`permissions:`
`36`	`36`	`contents: read`
`@@ -60,4 +60,4 @@ jobs:`
`60`	`60`	`- name: Build`
`61`	`61`	`run: make build-ci -j2 V=1 CONFIG_FLAGS="--error-on-warn"`
`62`	`62`	`- name: Test`
`63`		`- run: make run-ci -j2 V=1 TEST_CI_ARGS="-p actions"`
	`63`	`+ run: make run-ci -j2 V=1 TEST_CI_ARGS="-p actions --measure-flakiness 9"`

Collapse file

`‎tools/test.py‎`

Copy file name to clipboardExpand all lines: tools/test.py

+29-12Lines changed: 29 additions & 12 deletions

Original file line number	Diff line number	Diff line change
`@@ -96,10 +96,11 @@ def get_module(name, path):`
`96`	`96`
`97`	`97`	`class ProgressIndicator(object):`
`98`	`98`
`99`		`- def __init__(self, cases, flaky_tests_mode):`
	`99`	`+ def __init__(self, cases, flaky_tests_mode, measure_flakiness):`
`100`	`100`	`self.cases = cases`
`101`	`101`	`self.serial_id = 0`
`102`	`102`	`self.flaky_tests_mode = flaky_tests_mode`
	`103`	`+ self.measure_flakiness = measure_flakiness`
`103`	`104`	`self.parallel_queue = Queue(len(cases))`
`104`	`105`	`self.sequential_queue = Queue(len(cases))`
`105`	`106`	`for case in cases:`
`@@ -211,10 +212,22 @@ def RunSingle(self, parallel, thread_id):`
`211`	`212`	`if output.UnexpectedOutput():`
`212`	`213`	`if FLAKY in output.test.outcomes and self.flaky_tests_mode == DONTCARE:`
`213`	`214`	`self.flaky_failed.append(output)`
	`215`	`+ elif FLAKY in output.test.outcomes and self.flaky_tests_mode == KEEP_RETRYING:`
	`216`	`+ for _ in range(99):`
	`217`	`+ if not case.Run().UnexpectedOutput():`
	`218`	`+ self.flaky_failed.append(output)`
	`219`	`+ break`
	`220`	`+ else:`
	`221`	`+ # If after 100 tries, the test is not passing, it's not flaky.`
	`222`	`+ self.failed.append(output)`
`214`	`223`	`else:`
`215`	`224`	`self.failed.append(output)`
`216`	`225`	`if output.HasCrashed():`
`217`	`226`	`self.crashed += 1`
	`227`	`+ if self.measure_flakiness:`
	`228`	`+ outputs = [case.Run() for _ in range(self.measure_flakiness)]`
	`229`	`+ # +1s are there because the test already failed once at this point.`
	`230`	`+ print(f" failed {len([i for i in outputs if i.UnexpectedOutput()]) + 1} out of {self.measure_flakiness + 1}")`
`218`	`231`	`else:`
`219`	`232`	`self.succeeded += 1`
`220`	`233`	`self.remaining -= 1`
`@@ -436,8 +449,8 @@ def Done(self):`
`436`	`449`
`437`	`450`	`class CompactProgressIndicator(ProgressIndicator):`
`438`	`451`
`439`		`- def __init__(self, cases, flaky_tests_mode, templates):`
`440`		`- super(CompactProgressIndicator, self).__init__(cases, flaky_tests_mode)`
	`452`	`+ def __init__(self, cases, flaky_tests_mode, measure_flakiness, templates):`
	`453`	`+ super(CompactProgressIndicator, self).__init__(cases, flaky_tests_mode, measure_flakiness)`
`441`	`454`	`self.templates = templates`
`442`	`455`	`self.last_status_length = 0`
`443`	`456`	`self.start_time = time.time()`
`@@ -492,29 +505,29 @@ def PrintProgress(self, name):`
`492`	`505`
`493`	`506`	`class ColorProgressIndicator(CompactProgressIndicator):`
`494`	`507`
`495`		`- def __init__(self, cases, flaky_tests_mode):`
	`508`	`+ def __init__(self, cases, flaky_tests_mode, measure_flakiness):`
`496`	`509`	`templates = {`
`497`	`510`	`'status_line': "[%(mins)02i:%(secs)02i\|\033[34m%%%(remaining) 4d\033[0m\|\033[32m+%(passed) 4d\033[0m\|\033[31m-%(failed) 4d\033[0m]: %(test)s",`
`498`	`511`	`'stdout': "\033[1m%s\033[0m",`
`499`	`512`	`'stderr': "\033[31m%s\033[0m",`
`500`	`513`	`}`
`501`		`- super(ColorProgressIndicator, self).__init__(cases, flaky_tests_mode, templates)`
	`514`	`+ super(ColorProgressIndicator, self).__init__(cases, flaky_tests_mode, measure_flakiness, templates)`
`502`	`515`
`503`	`516`	`def ClearLine(self, last_line_length):`
`504`	`517`	`print("\033[1K\r", end='')`
`505`	`518`
`506`	`519`
`507`	`520`	`class MonochromeProgressIndicator(CompactProgressIndicator):`
`508`	`521`
`509`		`- def __init__(self, cases, flaky_tests_mode):`
	`522`	`+ def __init__(self, cases, flaky_tests_mode, measure_flakiness):`
`510`	`523`	`templates = {`
`511`	`524`	`'status_line': "[%(mins)02i:%(secs)02i\|%%%(remaining) 4d\|+%(passed) 4d\|-%(failed) 4d]: %(test)s",`
`512`	`525`	`'stdout': '%s',`
`513`	`526`	`'stderr': '%s',`
`514`	`527`	`'clear': lambda last_line_length: ("\r" + (" " * last_line_length) + "\r"),`
`515`	`528`	`'max_length': 78`
`516`	`529`	`}`
`517`		`- super(MonochromeProgressIndicator, self).__init__(cases, flaky_tests_mode, templates)`
	`530`	`+ super(MonochromeProgressIndicator, self).__init__(cases, flaky_tests_mode, measure_flakiness, templates)`
`518`	`531`
`519`	`532`	`def ClearLine(self, last_line_length):`
`520`	`533`	`print(("\r" + (" " * last_line_length) + "\r"), end='')`
`@@ -948,8 +961,8 @@ def GetTimeout(self, mode, section=''):`
`948`	`961`	`timeout = timeout * 6`
`949`	`962`	`return timeout`
`950`	`963`
`951`		`-def RunTestCases(cases_to_run, progress, tasks, flaky_tests_mode):`
`952`		`- progress = PROGRESS_INDICATORS[progress](cases_to_run, flaky_tests_mode)`
	`964`	`+def RunTestCases(cases_to_run, progress, tasks, flaky_tests_mode, measure_flakiness):`
	`965`	`+ progress = PROGRESS_INDICATORS[progress](cases_to_run, flaky_tests_mode, measure_flakiness)`
`953`	`966`	`return progress.Run(tasks)`
`954`	`967`
`955`	`968`	`# -------------------------------------------`
`@@ -967,6 +980,7 @@ def RunTestCases(cases_to_run, progress, tasks, flaky_tests_mode):`
`967`	`980`	`SLOW = 'slow'`
`968`	`981`	`FLAKY = 'flaky'`
`969`	`982`	`DONTCARE = 'dontcare'`
	`983`	`+KEEP_RETRYING = 'keep_retrying'`
`970`	`984`
`971`	`985`	`class Expression(object):`
`972`	`986`	`pass`
`@@ -1355,8 +1369,11 @@ def BuildOptions():`
`1355`	`1369`	`result.add_option("--cat", help="Print the source of the tests",`
`1356`	`1370`	`default=False, action="store_true")`
`1357`	`1371`	`result.add_option("--flaky-tests",`
`1358`		`- help="Regard tests marked as flaky (run\|skip\|dontcare)",`
	`1372`	`+ help="Regard tests marked as flaky (run\|skip\|dontcare\|keep_retrying)",`
`1359`	`1373`	`default="run")`
	`1374`	`+ result.add_option("--measure-flakiness",`
	`1375`	`+ help="When a test fails, re-run it x number of times",`
	`1376`	`+ default=0, type="int")`
`1360`	`1377`	`result.add_option("--skip-tests",`
`1361`	`1378`	`help="Tests that should not be executed (comma-separated)",`
`1362`	`1379`	`default="")`
`@@ -1433,7 +1450,7 @@ def ProcessOptions(options):`
`1433`	`1450`	`# -j and ignoring -J, which is the opposite of what we used to do before -J`
`1434`	`1451`	`# became a legacy no-op.`
`1435`	`1452`	`print('Warning: Legacy -J option is ignored. Using the -j option.')`
`1436`		`- if options.flaky_tests not in [RUN, SKIP, DONTCARE]:`
	`1453`	`+ if options.flaky_tests not in [RUN, SKIP, DONTCARE, KEEP_RETRYING]:`
`1437`	`1454`	`print("Unknown flaky-tests mode %s" % options.flaky_tests)`
`1438`	`1455`	`return False`
`1439`	`1456`	`return True`
`@@ -1733,7 +1750,7 @@ def should_keep(case):`
`1733`	`1750`	`else:`
`1734`	`1751`	`try:`
`1735`	`1752`	`start = time.time()`
`1736`		`- if RunTestCases(cases_to_run, options.progress, options.j, options.flaky_tests):`
	`1753`	`+ if RunTestCases(cases_to_run, options.progress, options.j, options.flaky_tests, options.measure_flakiness):`
`1737`	`1754`	`result = 0`
`1738`	`1755`	`else:`
`1739`	`1756`	`result = 1`

0 commit comments

Comments

0 (0)

You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.