GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — bpo-43950: Add option to opt-out of PEP-657 by ammaraskar · Pull Request #27023 · python/cpython

Conversation

@ammaraskar

@ammaraskar ammaraskar commented Jul 4, 2021 •
edited by bedevere-bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

To be merged after #26958

This adds the opt-out option and disables the compiler's column information and traceback printing.

https://bugs.python.org/issue43950

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@the-knights-who-say-ni the-knights-who-say-ni added the CLA signed label Jul 4, 2021
@bedevere-bot bedevere-bot added the awaiting review label Jul 4, 2021
@ammaraskar ammaraskar added the skip news label Jul 4, 2021
@ammaraskar
ammaraskar force-pushed the pep657-config branch 8 times, most recently from db3cbe3 to 6b3c6d7 Compare July 5, 2021 05:33
ammaraskar
ammaraskar commented Jul 5, 2021
View reviewed changes
Comment thread Lib/test/test_zipimport.py Resolved

Uh oh!

There was an error while loading. Please reload this page.
ammaraskar
ammaraskar commented Jul 5, 2021
View reviewed changes
Comment thread Objects/codeobject.c Resolved

Uh oh!

There was an error while loading. Please reload this page.
@ammaraskar
ammaraskar force-pushed the pep657-config branch 2 times, most recently from 1a555a1 to 0268bfe Compare July 5, 2021 06:06
isidentical
isidentical reviewed Jul 5, 2021
View reviewed changes
Comment thread Lib/test/test_code.py Resolved

Uh oh!

There was an error while loading. Please reload this page.
@ammaraskar ammaraskar mentioned this pull request Jul 5, 2021
bpo-43950: Print columns in tracebacks (PEP 657) #26958
Merged
pablogsal
pablogsal reviewed Jul 5, 2021
View reviewed changes
Comment thread Python/sysmodule.c Resolved

Uh oh!

There was an error while loading. Please reload this page.
@pablogsal

pablogsal commented Jul 5, 2021

Copy link
Copy Markdown
Member

Btw, do we have a test that checks that the flag works?
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@ammaraskar

ammaraskar commented Jul 5, 2021

Copy link
Copy Markdown
Member Author

Btw, do we have a test that checks that the flag works?

Not yet :)
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@ammaraskar

ammaraskar commented Jul 5, 2021 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member Author

@pablogsal After making these changes, I am a little worried about the sys.flags.no_debug_ranges conditionals, I fear that without it being automatically tested somewhere we could end up with code-rot in the test suite. Would it be possible to run a buildbot with -X no_debug_ranges just to ensure that no bugs in the opted-out code paths pop up.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@pablogsal

pablogsal commented Jul 5, 2021

Copy link
Copy Markdown
Member

Would it be possible to run a buildbot with -X no_debug_ranges just to ensure that no bugs in the opted-out code paths pop up.

Hmmm, is possible, but I would prefer to explicitly test that the flag + env work in some specific tests using a subprocess and that would only left the skip() as conditional. Is true that that could break, but adding an entire buildbot is a bit of a big deal, although we can explore that.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@ammaraskar
ammaraskar force-pushed the pep657-config branch from ed65e7e to d789c9a Compare July 5, 2021 19:10
@ammaraskar

ammaraskar commented Jul 5, 2021

Copy link
Copy Markdown
Member Author

Btw, do we have a test that checks that the flag works?

Just added some, please take a look.

Hmmm, is possible, but I would prefer to explicitly test that the flag + env work in some specific tests using a subprocess

For example running test_traceback.py with the flag set? That would certainly work too, I like that approach.

The last remaining component for this PR is to make marshal.c replace the tables with None if the config option is set and test that.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@pablogsal

pablogsal commented Jul 5, 2021

Copy link
Copy Markdown
Member

The last remaining component for this PR is to make marshal.c replace the tables with None if the config option is set and test that.

Maybe we should do the logic in the code object constructor and that way we don't need to deal with marshal.c and the compiler.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@ammaraskar

ammaraskar commented Jul 5, 2021 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member Author

Maybe we should do the logic in the code object constructor and that way we don't need to deal with marshal.c and the compiler.

Great idea, did this in the common code path in codeobject.c :)
That makes the code a lot simpler.

Added tests to make sure de-marshaling with the flag results in None for the endlinetable and columntable.

Hmmm, is possible, but I would prefer to explicitly test that the flag + env work in some specific tests using a subprocess

Going back to this, do you know if there is an example of such a test that already exists in the suite? Where should it go? A new test_traceback_no_debug_range.py or similar?
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@ammaraskar
ammaraskar marked this pull request as ready for review July 5, 2021 21:42
@ammaraskar
ammaraskar requested review from brettcannon, encukou and ericsnowcurrently as code owners July 5, 2021 21:42
isidentical
isidentical approved these changes Jul 6, 2021
View reviewed changes
@bedevere-bot bedevere-bot added awaiting merge and removed awaiting review labels Jul 6, 2021
@pablogsal

pablogsal commented Jul 7, 2021

Copy link
Copy Markdown
Member

We need more rebasing :(
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@ammaraskar @pablogsal @isidentical
bpo-43950: Add option to opt-out of PEP-657
d34f7a9 
Co-authored-by: Pablo Galindo <[email protected]>
Co-authored-by: Batuhan Taskaya <[email protected]>
Co-authored-by: Ammar Askar <[email protected]>
@ammaraskar
ammaraskar force-pushed the pep657-config branch from caa72dd to d34f7a9 Compare July 7, 2021 14:11
@ammaraskar

ammaraskar commented Jul 7, 2021

Copy link
Copy Markdown
Member Author

Rebased!
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@ammaraskar ammaraskar added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Jul 7, 2021
@bedevere-bot

bedevere-bot commented Jul 7, 2021

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @ammaraskar for commit d34f7a9 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Jul 7, 2021
@ammaraskar

ammaraskar commented Jul 7, 2021

Copy link
Copy Markdown
Member Author

buildbot/AMD64 Ubuntu Shared PR has a weird failure...

/srv/buildbot/buildarea/pull_request.bolen-ubuntu/build/python: error while loading shared libraries: libpython3.11d.so.1.0: cannot open shared object file: No such file or directory

I wonder if we need a @requires_* on it or if it's just a buildbot problem?
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@ammaraskar

ammaraskar commented Jul 7, 2021

Copy link
Copy Markdown
Member Author

Nope, no @requires annotation is needed to use assert_python_ok and assert_python_failure. This seems to just be a problem with the buildbot.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@ammaraskar

ammaraskar commented Jul 7, 2021

Copy link
Copy Markdown
Member Author

@pablogsal all lights are green, anything else for this one?
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
pablogsal
pablogsal approved these changes Jul 7, 2021
View reviewed changes
@pablogsal
pablogsal merged commit 4823d9a into python:main Jul 7, 2021
@bedevere-bot bedevere-bot removed the awaiting merge label Jul 7, 2021
@pablogsal

pablogsal commented Jul 7, 2021

Copy link
Copy Markdown
Member

@pablogsal all lights are green, anything else for this one?

Nop. Great job! 👌
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-bot

bedevere-bot commented Jul 7, 2021

Copy link
Copy Markdown

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 FreeBSD Shared 3.x has failed when building commit 4823d9a.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/483/builds/469) and take a look at the build logs.
  4. Check if the failure is related to this commit (4823d9a) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/483/builds/469

Failed tests:

  • test_marshal
  • test_traceback
  • test_code

Failed subtests:

  • test_endline_and_columntable_none_when_no_debug_ranges_env - test.test_code.CodeTest
  • test_endline_and_columntable_none_when_no_debug_ranges - test.test_code.CodeTest
  • test_no_caret_with_no_debug_ranges_flag - test.test_traceback.TracebackCases
  • test_no_columntable_and_endlinetable_with_no_debug_ranges - test.test_marshal.CodeTestCase
  • test_no_caret_with_no_debug_ranges_flag_python_traceback - test.test_traceback.TracebackCases

Summary of the results of the build (if available):

== Tests result: FAILURE then FAILURE ==

404 tests OK.

10 slowest tests:

  • test_smtpnet: 6 min 6 sec
  • test_nntplib: 3 min 45 sec
  • test_concurrent_futures: 3 min 33 sec
  • test_multiprocessing_spawn: 2 min 2 sec
  • test_multiprocessing_forkserver: 2 min 2 sec
  • test_unparse: 2 min 2 sec
  • test_urllib2: 2 min 1 sec
  • test_tokenize: 1 min 42 sec
  • test_multiprocessing_fork: 1 min 34 sec
  • test_asyncio: 1 min 23 sec

3 tests failed:
test_code test_marshal test_traceback

20 tests skipped:
test_dbm_gnu test_devpoll test_epoll test_idle test_ioctl
test_msilib test_ossaudiodev test_readline test_spwd
test_startfile test_tcl test_tix test_tk test_ttk_guionly
test_ttk_textonly test_turtle test_winconsoleio test_winreg
test_winsound test_zipfile64

3 re-run tests:
test_code test_marshal test_traceback

Total duration: 14 min 59 sec

Click to see traceback logs 
TracebackErrorCaretTests) ... ok


TracebackCases)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/test_traceback.py", line 110, in test_no_caret_with_no_debug_ranges_flag_python_traceback
    _, _, stderr = assert_python_ok(
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/support/script_helper.py", line 160, in assert_python_ok
    return _assert_python(True, *args, **env_vars)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/support/script_helper.py", line 145, in _assert_python
    res.fail(cmd_line)
    ^^^^^^^^^^^^^^^^^^
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/support/script_helper.py", line 72, in fail
    raise AssertionError("Process return code is %d\n"
AssertionError: Process return code is 1
command line: ['/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/python', '-X', 'faulthandler', '-X', 'no_debug_ranges', '@test_77265_tmpæ']


TracebackCases)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/test_traceback.py", line 110, in test_no_caret_with_no_debug_ranges_flag_python_traceback
    _, _, stderr = assert_python_ok(
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/support/script_helper.py", line 160, in assert_python_ok
    return _assert_python(True, *args, **env_vars)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/support/script_helper.py", line 145, in _assert_python
    res.fail(cmd_line)
    ^^^^^^^^^^^^^^^^^^
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/support/script_helper.py", line 72, in fail
    raise AssertionError("Process return code is %d\n"
AssertionError: Process return code is 1
command line: ['/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/python', '-X', 'faulthandler', '-X', 'no_debug_ranges', '@test_80109_tmpæ']


Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/test_marshal.py", line 150, in test_no_columntable_and_endlinetable_with_no_debug_ranges
    assert_python_ok('-X', 'no_debug_ranges',
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/support/script_helper.py", line 160, in assert_python_ok
    return _assert_python(True, *args, **env_vars)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/support/script_helper.py", line 145, in _assert_python
    res.fail(cmd_line)
    ^^^^^^^^^^^^^^^^^^
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/support/script_helper.py", line 72, in fail
    raise AssertionError("Process return code is %d\n"
AssertionError: Process return code is 1
command line: ['/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/python', '-X', 'faulthandler', '-X', 'no_debug_ranges', '-c', "\nimport sys\nimport marshal\nwith open(sys.argv[1], 'rb') as f:\n    co = marshal.load(f)\n\n    assert co.co_endlinetable is None\n    assert co.co_columntable is None\n", '@test_77265_tmpæ']


TracebackFormatTests) ... ok


TracebackCases) ... ok


Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/test_code.py", line 398, in test_endline_and_columntable_none_when_no_debug_ranges_env
    assert_python_ok('-c', code, PYTHONNODEBUGRANGES='1', __cleanenv=True)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/support/script_helper.py", line 160, in assert_python_ok
    return _assert_python(True, *args, **env_vars)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/support/script_helper.py", line 145, in _assert_python
    res.fail(cmd_line)
    ^^^^^^^^^^^^^^^^^^
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/support/script_helper.py", line 72, in fail
    raise AssertionError("Process return code is %d\n"
AssertionError: Process return code is 1
command line: ['/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/python', '-X', 'faulthandler', '-c', '\ndef f():\n    pass\n\nassert f.__code__.co_endlinetable is None\nassert f.__code__.co_columntable is None\n']


TracebackErrorLocationCaretTests) ... ok


TracebackException) ... ok


TracebackCases) ... FAIL


Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/test_code.py", line 387, in test_endline_and_columntable_none_when_no_debug_ranges
    assert_python_ok('-X', 'no_debug_ranges', '-c', code, __cleanenv=True)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/support/script_helper.py", line 160, in assert_python_ok
    return _assert_python(True, *args, **env_vars)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/support/script_helper.py", line 145, in _assert_python
    res.fail(cmd_line)
    ^^^^^^^^^^^^^^^^^^
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/support/script_helper.py", line 72, in fail
    raise AssertionError("Process return code is %d\n"
AssertionError: Process return code is 1
command line: ['/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/python', '-X', 'faulthandler', '-X', 'no_debug_ranges', '-c', '\ndef f():\n    pass\n\nassert f.__code__.co_endlinetable is None\nassert f.__code__.co_columntable is None\n']


TracebackCases)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/test_traceback.py", line 90, in test_no_caret_with_no_debug_ranges_flag
    self.assertEqual(len(lines), 4)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 1 != 4


Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/test_marshal.py", line 150, in test_no_columntable_and_endlinetable_with_no_debug_ranges
    assert_python_ok('-X', 'no_debug_ranges',
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/support/script_helper.py", line 160, in assert_python_ok
    return _assert_python(True, *args, **env_vars)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/support/script_helper.py", line 145, in _assert_python
    res.fail(cmd_line)
    ^^^^^^^^^^^^^^^^^^
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/support/script_helper.py", line 72, in fail
    raise AssertionError("Process return code is %d\n"
AssertionError: Process return code is 1
command line: ['/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/python', '-X', 'faulthandler', '-X', 'no_debug_ranges', '-c', "\nimport sys\nimport marshal\nwith open(sys.argv[1], 'rb') as f:\n    co = marshal.load(f)\n\n    assert co.co_endlinetable is None\n    assert co.co_columntable is None\n", '@test_82013_tmpæ']
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-bot

bedevere-bot commented Jul 7, 2021

Copy link
Copy Markdown

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Ubuntu Shared 3.x has failed when building commit 4823d9a.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/506/builds/502) and take a look at the build logs.
  4. Check if the failure is related to this commit (4823d9a) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/506/builds/502

Failed tests:

  • test_marshal
  • test_traceback
  • test_code

Failed subtests:

  • test_endline_and_columntable_none_when_no_debug_ranges_env - test.test_code.CodeTest
  • test_endline_and_columntable_none_when_no_debug_ranges - test.test_code.CodeTest
  • test_no_caret_with_no_debug_ranges_flag - test.test_traceback.TracebackCases
  • test_no_columntable_and_endlinetable_with_no_debug_ranges - test.test_marshal.CodeTestCase
  • test_no_caret_with_no_debug_ranges_flag_python_traceback - test.test_traceback.TracebackCases

Summary of the results of the build (if available):

== Tests result: FAILURE then FAILURE ==

407 tests OK.

10 slowest tests:

  • test_concurrent_futures: 4 min 6 sec
  • test_gdb: 3 min 30 sec
  • test_peg_generator: 3 min 6 sec
  • test_capi: 2 min 42 sec
  • test_multiprocessing_spawn: 2 min 34 sec
  • test_asyncio: 1 min 56 sec
  • test_unparse: 1 min 48 sec
  • test_lib2to3: 1 min 33 sec
  • test_multiprocessing_forkserver: 1 min 28 sec
  • test_tokenize: 1 min 16 sec

3 tests failed:
test_code test_marshal test_traceback

17 tests skipped:
test_devpoll test_idle test_ioctl test_kqueue test_msilib
test_ossaudiodev test_startfile test_tcl test_tix test_tk
test_ttk_guionly test_ttk_textonly test_turtle test_winconsoleio
test_winreg test_winsound test_zipfile64

3 re-run tests:
test_code test_marshal test_traceback

Total duration: 26 min 47 sec

Click to see traceback logs 
TracebackErrorCaretTests) ... ok


Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test_marshal.py", line 150, in test_no_columntable_and_endlinetable_with_no_debug_ranges
    assert_python_ok('-X', 'no_debug_ranges',
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/support/script_helper.py", line 160, in assert_python_ok
    return _assert_python(True, *args, **env_vars)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/support/script_helper.py", line 145, in _assert_python
    res.fail(cmd_line)
    ^^^^^^^^^^^^^^^^^^
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/support/script_helper.py", line 72, in fail
    raise AssertionError("Process return code is %d\n"
AssertionError: Process return code is 127
command line: ['/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/python', '-X', 'faulthandler', '-X', 'no_debug_ranges', '-c', "\nimport sys\nimport marshal\nwith open(sys.argv[1], 'rb') as f:\n    co = marshal.load(f)\n\n    assert co.co_endlinetable is None\n    assert co.co_columntable is None\n", '@test_15164_tmpæ']


TracebackCases)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test_traceback.py", line 110, in test_no_caret_with_no_debug_ranges_flag_python_traceback
    _, _, stderr = assert_python_ok(
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/support/script_helper.py", line 160, in assert_python_ok
    return _assert_python(True, *args, **env_vars)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/support/script_helper.py", line 145, in _assert_python
    res.fail(cmd_line)
    ^^^^^^^^^^^^^^^^^^
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/support/script_helper.py", line 72, in fail
    raise AssertionError("Process return code is %d\n"
AssertionError: Process return code is 127
command line: ['/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/python', '-X', 'faulthandler', '-X', 'no_debug_ranges', '@test_15685_tmpæ']


TracebackCases)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test_traceback.py", line 90, in test_no_caret_with_no_debug_ranges_flag
    self.assertEqual(len(lines), 4)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 1 != 4


Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test_marshal.py", line 150, in test_no_columntable_and_endlinetable_with_no_debug_ranges
    assert_python_ok('-X', 'no_debug_ranges',
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/support/script_helper.py", line 160, in assert_python_ok
    return _assert_python(True, *args, **env_vars)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/support/script_helper.py", line 145, in _assert_python
    res.fail(cmd_line)
    ^^^^^^^^^^^^^^^^^^
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/support/script_helper.py", line 72, in fail
    raise AssertionError("Process return code is %d\n"
AssertionError: Process return code is 127
command line: ['/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/python', '-X', 'faulthandler', '-X', 'no_debug_ranges', '-c', "\nimport sys\nimport marshal\nwith open(sys.argv[1], 'rb') as f:\n    co = marshal.load(f)\n\n    assert co.co_endlinetable is None\n    assert co.co_columntable is None\n", '@test_29458_tmpæ']


TracebackFormatTests) ... ok


TracebackCases) ... ok


TracebackCases)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test_traceback.py", line 110, in test_no_caret_with_no_debug_ranges_flag_python_traceback
    _, _, stderr = assert_python_ok(
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/support/script_helper.py", line 160, in assert_python_ok
    return _assert_python(True, *args, **env_vars)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/support/script_helper.py", line 145, in _assert_python
    res.fail(cmd_line)
    ^^^^^^^^^^^^^^^^^^
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/support/script_helper.py", line 72, in fail
    raise AssertionError("Process return code is %d\n"
AssertionError: Process return code is 127
command line: ['/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/python', '-X', 'faulthandler', '-X', 'no_debug_ranges', '@test_29458_tmpæ']


Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test_code.py", line 398, in test_endline_and_columntable_none_when_no_debug_ranges_env
    assert_python_ok('-c', code, PYTHONNODEBUGRANGES='1', __cleanenv=True)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/support/script_helper.py", line 160, in assert_python_ok
    return _assert_python(True, *args, **env_vars)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/support/script_helper.py", line 145, in _assert_python
    res.fail(cmd_line)
    ^^^^^^^^^^^^^^^^^^
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/support/script_helper.py", line 72, in fail
    raise AssertionError("Process return code is %d\n"
AssertionError: Process return code is 127
command line: ['/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/python', '-X', 'faulthandler', '-c', '\ndef f():\n    pass\n\nassert f.__code__.co_endlinetable is None\nassert f.__code__.co_columntable is None\n']


TracebackErrorLocationCaretTests) ... ok


TracebackException) ... ok


TracebackCases) ... FAIL


Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test_code.py", line 387, in test_endline_and_columntable_none_when_no_debug_ranges
    assert_python_ok('-X', 'no_debug_ranges', '-c', code, __cleanenv=True)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/support/script_helper.py", line 160, in assert_python_ok
    return _assert_python(True, *args, **env_vars)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/support/script_helper.py", line 145, in _assert_python
    res.fail(cmd_line)
    ^^^^^^^^^^^^^^^^^^
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/support/script_helper.py", line 72, in fail
    raise AssertionError("Process return code is %d\n"
AssertionError: Process return code is 127
command line: ['/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/python', '-X', 'faulthandler', '-X', 'no_debug_ranges', '-c', '\ndef f():\n    pass\n\nassert f.__code__.co_endlinetable is None\nassert f.__code__.co_columntable is None\n']
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@pablogsal

pablogsal commented Jul 7, 2021

Copy link
Copy Markdown
Member

@ammaraskar @isidentical Hummm, have we bumped the magic number? This buildbot failure seems quite related. I'm in some meetings right now and cannot check but may need to revert:(
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@isidentical

isidentical commented Jul 7, 2021

Copy link
Copy Markdown
Member

AFAIK we did. Checking the failures now...
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@ammaraskar

ammaraskar commented Jul 7, 2021

Copy link
Copy Markdown
Member Author

The errors on both are:

/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/python: error while loading shared libraries: libpython3.11d.so.1.0: cannot open shared object file: No such file or directory

stderr:
---
ld-elf.so.1: Shared object "libpython3.11d.so.1.0" not found, required by "python"
---

when running assert_python_ok and assert_python_failure and both buildbots have "Shared" in the name.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@ammaraskar

ammaraskar commented Jul 7, 2021

Copy link
Copy Markdown
Member Author

The confusing thing is, it only happens with the new tests, existing assert_python_ok tests are working just fine.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@isidentical

isidentical commented Jul 7, 2021

Copy link
Copy Markdown
Member

To reproduce locally;

$ git clean -fxd
$ ./configure --with-pydebug --enable-shared -C
$ make -j
$ LD_LIBRARY_PATH=$PWD ./python -m test -v test_traceback
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@isidentical

isidentical commented Jul 7, 2021

Copy link
Copy Markdown
Member

We don't preserve the original environment, so shared builds fail. See here:

cpython/Lib/test/support/script_helper.py

Lines 117 to 121 in bb3e0c2

else:
# Need to preserve the original environment, for in-place testing of
# shared library builds.
env = os.environ.copy()

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@isidentical

isidentical commented Jul 7, 2021

Copy link
Copy Markdown
Member

I can confirm that setting __cleanenv to False fixes the issue
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@ammaraskar

ammaraskar commented Jul 7, 2021

Copy link
Copy Markdown
Member Author

Aah, let's remove __cleanenv in that case. I set it primarily because I thought it would avoid propagation of the testing Interpreter's config.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@isidentical

isidentical commented Jul 7, 2021

Copy link
Copy Markdown
Member

Also see this example usage of how it is used (it creates a new env based on the existing one) if you want to isolate these;

    def get_hash(self, repr_, seed=None):
        env = os.environ.copy()
        env['__cleanenv'] = True  # signal to assert_python not to do a copy
                                  # of os.environ on its own
        if seed is not None:
            env['PYTHONHASHSEED'] = str(seed)
        else:
            env.pop('PYTHONHASHSEED', None)
        out = assert_python_ok(
            '-c', self.get_hash_command(repr_),
            **env)
        stdout = out[1].strip()
        return int(stdout)
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@ammaraskar ammaraskar mentioned this pull request Jul 7, 2021
Remove __cleanenv from PEP-657 tests #27060
Merged
@pablogsal pablogsal mentioned this pull request Apr 13, 2022
Include column offsets for bytecode instructions #88116
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pablogsal pablogsal pablogsal approved these changes
@isidentical isidentical isidentical approved these changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

skip news

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

5 participants

@ammaraskar @pablogsal @bedevere-bot @isidentical @the-knights-who-say-ni
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.