Description

AlexWaygood

opened on May 21, 2023

Issue body actions

With a fresh CPython build (be0c106), IDLE is unable to open any .py files.

To reproduce:

1. Create an empty .py file with the name repro.py
2. Run python -m idlelib repro.py

IDLE still seems able to create new .py files and save them; it just can't open pre-existing .py files right now.

Traceback observed

C:\Users\alexw\coding\cpython>python -m idlelib repro.py
Running Debug|x64 interpreter...
Traceback (most recent call last):
  File "C:\Users\alexw\coding\cpython\Lib\runpy.py", line 198, in _run_module_as_main
    return _run_code(code, main_globals, None,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\alexw\coding\cpython\Lib\runpy.py", line 88, in _run_code
    exec(code, run_globals)
  File "C:\Users\alexw\coding\cpython\Lib\idlelib\__main__.py", line 7, in <module>
    idlelib.pyshell.main()
  File "C:\Users\alexw\coding\cpython\Lib\idlelib\pyshell.py", line 1640, in main
    if flist.open(filename) is None:
       ^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\alexw\coding\cpython\Lib\idlelib\filelist.py", line 37, in open
    edit = self.EditorWindow(self, filename, key)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\alexw\coding\cpython\Lib\idlelib\pyshell.py", line 135, in __init__
    EditorWindow.__init__(self, *args)
  File "C:\Users\alexw\coding\cpython\Lib\idlelib\editor.py", line 289, in __init__
    self.set_indentation_params(is_py_src)
  File "C:\Users\alexw\coding\cpython\Lib\idlelib\editor.py", line 1327, in set_indentation_params
    i = self.guess_indent()
        ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\alexw\coding\cpython\Lib\idlelib\editor.py", line 1574, in guess_indent
    opener, indented = IndentSearcher(self.text, self.tabwidth).run()
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\alexw\coding\cpython\Lib\idlelib\editor.py", line 1646, in run
    save_tabsize = tokenize.tabsize
                   ^^^^^^^^^^^^^^^^
AttributeError: module 'tokenize' has no attribute 'tabsize'

Environment

(Given the cause of the bug, the environment details shouldn't really be relevant; but they're included here anyway, for completeness.)

Python 3.12.0a7+ (heads/main:be0c106789, May 21 2023, 12:00:27) [MSC v.1932 64 bit (AMD64)] on win32

Reproduces on a debug and non-debug build, FWIW.

Linked PRs

• gh-104719: Restore Tokenize module constants #104722
• gh-104719: IDLE - delete useless monkeypatch of tokenize #104726
• [3.11] gh-104719: IDLE - delete useless monkeypatch of tokenize (GH-104726) #104727
• gh-104719: IDLE - test existence of all tokenize references. #104767
• [3.12] gh-104719: IDLE - test existence of all tokenize references. (GH-104767) #104844
• [3.11] gh-104719: IDLE - test existence of all tokenize references. (GH-104767) #104845

Activity

1. AlexWaygood

   added

   type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

   topic-IDLE

   release-blocker

   on May 21, 2023
2. AlexWaygood

   assigned

   pablogsal

   and

   terryjreedy

   on May 21, 2023
3. ezio-melotti

   added this to Release and Deferred blockers 🚫on May 21, 2023
4. github-project-automation

   moved this to Todo in Release and Deferred blockers 🚫on May 21, 2023
5. AlexWaygood commented on May 21, 2023

   

   AlexWaygood

   on May 21, 2023

   MemberAuthor

   More actions

   Bisects to 6715f91 (unsurprisingly 😄)

   Cc. @mgmacias95, @pablogsal, @lysnikolaou, @isidentical
6. AlexWaygood

   added

   3.12only security fixesonly security fixes

   on May 21, 2023
7. pablogsal commented on May 21, 2023

   

   pablogsal

   on May 21, 2023

   Member

   More actions

   Will check this today

   Reacted by Alex Waygood
8. pablogsal commented on May 21, 2023

   

   pablogsal

   on May 21, 2023

   Member

   More actions

   Looks like IDLE is using a bunch of non documented attributes of the tokenize module that are no longer used internally. The easier course of action is probably just restore these but note that there will be no guarantee that whatever idle is going with them will continue working.

   I ran the buildbots and the CI before merge so another question is why CI didn't catch this.
9. AlexWaygood commented on May 21, 2023

   

   AlexWaygood

   on May 21, 2023

   MemberAuthor

   More actions

   I ran the buildbots and the CI before merge so another question is why CI didn't catch this.

   Do any of the buildbots run the test suite with -u gui (or -u all), I wonder? A bunch of IDLE's tests actually open and close GUI windows, so they're not run unless you run the test suite with that option.

   Reacted by Erlend E. Aasland
10. AlexWaygood commented on May 21, 2023

    

    AlexWaygood

    on May 21, 2023

    MemberAuthor

    More actions

    I ran the buildbots and the CI before merge so another question is why CI didn't catch this.

    Do any of the buildbots run the test suite with -u gui (or -u all), I wonder? A bunch of IDLE's tests actually open and close GUI windows, so they're not run unless you run the test suite with that option.

    That doesn't seem relevant; python -m test test_idle -u gui passes on main, on both debug and non-debug builds. So it does seem like there's some missing test coverage here.
11. pablogsal commented on May 21, 2023

    

    pablogsal

    on May 21, 2023

    Member

    More actions

    I checked a bit more how the constant that was eliminated is being used and seems that IDLE is monkey patching the value to modify the behaviour of the now deleted _tokenize private function to adapt something regarding tabsize.

    Unfortunately I don't think this is going to work and was out of contract anyway so I think the best course of action here is to eliminate that code from idlelib as restoring the constant will not restore whatever was happening before.
12. AlexWaygood commented on May 21, 2023

    

    AlexWaygood

    on May 21, 2023

    MemberAuthor

    More actions

    FWIW, I can find other places in prominent open-source projects where these undocumented constants (which have now been removed) are used:

    • https://github.com/google-research/google-research/blob/b7004fc7046e5eb27bfcc3da57d11e17f191b1a6/cubert/python_tokenizer.py#L60-L63
    • https://github.com/asottile/reorder-python-imports/blob/d3109c3717004fdeac2a59485103175a078da244/testing/generate-tokenize#L36-L37
    • https://github.com/mhammond/pywin32/blob/f6300f84dad8a165f45a65d7823a554fa3cc4a5a/Pythonwin/pywin/idle/AutoIndent.py#L530-L531

    So it might be worth adding them back anyway, to limit the disruption caused by this change.
13. 33 remaining items
14. terryjreedy

    added this to IDLE Issueson May 22, 2023
15. terryjreedy

    added a commit that references this issue on May 22, 2023

    pythongh-104719: IDLE - test editor.IndentSearcher

    446ddc4
16. terryjreedy commented on May 22, 2023

    

    terryjreedy

    on May 22, 2023

    Member

    More actions

    I decided that it would in general be a good idea to have test coverage of references to non-idlelib modules. I have a toktest branch that adds coverage of the class containing the failure and all tokenize references in editor.py. I will make a PR tomorrow with or without coverage of tokenize references in 2 other idlelib modules.

    Reacted by Alex Waygood, Pablo Galindo Salgado and Marta Gómez Macías
17. Yhg1s commented on May 22, 2023

    

    Yhg1s

    on May 22, 2023

    Member

    More actions

    FWIW, whether to deprecate or document the unused globals isn't really a RM question, but considering the difficulty of emitting warnings for them, I'd rather not do that right now (but for 3.13 it's fine). Documenting them as for internal use only and/or deprecated can be done after b1.
18. terryjreedy

    moved this to In Progress in IDLE Issueson May 23, 2023
19. terryjreedy

    added a commit that references this issue on May 24, 2023

    gh-104719: IDLE - test existence of all tokenize references. (#104767)

    e561c09
20. miss-islington

    added 2 commits that reference this issue on May 24, 2023

    pythongh-104719: IDLE - test existence of all tokenize references. (p…

    1061d30

    pythongh-104719: IDLE - test existence of all tokenize references. (p…

    63336e2
21. terryjreedy

    added 2 commits that reference this issue on May 24, 2023

    [3.12] gh-104719: IDLE - test existence of all tokenize references. (G…

    5e1799e

    [3.11] gh-104719: IDLE - test existence of all tokenize references. (G…

    22618a8
22. terryjreedy commented on May 24, 2023

    

    terryjreedy

    on May 24, 2023

    Member

    More actions

    IDLE now has a regression test that will fail if any tokenizer names change. I suggest we close this and someone open a new issue about the now obsolete constants.

    Reacted by Alex Waygood
23. AlexWaygood commented on May 24, 2023

    

    AlexWaygood

    on May 24, 2023

    MemberAuthor

    More actions

    IDLE now has a regression test that will fail if any tokenizer names change. I suggest we close this and someone open a new issue about the now obsolete constants.

    That sounds good to me. I'll try to open an issue later today.

    Thanks @terryjreedy, @pablogsal and @mgmacias95!
24. AlexWaygood

    closed this as completedon May 24, 2023
25. github-project-automation

    moved this from In Progress to Done in Release and Deferred blockers 🚫on May 24, 2023
26. github-project-automation

    moved this from In Progress to Done in IDLE Issueson May 24, 2023

Issue actions