Description

johnslavik

opened on Dec 20, 2025

Issue body actions

Feature or enhancement

Proposal:

Remove the unused tokenize._compile function. EDIT: Deprecate the unused tokenize._compile function.

It very much looks like the internal tokenize._compile function hasn't been needed since GH-104323 when it was initially removed, but it was mechanically brought back to tokenize in GH-104722 to fix GH-104719.

From a quick search, it seems that, prior to GH-104323, tokenize._compile was only used internally in tokenize._tokenize.

Most of the fairly fresh code I skimmed through on GitHub operating on tokenize._compile properly checks if Python is version <3.10 / tokenize._compile exists.

Except CheetahTemplate3, so removing tokenize._compile immediately could break them -- we need to deprecate first.

In terms of benefits of removing this at some point: it wouldn't necessarily improve the import time of tokenize as long as re still imports functools to cache template compilation. Therefore, removing tokenize._compile in the future is just a small cleanup that yields no other benefits than smaller, less bloated code.

Has this already been discussed elsewhere?

This is a minor feature, which does not need previous discussion elsewhere

Links to previous discussion of this feature:

No response

Activity

1. johnslavik

   added

   type-featureA feature request or enhancementA feature request or enhancement

   on Dec 20, 2025
2. johnslavik commented on Dec 20, 2025

   

   johnslavik

   on Dec 20, 2025

   MemberAuthor

   More actions

   @ZeroIntensity stdlib topic-parser
   CC-ing @pablogsal @Yhg1s @lysnikolaou
3. StanFromIreland

   added

   stdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directory

   topic-parser

   on Dec 20, 2025
4. johnslavik commented on Dec 20, 2025

   

   johnslavik

   on Dec 20, 2025

   MemberAuthor

   More actions

   Talked with @ZeroIntensity, the general guideline is to avoid breaking things, so I'll rephrase this issue to instead propose deprecating first.
5. johnslavik

   changed the title [-]Remove the unused `tokenize._compile` function[/-] [+]Deprecate the unused `tokenize._compile` function[/+] on Dec 20, 2025
6. pablogsal commented on Dec 20, 2025

   

   pablogsal

   on Dec 20, 2025

   Member

   More actions

   I am not against it but I think is weird to deprecate a private function and the gain for maintainance is very little as we don't have any good reason for doing it right now (security, bugs...etc)
7. johnslavik commented on Dec 20, 2025

   

   johnslavik

   on Dec 20, 2025

   MemberAuthor

   More actions

   the gain for maintainance is very little as we don't have any good reason for doing it right now (security, bugs...etc)

   I see where you're coming from. Do you think it's better to wait for a good reason to deprecate or remove, instead of doing it right now, when there is no immediate benefit?
8. pablogsal commented on Dec 20, 2025

   

   pablogsal

   on Dec 20, 2025

   Member

   More actions

   the gain for maintainance is very little as we don't have any good reason for doing it right now (security, bugs...etc)

   I see where you're coming from. Do you think it's better to wait for a good reason to deprecate or remove, instead of doing it right now, when there is no immediate benefit?

   Pretty much. In general these kind of changes have more chances to annoy some users then to really improve the maintainance of the code. It's a bit unfortunate that people rely on private functions but at this stage there is no much obvious benefit and I just see risk

   Reacted by Bartosz Sławecki
9. johnslavik commented on Dec 20, 2025

   

   johnslavik

   on Dec 20, 2025

   MemberAuthor

   More actions

   I can see the problem. I'll close this -- I also think it's not worth the risk. Thanks @pablogsal!
10. johnslavik

    closed this as completedon Dec 20, 2025
11. johnslavik

    closed this as not plannedon Dec 20, 2025

Issue actions