GeezerVison

I have a branch which adds a filtering logic to the completer (see https://discuss.python.org/t/repl-introduce-generic-filters-to-filter-auto-completion-matches/77553) and I wondered whether we could expand the Completer interface in order to have a more generic hook category. Both your work and mine can be orthogonal but we might take this opportunity to make the completer class more generic (I haven't looked at the PR in details as I'm travelling now)

Right, I remember seeing your dpo post before! I think it's a good idea :) With this PR I decided to target PyREPL since it is less stable than rlcompleter and thus it's easier to expand/make changes. Though we could definitely think about unifying the interface with that of rlcompleter. For now the API is a bit ad hoc so I'd welcome some discussion in that direction 🙂

1. On macOS, type import re<tab>:

>>> import re
[ complete but not unique ]

2. Press <tab> again:

>>> import re
re        reprlib   readline  resource  requests

3. Then change your mind and press ctrl+C, and you get:

>>> import re
KeyboardInterrupt   readline  resource  requests
>>>
re        reprlib   readline  resource  requests

I'd expect something more like:

>>> import re
KeyboardInterrupt
>>>

4. Press ctrl+C a couple more times:

>>> import re
KeyboardInterrupt   readline  resource  requests
>>>
KeyboardInterrupt   readline  resource  requests
>>>
KeyboardInterrupt   readline  resource  requests
>>>
re        reprlib   readline  resource  requests

I'd expect something more like:

>>> import re
KeyboardInterrupt
>>>
KeyboardInterrupt
>>>
KeyboardInterrupt
>>>

Answering here what I wrote on Discord - the ctrl+C behaviour is present for the existing name/attribute autocomplete as well, this PR is not changing that. I think it's something we could add in a separate PR though!

I only did a quick look at the code. Are you trying to import the module during completion?

I only did a quick look at the code. Are you trying to import the module during completion?

Yes I try to import it to ensure it's a valid package and so that I can easily list the submodules (e.g. import foo.<tab>). IPython does something similar. Is it a concern?

Yes I try to import it to ensure it's a valid package and so that I can easily list the submodules (e.g. import foo.<tab>). IPython does something similar. Is it a concern?

I think so. Implicitly import a module during auto-completion seems a bit unintentional to me. For example, importing torch could take 2~3 seconds, and to user it would be a freeze during completion. Some libraries monkeypatch stdlib or even Python during import, and that's some hidden side effect. Personally I don't like this implicit behavior. Also, will the import only triggered if . is in the text? Or for all possible completion? Say I do import t<tab> does it import all the modules starting with a t?

Say I do import t does it import all the modules starting with a t?

This PR uses pkgutil.iter_modules so just typing import t<tab> does not actually import anything. The results of pkgutil.iter_modules are also cached so the top-level packages are only searched once.

I do import in case you type import torch.x<tab>, but I think I might be able to avoid that.

What I think I cannot avoid is an import in case you type from foo import x<tab>. In order for me to know which objects are available for import, I need to actually import the module. I don't think there is a way around that? What do you think?

Personally, I would prefer if the auto-completer never imports any module without my knowledge, even if that means it can't do its job in some cases. It's okay if it completes when the module already imported, but importing a new module during auto-completion seems wrong to me. I think we should at least get opinions from a few other core devs about this, because in my mind this is a relatively large new behavior.

I think we should at least get opinions from a few other core devs about this, because in my mind this is a relatively large new behavior.

I admit I do not fully understand the implications of this so I think getting more opinions is a good idea :) Do you want me to start a discussion on Discord?

Discord or/and dpo maybe? We should at least get some opinions from repl owners.

ok! It's getting a bit late here, so I'll do it tomorrow 🙂

I updated the PR following the discussion on DPO: https://discuss.python.org/t/looking-for-feedback-on-adding-import-autocomplete-to-pyrepl/82281

• Modules are no longer imported. Modules are discovered using a combination of pkgutil.iter_modules and submodule_search_locations.
• Removed support for module attributes. This relied on actually importing the modules. We can add this once we have decided in which way we should discover module attributes.

@tomasr8 can you try to play a bit with some edge cases (tons of packages, etc) to see how the performance is for extreme circumstances? I want to be aware of the ways this can "break"

Thanks for taking the time to review this, I really appreciate it!

@tomasr8 can you try to play a bit with some edge cases (tons of packages, etc) to see how the performance is for extreme circumstances? I want to be aware of the ways this can "break"

When it comes to performance, the biggest bottleneck is the call to pkgutil.iter_modules() to find all top-level packages.
(the result is cached so we pay the cost only the first time you hit TAB).

Here are some timings for lots of packages. I installed the top 1000 pypi packages (at least those that can be installed on 3.14 so 836 packages installed):

~ pip freeze | wc -l
836

The timings are on a laptop with Intel ultra 9 185H CPU and a new-ish SSD (relevant because pkgutil.iter_modules interacts with the file system).

For those ~800 packages in a single location, pkgutil.iter_modules takes about 0.03s:

>>> import time, pkgutil
>>> start = time.time(); list(pkgutil.iter_modules()); time.time() - start
0.03072071075439453

Typing import <tab> takes about 0.03s as well so most of the cost is really inside pkgutil.iter_modules:

>>> import <tab>
0.03162884712219238

Subsequent completion requests are much faster since the results are cached:

>>> import <tab>
0.002469778060913086

Another thing I tried is multiple search locations (i.e. multiple sys.path entries):

5 different locations and ~4000 packages total:

`pkgutil.iter_modules`: 0.3535459041595459s
Initial `import <tab>`: 0.3592982292175293s
Subsequent `import <tab>`: 0.0021576881408691406s

10 different locations and ~8000 packages total:

`pkgutil.iter_modules`: 0.6836235523223877s
Initial `import <tab>`: 0.6652779579162598s
Subsequent `import <tab>`: 0.0002522468566894531s

The initial search is about 0.35s and 0.68s respectively, while the subsequent ones are again very cheap.

I still get this failure locally:

======================================================================
FAIL: test_import_completions (test.test_pyrepl.test_pyrepl.TestPyReplModuleCompleter.test_import_completions) (code='import path\t\n')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/hugo/github/python/cpython/main/Lib/test/test_pyrepl/test_pyrepl.py", line 938, in test_import_completions
    self.assertEqual(output, expected)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
AssertionError: 'import path' != 'import pathlib'
- import path
+ import pathlib
?            +++

It's because my env isn't a clean slate because I've pip installed some packages for testing on 3.14, and a dependency is interfering:

Python 3.14.0a7+ (heads/completer:f4e290a03f1, Apr 23 2025, 13:32:14) [Clang 16.0.0 (clang-1600.0.26.6)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import path<tab>
pathlib       pathvalidate

❯ ./python.exe -m pip freeze | grep path
pathvalidate==3.2.0

I don't think this needs to block initial merge before the freeze, but would be good to fix.

(PS there's a merge conflict on some imports.)

Weird, I tested this locally with some scripts on sys.path and it passed for me. Sorry about that, I'll take a look later today!

Ok so iter_modules also gets FileFinders from sys.path so I had to change the approach but it should work now. Tested locally with pathvalidate installed and the tests are passing.

Passing now, thanks!

LGTM, fantastic job @tomasr8