GeezerVisonThe World Wide Web, restored to 800 pixels of glory. | ||||||||||||||||||||||||||||||
| Home|About|Gaming cabinet|Archives|How it works|Starter sites | ||||||||||||||||||||||||||||||
GeezerVison is showing github.com — Pre-commit performance issues in our large repository · Issue #3603 · pre-commit/pre-commit Closed Descriptionsearch you tried in the issue trackeris:issue state:open rev-list describe your issueSummaryWe are experiencing two significant performance bottlenecks:
Issue 1:
|
| Command | Time | Notes |
|---|---|---|
git rev-list --quiet HEAD |
5,201ms | Current - walks ~1M commits |
git rev-parse --verify --quiet HEAD |
2.6ms | Fix - just validates ref |
Speedup: 2000x
def _rev_exists(rev: str) -> bool: return not subprocess.call( ('git', 'rev-parse', '--verify', '--quiet', rev), stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, )
Safety verification:
remote_sha != Z40 and _rev_exists(remote_sha)_get_diff() in run.py (PRE-COMMIT)def _get_diff() -> bytes: _, out, _ = cmd_output_b( 'git', 'diff', '--no-ext-diff', '--no-textconv', '--ignore-submodules', check=False, ) return out
Called after every hook that runs to detect if files were modified. Scans entire working tree.
| Scope | Time | Speedup |
|---|---|---|
| Full working tree | 250ms | 1x |
| Scoped to hook files | 16ms | 15x |
With 50 hooks running: 50 calls × 250ms = 12.5s seconds overhead
Scope diff to the files the hook processed:
def _get_diff(files: tuple[str, ...] = ()) -> bytes: cmd = ['git', 'diff', '--no-ext-diff', '--no-textconv', '--ignore-submodules'] if files: cmd.extend(['--', *files]) _, out, _ = cmd_output_b(*cmd, check=False) return out
Important caveat: Only safe when pass_filenames: true (the default).
For hooks with pass_filenames: false, must fall back to full tree diff since they may modify files outside their trigger patterns (e.g., code generators).
In _run_single_hook():
# filenames from classifier are available before pass_filenames check filenames = tuple(classifier.filenames_for_hook(hook)) ... if not hook.pass_filenames: # Can't scope diff - hook may modify any file diff_after = _get_diff() else: # Safe to scope diff to the files hook received diff_after = _get_diff(filenames)
These scan the working tree and can't be easily optimized:
| Operation | Time | Location | Notes |
|---|---|---|---|
intent_to_add_files() |
257ms | git.py | Full tree scan required |
_unstaged_changes_cleared() |
265ms | staged_files_only.py | Full tree scan required |
Could benefit from git's fsmonitor?
# Test _rev_exists performance python3 -c " import time, subprocess for cmd, name in [ (['git', 'rev-list', '--quiet', 'HEAD'], 'current'), (['git', 'rev-parse', '--verify', '--quiet', 'HEAD'], 'fix'), ]: start = time.perf_counter() subprocess.call(cmd, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) print(f'{name}: {(time.perf_counter()-start)*1000:.1f}ms') " # Test _get_diff performance python3 -c " import time, subprocess for scope in ['', '-- path/to/file.py']: cmd = ['git', 'diff', '--no-ext-diff', '--no-textconv', '--ignore-submodules'] + scope.split() start = time.perf_counter() subprocess.run(cmd, capture_output=True) print(f'{scope or 'full tree'}: {(time.perf_counter()-start)*1000:.1f}ms') "
4.5.0
(sadly I can't share)No response
please don't submit AI generated issues
your "second" issue is also a duplicate -- please search!