GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — Pre-commit performance issues in our large repository · Issue #3603 · pre-commit/pre-commit

Pre-commit performance issues in our large repository #3603

New issue
Copy link
New issue
Copy link
Closed
Closed
Pre-commit performance issues in our large repository#3603
Copy link

Description

@ptarjan
ptarjan
opened on Dec 26, 2025
Issue body actions

search you tried in the issue tracker

is:issue state:open rev-list

describe your issue

Summary

We are experiencing two significant performance bottlenecks:

  1. _rev_exists() (pre-push): Uses git rev-list --quiet which walks the entire commit history (~5.2s)
  2. _get_diff() (pre-commit): Scans the entire working tree after each hook (~250ms × N hooks)

Issue 1: _rev_exists() in hook_impl.py (PRE-PUSH)

Problem

def _rev_exists(rev: str) -> bool:
    return not subprocess.call(('git', 'rev-list', '--quiet', rev))

git rev-list --quiet walks the entire commit history to verify a revision exists.

Profiling

Command Time Notes
git rev-list --quiet HEAD 5,201ms Current - walks ~1M commits
git rev-parse --verify --quiet HEAD 2.6ms Fix - just validates ref

Speedup: 2000x

Verified Fix

def _rev_exists(rev: str) -> bool:
    return not subprocess.call(
        ('git', 'rev-parse', '--verify', '--quiet', rev),
        stdout=subprocess.DEVNULL,
        stderr=subprocess.DEVNULL,
    )

Safety verification:

  • Z40 (null SHA) case is already guarded: remote_sha != Z40 and _rev_exists(remote_sha)
  • Exit codes: both return 0 for valid refs, non-zero for invalid
  • Tested with: valid refs, invalid refs, commit SHAs

Issue 2: _get_diff() in run.py (PRE-COMMIT)

Problem

def _get_diff() -> bytes:
    _, out, _ = cmd_output_b(
        'git', 'diff', '--no-ext-diff', '--no-textconv', '--ignore-submodules',
        check=False,
    )
    return out

Called after every hook that runs to detect if files were modified. Scans entire working tree.

Profiling

Scope Time Speedup
Full working tree 250ms 1x
Scoped to hook files 16ms 15x

With 50 hooks running: 50 calls × 250ms = 12.5s seconds overhead

Verified Fix

Scope diff to the files the hook processed:

def _get_diff(files: tuple[str, ...] = ()) -> bytes:
    cmd = ['git', 'diff', '--no-ext-diff', '--no-textconv', '--ignore-submodules']
    if files:
        cmd.extend(['--', *files])
    _, out, _ = cmd_output_b(*cmd, check=False)
    return out

Important caveat: Only safe when pass_filenames: true (the default).

For hooks with pass_filenames: false, must fall back to full tree diff since they may modify files outside their trigger patterns (e.g., code generators).

In _run_single_hook():

# filenames from classifier are available before pass_filenames check
filenames = tuple(classifier.filenames_for_hook(hook))
...
if not hook.pass_filenames:
    # Can't scope diff - hook may modify any file
    diff_after = _get_diff()
else:
    # Safe to scope diff to the files hook received
    diff_after = _get_diff(filenames)

Other Slow Operations (No Easy Fix)

These scan the working tree and can't be easily optimized:

Operation Time Location Notes
intent_to_add_files() 257ms git.py Full tree scan required
_unstaged_changes_cleared() 265ms staged_files_only.py Full tree scan required

Could benefit from git's fsmonitor?


Reproduction

# Test _rev_exists performance
python3 -c "
import time, subprocess
for cmd, name in [
    (['git', 'rev-list', '--quiet', 'HEAD'], 'current'),
    (['git', 'rev-parse', '--verify', '--quiet', 'HEAD'], 'fix'),
]:
    start = time.perf_counter()
    subprocess.call(cmd, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
    print(f'{name}: {(time.perf_counter()-start)*1000:.1f}ms')
"

# Test _get_diff performance
python3 -c "
import time, subprocess
for scope in ['', '-- path/to/file.py']:
    cmd = ['git', 'diff', '--no-ext-diff', '--no-textconv', '--ignore-submodules'] + scope.split()
    start = time.perf_counter()
    subprocess.run(cmd, capture_output=True)
    print(f'{scope or 'full tree'}: {(time.perf_counter()-start)*1000:.1f}ms')
"

pre-commit --version

4.5.0

.pre-commit-config.yaml

(sadly I can't share)

~/.cache/pre-commit/pre-commit.log (if present)

No response

Activity

  1. ptarjan
    changed the title [-]Pre-commit Performance Issues in Our Large Repository[/-] [+]Pre-commit performance issues in our large repository[/+] on Dec 26, 2025
  2. asottile commented on Dec 26, 2025

    @asottile
    asottile
    on Dec 26, 2025
    Member
    More actions

    please don't submit AI generated issues

  3. asottile
    closed this as completedon Dec 26, 2025
  4. asottile commented on Dec 26, 2025

    @asottile
    asottile
    on Dec 26, 2025
    Member
    More actions

    your "second" issue is also a duplicate -- please search!

  5. pre-commit
    locked as off topic and limited conversation to collaborators on Dec 26, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions

      You are visitor 01402503 — GeezerVison does not store the pages you look up.
      Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.