Description

Abbyyan

opened on Dec 19, 2019

Issue body actions

I've used Gitpython to clone some repos but it hangs with a specified repo with size of 17G. I've create a Pool to do git clone using Gitpython. There is a large git repo and needs more time than others to clone. Each process do a clone work for one repo. The Pool i used as follows:

  multi_res = [p.apply_async(runfunc, args=(
            incl_info, project_root, skip_dirs,)) for incl_info in incl_infos]
    LogInfo('Waiting for all subprocesses done...')
    for i in range(len(incl_infos)):
        while not multi_res[i].ready():
            LogInfo("Downloading now")
            time.sleep(5)
    p.close()
    p.join()

It works perfectly in most case. But will often hangs in the largest repo. It's wired that when i just clone the repo individually, It works fine. So i wonder if there is some block in python multiprocessing Pool at first.

I've strace the hanged git clone process . The git process output as follows:

Process 27649 attached
read(6, 0x7ffc36dae050, 4)              = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
--- SIGALRM {si_signo=SIGALRM, si_code=SI_KERNEL, si_value={int=2895997, ptr=0x2c307d}} ---
rt_sigreturn()                          = 0
read(6, 0x7ffc36dae050, 4)              = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
--- SIGALRM {si_signo=SIGALRM, si_code=SI_KERNEL, si_value={int=2895997, ptr=0x2c307d}} ---
rt_sigreturn()                          = 0
read(6, 0x7ffc36dae050, 4)              = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
--- SIGALRM {si_signo=SIGALRM, si_code=SI_KERNEL, si_value={int=2895997, ptr=0x2c307d}} ---
rt_sigreturn()                          = 0
read(6, 0x7ffc36dae050, 4)              = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
--- SIGALRM {si_signo=SIGALRM, si_code=SI_KERNEL, si_value={int=2895997, ptr=0x2c307d}} ---
rt_sigreturn()   

The git-lfs output as follows:

Process 28006 attached
[ Process PID=28006 runs in 32 bit mode. ]
futex(0x88b982c, FUTEX_WAIT_PRIVATE, 0, NULL

But when i replace the git.repo.clone_from with shell script git clone in a new subprocess, it works fine. So maybe there are some block in git.repo.clone_from, and i wonder whether it's solved. Thanks a lot.

Activity

1. Byron

   added

   Q&A

   on Dec 29, 2019
2. Byron commented on Dec 29, 2019

   

   Byron

   on Dec 29, 2019

   Member

   More actions

   Thanks for the detailed investigation! If memory serves, the way GitPython handles progress reporting on long-running clones can be prone to hanging. Even though it was thought to be fixed, apparently there is still a chance of it failing.

   The workaround proposed here is certainly preferred over using GitPython at all, since it's doing what's needed much more directly. GitPython in the end just spawns a git process itself and fails to properly handle it's output on long-running process.

   I am closing this issue as I don't think the underlying cause can clearly be determined or fixed, and due to the presence of a viable workaround. Please feel free to keep commenting here in case you would propose a different way of handling this - your opinion would be greatly appreciated.
3. Byron

   closed this as completedon Dec 29, 2019
4. Abbyyan commented on Jan 7, 2020

   

   Abbyyan

   on Jan 7, 2020

   Author

   More actions

   Thanks a lot. I enter the same problem today using checkout function and I wonder if there is a timeout when Gitpython start a new subprocess?
5. Byron commented on Feb 8, 2020

   

   Byron

   on Feb 8, 2020

   Member

   More actions

   Unfortunately no, GitPython is inherently synchronous and blocking. Everyone is invited to have a look at the code, communicating to a subprocess should not ever hang :/.
6. Byron

   reopened this on Feb 26, 2021
7. Byron

   closed this as completedon Feb 26, 2021
8. gitpython-developers

   locked and limited conversation to collaborators on Feb 26, 2021

Issue actions