GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — Fix Fetch progress bar by fvalette-ledger · Pull Request #1971 · gitpython-developers/GitPython

Conversation

@fvalette-ledger

@fvalette-ledger fvalette-ledger commented Oct 14, 2024

Copy link
Copy Markdown
Contributor

See #1969

stderr parser call RemoteProgress update on each line received. With universal_newlines set to False, there is a mixup between line feed and carriage return.
In the handle_process_output thread, this is thus seen as a single line for the whole output on each steps.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@fvalette-ledger
git,remote: use universal new lines for fetch/pull stderr capture
1bb4651
See gitpython-developers#1969

stderr parser call RemoteProgress update on each line received.
With universal_newlines set to False, there is a mixup between
line feed and carriage return.
In the `handle_process_output` thread, this is thus seen as a single
line for the whole output on each steps.
@Byron

Byron commented Oct 15, 2024

Copy link
Copy Markdown
Member

Thanks a lot for contributing this fix!

It appears that CI is legitimately failing though.

Could you investigate and see what changed between the last known working version, and the first broken one? Was it really this that changed, or maybe something else?

I am wondering if there could be a different fix for this, that would then also pass CI naturally.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@Byron Byron added the waiting for feedback label Oct 15, 2024
@fvalette-ledger

fvalette-ledger commented Oct 15, 2024

Copy link
Copy Markdown
Contributor Author

Thanks a lot for contributing this fix!

It appears that CI is legitimately failing though.

Could you investigate and see what changed between the last known working version, and the first broken one? Was it really this that changed, or maybe something else?

I am wondering if there could be a different fix for this, that would then also pass CI naturally.

Hi,
Indeed, it seems that a security fix for windows broke the progress bar.
At first sight, it seems that universal_newlines arguments does more than it said, clearly the fix will be more complex than this first try.
i'll have a look, until CI is green, i convert this PR to draft.

By the way, python 3.7 is supported by EOL and not package on ubuntu latest (i.e. ubuntu 24.04 ==> https://github.com/gitpython-developers/GitPython/actions/runs/11334079345/job/31534626048?pr=1971).

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@fvalette-ledger
fvalette-ledger marked this pull request as draft October 15, 2024 06:01
@Byron Byron removed the waiting for feedback label Oct 15, 2024
@Byron

Byron commented Oct 15, 2024

Copy link
Copy Markdown
Member

That's interesting, as it sounds like a python security update on Windows broke GitPython? Or only universal newlines?
In any case, I am looking forward to you tackling it, thanks again for your help.

By the way, python 3.7 is supported by EOL and not package on ubuntu latest (i.e. ubuntu 24.04 ==> https://github.com/gitpython-developers/GitPython/actions/runs/11334079345/job/31534626048?pr=1971).

That's true - thanks for pointing it out. Maybe you can remove testing it on CI while at it.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@fvalette-ledger

fvalette-ledger commented Oct 15, 2024 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor Author

I didn't bisect yet, but the regression is introduced between 3.1.40 and 3.1.41.
As said in the related issue, i'm using python 3.10 and ubuntu 22.04.1 and i tried different git version with the same result (from 2.30, 2.34.1 to last release with the result). Whatever the git version, the last working progress bar for fetch (and likely pull) is w/ GitPython 3.1.40

Concerning git/remote.py file, the diff only concern documentation fix (rewording, style, typo fixes). So, in the last working release use stderr as bytes and not text, convert in handle_process_output.
But there is major security fix for windows that, somehow, changes the behavior in handle_process_output.pump_stream where

GitPython/git/cmd.py

Line 152 in 49ca909

for line in stream:

yields only once on <LF> with all progress lines (sep w/ <CR>) concatenated and thus broke progress bar.

Byron reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@fvalette-ledger

fvalette-ledger commented Oct 15, 2024

Copy link
Copy Markdown
Contributor Author

That's true - thanks for pointing it out. Maybe you can remove testing it on CI while at it.

As python 3.7 is officially supported by the package, IMHO, it must be covered by unit testing.
Maybe use matrix configuration to select suitable ubuntu version (ubuntu-22.04 is a valid label, see https://github.com/actions/runner-images?tab=readme-ov-file#available-images).

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@Byron

Byron commented Oct 15, 2024

Copy link
Copy Markdown
Member

I didn't bisect yet, but the regression is introduced between 3.1.40 and 3.1.41.

I am definitely curious what it turns out to be in the end. As it seems, the issue is definitely within GitPython, and I'd hope the security-related change can also be made so that it doesn't break anything.

As python 3.7 is officially supported by the package, IMHO, it must be covered by unit testing.
Maybe use matrix configuration to select suitable ubuntu version (ubuntu-22.04 is a valid label, see https://github.com/actions/runner-images?tab=readme-ov-file#available-images).

That's a great point! I also think that using a specific version label should be the way to go, if it's still available.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@fvalette-ledger
git,cmd: add encoding arg to popen if universal newlines is True
52cceaf
@fvalette-ledger
fvalette-ledger marked this pull request as ready for review October 15, 2024 16:12
@fvalette-ledger

fvalette-ledger commented Oct 15, 2024

Copy link
Copy Markdown
Contributor Author

It appears that CI is legitimately failing though.

It was an encoding issue, while using universal_newlines, one may specify the right charset.
This is fixed by 52cceaf

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@fvalette-ledger

fvalette-ledger commented Oct 15, 2024 via email

Copy link
Copy Markdown
Contributor Author
FYI, encoding argument is missing for Popen. It should be defined if text output is required (i.e. w/ universal_newlines or text set to True). The last commit on the PR fixes Windows unit test. As windows does not use utf-8 as encoding (unless linux/macos), it's required. Security fix, using safe_popen, is not impacted by the fix, may be the author of this fix could review the pull request to be sure that nothing else is broken under the hood. Regards,
…
On Tue, Oct 15, 2024 at 2:02 PM Sebastian Thiel ***@***.***> wrote: I didn't bisect yet, but the regression is introduced between 3.1.40 and 3.1.41. I am definitely curious what it turns out to be in the end. As it seems, the issue is definitely within GitPython, and I'd hope the security-related change can also be made so that it doesn't break anything. As python 3.7 is officially supported by the package, IMHO, it must be covered by unit testing. Maybe use matrix configuration to select suitable ubuntu version ( ubuntu-22.04 is a valid label, see https://github.com/actions/runner-images?tab=readme-ov-file#available-images ). That's a great point! I also think that using a specific version label should be the way to go, if it's still available. — Reply to this email directly, view it on GitHub <#1971 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A2KVVAXMJTLB2WKBNEGMFN3Z3T77DAVCNFSM6AAAAABP5U6ITOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMJTG4ZDGNZSG4> . You are receiving this because you authored the thread.Message ID: ***@***.***>
-- [image: Ledger Logo]www.ledger.com 1 rue du Mail 75002 Paris France Florent VALETTE Senior Firmware Engineer ***@***.*** [image: Tw] <https://twitter.com/Ledger/>[image: Ln] <https://www.linkedin.com/company/ledgerhq/>[image: Fb] <https://github.com/LedgerHQ/>[image: Fb] <https://www.facebook.com/Ledger/>[image: Ig] <https://www.instagram.com/ledger/>
-- Les informations contenues dans ce message électronique ainsi que celles contenues dans les documents attachés sont strictement confidentielles et sont destinées à l'usage exclusif du (des) destinataire(s) nommé(s). Toute divulgation, distribution ou reproduction, même partielle, en est strictement interdite sauf autorisation écrite et expresse de l’émetteur. Si vous recevez ce message par erreur, veuillez le notifier immédiatement à son émetteur par retour, et le détruire ainsi que tous les documents qui y sont attachés. The information contained in this email and in any document enclosed is strictly confidential and is intended solely for the use of the individual or entity to which it is addressed. Partial or total disclosure, distribution or reproduction of its contents is strictly prohibited unless expressly approved in writing by the sender. If you have received this communication in error, please notify us immediately by responding to this email, and then delete the message and its attached files from your system.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@Byron
See if python 3.7 still works when using an older Ubuntu version.
d6cdb67
This should be undone once python 3.7 is EOL.
Byron
Byron approved these changes Oct 15, 2024
View reviewed changes

@Byron Byron left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot for helping with this!

I think this will always be prone to breaking as the interactivity of remote progress isn't tested, but let's hope it will work for a while.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@Byron
Byron merged commit a527224 into gitpython-developers:main Oct 15, 2024
@fvalette-ledger fvalette-ledger mentioned this pull request Jan 14, 2025
Progress bar on git pull outpost-os/barbican#14
Merged
fvalette-ledger added a commit to outpost-os/barbican that referenced this pull request Jan 14, 2025
@fvalette-ledger
Progress bar on git pull (#14)
a417f59
> Fix Fetch progress bar by @fvalette-ledger in
gitpython-developers/GitPython#1971

GitPython 3.1.44 got my fix merged, and thus, we can reintroduce
progress bar on git pull (barbican update)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Byron Byron Byron approved these changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

2 participants

@fvalette-ledger @Byron
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.