GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — Fix Git.execute shell use and reporting bugs by EliahKagan · Pull Request #1687 · gitpython-developers/GitPython

Conversation

@EliahKagan

@EliahKagan EliahKagan commented Oct 2, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

Fixes #1685
Fixes #1686

Changes to git module code

This updates the shell variable itself, only when it is None, from self.USE_SHELL.

(That attribute usually gets its value from Git.USE_SHELL rather than being separately an instance attribute. But self.USE_SHELL is an idiomatic way to access it. Furthermore, although people probably shouldn't set it on instances, people may expect that this is permitted.)

Now:

  • USE_SHELL is used as a fallback only. When shell=False is passed, USE_SHELL is no longer consulted. Thus shell=False always keeps a shell from being used, even in the non-default case where the USE_SHELL attribute has been set to True.
  • The debug message printed to the log shows the actual value that is being passed to Popen, because the updated shell variable is used both to produce that message and in the Popen call.

New tests

This also (actually, first) adds tests, some to check that the correct value is always passed as shell= when Git.execute calls Popen, and others to check that whatever it passes is the same as it claims to be passing in the log. I've made them two separate test methods rather than a test method with multiple assertions, because I think the results of these particular tests are easier to understand that way (even compared to the option of using self.subTest). But the major shared logic is extracted to a helper function, where its important subtleties are commented. I have verified that the tests fail, in the expected way, before the bugs are fixed, and then pass, and that these are both the case even on native Windows where we don't (yet) have CI jobs.

The tests cover the six combinations of valid values of the shell= argument to Git.execute (None, False, and True) and Git.USE_SHELL (False and True). I used ddt for this, so there are only two test methods, even though there are six test cases. The tests are simple in principle, with the main source of complexity being the question of how to deal with how the command argument to Popen typically differs, including by type, based on the shell argument, yet the tests must not misbehave, nor induce extraneous misbehavior, even when the code under test has a bug that would ordinarily prevent such agreement.

On Python 3.7 only, this adds mock as a development dependency (in the test extra, not as a regular dependency). mock is a backport of unittest.mock, and a couple of my assertions use a handy feature that was introduced to the mock library in Python 3.8. Besides being harder to write without it, I believe it would be less clearly expressed. (Details are commented with the import logic.)

While adding the new tests, I also renamed test_it_executes_git_to_shell_and_returns_result to test_it_executes_git_and_returns_result, because that test case does not test that a shell is used to run git (and a shell is not used in that test).

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
EliahKagan added 7 commits October 2, 2023 04:27
@EliahKagan
Fix the name of the "executes git" test
5e71c27
That test is not testing whether or not a shell is used (nor does
it need to test that), but just whether the library actually runs
git, passes an argument to it, and returns text from its stdout.
@EliahKagan
Test whether a shell is used
5944060
In the Popen calls in Git.execute, for all combinations of allowed
values for Git.USE_SHELL and the shell= keyword argument.
@EliahKagan
Test if whether a shell is used is logged
aa5e2f6
The log message shows "Popen(...)", not "execute(...)". So it
should faithfully report what is about to be passed to Popen in
cases where a user reading the log would otherwise be misled into
thinking this is what has happened.

Reporting the actual "shell=" argument passed to Popen is also more
useful to log than the argument passed to Git.execute (at least if
only one of them is to be logged).
@EliahKagan
Fix tests so they don't try to run "g"
0f19fb0
Both new shell-related tests were causing the code under test to
split "git" into ["g", "i", "t"] and thus causing the code under
test to attempt to execute a "g" command. This passes the command
as a one-element list of strings rather than as a string, which
avoids this on all operating systems regardless of whether the code
under test has a bug being tested for.

This would not occur on Windows, which does not iterate commands of
type str character-by-character even when the command is run
without a shell. But it did happen on other systems.

Most of the time, the benefit of using a command that actually runs
"git" rather than "g" is the avoidance of confusion in the error
message. But this is also important because it is possible for the
user who runs the tests to have a "g" command, in which case it
could be very inconvenient, or even unsafe, to run "g". This should
be avoided even when the code under test has a bug that causes a
shell to be used when it shouldn't or not used when it should, so
having separate commands (list and str) per test case parameters
would not be a sufficient solution: it would only guard against
running "g" when a bug in the code under test were absent.
@EliahKagan
Extract shared test logic to a helper
da3460c
This also helps mock Popen over a smaller scope, which may be
beneficial (especially if it is mocked in the subprocess module,
rather than the git.cmd module, in the future).
@EliahKagan
Use the mock backport on Python 3.7
41294d5
Because mock.call.kwargs, i.e. the ability to examine
m.call_args.kwargs where m is a Mock or MagicMock, was introduced
in Python 3.8.

Currently it is only in test/test_git.py that any use of mocks
requires this, so I've put the conditional import logic to import
mock (the top-level package) rather than unittest.mock only there.

The mock library is added as a development (testing) dependency
only when the Python version is lower than 3.8, so it is not
installed when not needed.

This fixes a problem in the new tests of whether a shell is used,
and reported as used, in the Popen call in Git.execute. Those
just-introduced tests need this, to be able to use
mock_popen.call_args.kwargs on Python 3.7.
@EliahKagan
Fix Git.execute shell use and reporting bugs
baf3457
This updates the shell variable itself, only when it is None, from
self.USE_SHELL.

(That attribute is usually Git.USE_SHELL rather than an instance
attribute. But although people probably shouldn't set it on
instances, people will expect that this is permitted.)

Now:

- USE_SHELL is used as a fallback only. When shell=False is passed,
  USE_SHELL is no longer consuled. Thus shell=False always keeps a
  shell from being used, even in the non-default case where the
  USE_SHELL attribue has been set to True.

- The debug message printed to the log shows the actual value that
  is being passed to Popen, because the updated shell variable is
  used both to produce that message and in the Popen call.
@EliahKagan
EliahKagan marked this pull request as ready for review October 2, 2023 09:25
This was referenced Oct 2, 2023
Git.execute treats shell=False the same as shell=None #1685
Closed
Git.execute debug message has misleading Popen(shell=...) #1686
Closed
@Byron Byron added this to the v3.1.38 - Bugfixes milestone Oct 2, 2023
Byron
Byron approved these changes Oct 2, 2023
View reviewed changes

@Byron Byron left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks so much for the fix which brings the behaviour in line with the documentation, and for adding regression protection.

This PR is also a wonderful example on what's possible with the unittest and mock libraries and generally what's possible with python testing.

🙏

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

EliahKagan reacted with thumbs up emoji
All reactions
  • 👍 1 reaction
@Byron
Byron merged commit 874f0bf into gitpython-developers:main Oct 2, 2023
@EliahKagan
EliahKagan deleted the popen-shell branch October 2, 2023 19:59
@EliahKagan EliahKagan mentioned this pull request Oct 3, 2023
Clarify Git.execute and Popen arguments #1688
Merged
renovate Bot referenced this pull request in allenporter/flux-local Oct 20, 2023
@renovate
Update dependency GitPython to v3.1.40 (#361)
23eac62
[![Mend
Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com)

This PR contains the following updates:

| Package | Change | Age | Adoption | Passing | Confidence |
|---|---|---|---|---|---|
| [GitPython](https://togithub.com/gitpython-developers/GitPython) |
`==3.1.37` -> `==3.1.40` |
[![age](https://developer.mend.io/api/mc/badges/age/pypi/GitPython/3.1.40?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![adoption](https://developer.mend.io/api/mc/badges/adoption/pypi/GitPython/3.1.40?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![passing](https://developer.mend.io/api/mc/badges/compatibility/pypi/GitPython/3.1.37/3.1.40?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![confidence](https://developer.mend.io/api/mc/badges/confidence/pypi/GitPython/3.1.37/3.1.40?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|

---

### Release Notes

<details>
<summary>gitpython-developers/GitPython (GitPython)</summary>

###
[`v3.1.40`](https://togithub.com/gitpython-developers/GitPython/compare/3.1.38...3.1.40)

[Compare
Source](https://togithub.com/gitpython-developers/GitPython/compare/3.1.38...3.1.40)

###
[`v3.1.38`](https://togithub.com/gitpython-developers/GitPython/releases/tag/3.1.38)

[Compare
Source](https://togithub.com/gitpython-developers/GitPython/compare/3.1.37...3.1.38)

#### What's Changed

- Add missing assert keywords by
[@&#8203;EliahKagan](https://togithub.com/EliahKagan) in
[https://github.com/gitpython-developers/GitPython/pull/1678](https://togithub.com/gitpython-developers/GitPython/pull/1678)
- Make clear every test's status in every CI run by
[@&#8203;EliahKagan](https://togithub.com/EliahKagan) in
[https://github.com/gitpython-developers/GitPython/pull/1679](https://togithub.com/gitpython-developers/GitPython/pull/1679)
- Fix new link to license in readme by
[@&#8203;EliahKagan](https://togithub.com/EliahKagan) in
[https://github.com/gitpython-developers/GitPython/pull/1680](https://togithub.com/gitpython-developers/GitPython/pull/1680)
- Drop unneeded flake8 suppressions by
[@&#8203;EliahKagan](https://togithub.com/EliahKagan) in
[https://github.com/gitpython-developers/GitPython/pull/1681](https://togithub.com/gitpython-developers/GitPython/pull/1681)
- Update instructions and test helpers for git-daemon by
[@&#8203;EliahKagan](https://togithub.com/EliahKagan) in
[https://github.com/gitpython-developers/GitPython/pull/1684](https://togithub.com/gitpython-developers/GitPython/pull/1684)
- Fix Git.execute shell use and reporting bugs by
[@&#8203;EliahKagan](https://togithub.com/EliahKagan) in
[https://github.com/gitpython-developers/GitPython/pull/1687](https://togithub.com/gitpython-developers/GitPython/pull/1687)
- No longer allow CI to select a prerelease for 3.12 by
[@&#8203;EliahKagan](https://togithub.com/EliahKagan) in
[https://github.com/gitpython-developers/GitPython/pull/1689](https://togithub.com/gitpython-developers/GitPython/pull/1689)
- Clarify Git.execute and Popen arguments by
[@&#8203;EliahKagan](https://togithub.com/EliahKagan) in
[https://github.com/gitpython-developers/GitPython/pull/1688](https://togithub.com/gitpython-developers/GitPython/pull/1688)
- Ask git where its daemon is and use that by
[@&#8203;EliahKagan](https://togithub.com/EliahKagan) in
[https://github.com/gitpython-developers/GitPython/pull/1697](https://togithub.com/gitpython-developers/GitPython/pull/1697)
- Fix bugs affecting exception wrapping in rmtree callback by
[@&#8203;EliahKagan](https://togithub.com/EliahKagan) in
[https://github.com/gitpython-developers/GitPython/pull/1700](https://togithub.com/gitpython-developers/GitPython/pull/1700)
- Fix dynamically-set **all** variable by
[@&#8203;DeflateAwning](https://togithub.com/DeflateAwning) in
[https://github.com/gitpython-developers/GitPython/pull/1659](https://togithub.com/gitpython-developers/GitPython/pull/1659)
- Fix small
[#&#8203;1662](https://togithub.com/gitpython-developers/GitPython/issues/1662)
regression due to
[#&#8203;1659](https://togithub.com/gitpython-developers/GitPython/issues/1659)
by [@&#8203;EliahKagan](https://togithub.com/EliahKagan) in
[https://github.com/gitpython-developers/GitPython/pull/1701](https://togithub.com/gitpython-developers/GitPython/pull/1701)
- Drop obsolete info on yanking from security policy by
[@&#8203;EliahKagan](https://togithub.com/EliahKagan) in
[https://github.com/gitpython-developers/GitPython/pull/1703](https://togithub.com/gitpython-developers/GitPython/pull/1703)
- Have Dependabot offer submodule updates by
[@&#8203;EliahKagan](https://togithub.com/EliahKagan) in
[https://github.com/gitpython-developers/GitPython/pull/1702](https://togithub.com/gitpython-developers/GitPython/pull/1702)
- Bump git/ext/gitdb from `49c3178` to `8ec2390` by
[@&#8203;dependabot](https://togithub.com/dependabot) in
[https://github.com/gitpython-developers/GitPython/pull/1704](https://togithub.com/gitpython-developers/GitPython/pull/1704)
- Bump git/ext/gitdb from `8ec2390` to `6a22706` by
[@&#8203;dependabot](https://togithub.com/dependabot) in
[https://github.com/gitpython-developers/GitPython/pull/1705](https://togithub.com/gitpython-developers/GitPython/pull/1705)
- Update readme for milestone-less releasing by
[@&#8203;EliahKagan](https://togithub.com/EliahKagan) in
[https://github.com/gitpython-developers/GitPython/pull/1707](https://togithub.com/gitpython-developers/GitPython/pull/1707)
- Run Cygwin CI workflow commands in login shells by
[@&#8203;EliahKagan](https://togithub.com/EliahKagan) in
[https://github.com/gitpython-developers/GitPython/pull/1709](https://togithub.com/gitpython-developers/GitPython/pull/1709)

#### New Contributors

- [@&#8203;DeflateAwning](https://togithub.com/DeflateAwning) made their
first contribution in
[https://github.com/gitpython-developers/GitPython/pull/1659](https://togithub.com/gitpython-developers/GitPython/pull/1659)

**Full Changelog**:
gitpython-developers/GitPython@3.1.37...3.1.38

</details>

---

### Configuration

📅 **Schedule**: Branch creation - At any time (no schedule defined),
Automerge - At any time (no schedule defined).

🚦 **Automerge**: Enabled.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.

---

- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box

---

This PR has been generated by [Mend
Renovate](https://www.mend.io/free-developer-tools/renovate/). View
repository job log
[here](https://developer.mend.io/github/allenporter/flux-local).

<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy4xOS4yIiwidXBkYXRlZEluVmVyIjoiMzcuMTkuMiIsInRhcmdldEJyYW5jaCI6Im1haW4ifQ==-->

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Byron Byron Byron approved these changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Milestone

v3.1.38 - Bugfixes

Development

Successfully merging this pull request may close these issues.

Git.execute debug message has misleading Popen(shell=...) Git.execute treats shell=False the same as shell=None

Uh oh!

There was an error while loading. Please reload this page.

2 participants

@EliahKagan @Byron
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402520 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.