GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-109408: Run `patchcheck` in GitHub Actions by sobolevn · Pull Request #109459 · python/cpython

Conversation

@sobolevn

@sobolevn sobolevn commented Sep 15, 2023 •
edited by bedevere-app Bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

Continuing your idea in #109408 (comment)

Refs #109452

  • Issue: Move Azure Pipelines CI to GitHub Actions #109408

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@sobolevn
pythongh-109408: Run patchcheck in GitHub Actions
fd695d5
@sobolevn
sobolevn requested review from ezio-melotti and hugovk as code owners September 15, 2023 16:49
@bedevere-app bedevere-app Bot added the awaiting review label Sep 15, 2023
@bedevere-app bedevere-app Bot mentioned this pull request Sep 15, 2023
Move Azure Pipelines CI to GitHub Actions #109408
Closed
@sobolevn sobolevn added the skip news label Sep 15, 2023
sobolevn added 2 commits September 15, 2023 19:51
@sobolevn
try to fix ci
7bc1997
@sobolevn
try to fix ci
86622a8
@sobolevn
sobolevn marked this pull request as draft September 15, 2023 17:00
@bedevere-app bedevere-app Bot removed the awaiting review label Sep 15, 2023
sobolevn added 5 commits September 15, 2023 20:09
@sobolevn
try to fix ci
47da9c3
@sobolevn
try to fix ci
073d8ee
@sobolevn
try to fix ci
3c0630f
@sobolevn
try to fix ci
dbdc48a
@sobolevn
try to fix ci
c6e2717
vstinner
vstinner reviewed Sep 15, 2023
View reviewed changes
Comment thread .github/workflows/lint.yml Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread .github/workflows/lint.yml Resolved

Uh oh!

There was an error while loading. Please reload this page.

@sobolevn
debug
375cff9
hugovk
hugovk reviewed Sep 15, 2023
View reviewed changes
Comment thread .github/workflows/lint.yml Resolved

Uh oh!

There was an error while loading. Please reload this page.

AA-Turner
AA-Turner reviewed Sep 15, 2023
View reviewed changes
Comment thread .github/workflows/lint.yml Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread .github/workflows/lint.yml Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Tools/patchcheck/patchcheck.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

sobolevn added 2 commits September 15, 2023 22:56
@sobolevn
Run patchcheck as a part of "Check generated files" job
89ff09e
@sobolevn
Use make
703be97
@sobolevn
sobolevn marked this pull request as ready for review September 15, 2023 20:21
@bedevere-app bedevere-app Bot added the awaiting review label Sep 15, 2023
@sobolevn

sobolevn commented Sep 15, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member Author

Finally! :)
Sorry for the noise, it took me quite some time to reproduce this in CI.

hugovk reacted with hooray emoji
All reactions
  • 🎉 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

hugovk
hugovk reviewed Sep 15, 2023
View reviewed changes
Comment thread .github/workflows/build.yml
- name: Run patchcheck
if: github.event_name == 'pull_request'
run: |
git fetch origin

@hugovk hugovk Sep 15, 2023

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to fetch origin? It takes 1m 49s for this step.

We don't do it on Azure Pipelines and patchcheck takes 2s.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@sobolevn sobolevn Sep 16, 2023

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's try :)

I think that we might need it because of the heavy git machinery inside patchcheck.
I think we might need it during backports for older branches.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@sobolevn sobolevn Sep 16, 2023

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nope, it does not work:

Run # git fetch origin
Checked 107 modules (31 built-in, 75 shared, 1 n/a on linux-x86_64, 0 disabled, 0 missing, 0 failed on import)
LD_LIBRARY_PATH=/home/runner/work/cpython/cpython ./python ./Tools/patchcheck/patchcheck.py --ci true
Getting base branch for PR ... origin/main
fatal: ambiguous argument 'origin/main': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
error running git diff --name-status origin/main
make: *** [Makefile:2914: patchcheck] Error 1
Getting the list of files that have been added/changed ... 

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@AA-Turner

AA-Turner commented Sep 16, 2023

Copy link
Copy Markdown
Member

@sobolevn -- if you want to keep patchcheck in its own job for speed, AA-Turner@8af0f2c is a sketch of an approach.

Two changes are needed to patchcheck, both due to prior assumptions that patchcheck runs on a local build of CPython.

  • On CI, use the current working directory as SRCDIR rather than using sysconfig.
  • We use patchlevel.h to get the Python version, not sys.version_info (the docs use this approach too)

It may be worth considering using the CI environment variable rather than a ci=true CLI flag, but that could be delayed to a later date.

A

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@sobolevn
Try without git fetch origin
0f33871
@sobolevn

sobolevn commented Sep 16, 2023

Copy link
Copy Markdown
Member Author

@AA-Turner I am not quite comfortable refactoring code that I don't quite understand :)
And since this is liteally the first time I see patchcheck, I would prefer to keep this PR as simple as possible.

Later we can change the tooling to be more convenient / faster, but for now I would like to stick with my solution.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@sobolevn
Return git fetch origin back
e630416
@hugovk

hugovk commented Sep 16, 2023

Copy link
Copy Markdown
Member

In the end I'd like to remove or replace as much of patchcheck with equivalents that we do understand, like pre-commit or Ruff linting. But this is a good first step.

sobolevn reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@sobolevn

sobolevn commented Sep 16, 2023

Copy link
Copy Markdown
Member Author

I agree 100%, that what I was thinking about all the way :)

hugovk reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@hugovk

hugovk commented Sep 16, 2023

Copy link
Copy Markdown
Member

@sobolevn -- if you want to keep patchcheck in its own job for speed, AA-Turner@8af0f2c is a sketch of an approach.

This approach takes 22s, including fetching source and installing a prebuilt Python.

This PR piggy backs on 'Check if generated files are up to date' to take advantage of a local Python build, but is +1m 48s, mostly fetching source.

  • We use patchlevel.h to get the Python version, not sys.version_info (the docs use this approach too)

Could be nice to refactor Doc/tools/extensions/patchlevel.py and put it somewhere common for use by both?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@sobolevn

sobolevn commented Sep 16, 2023

Copy link
Copy Markdown
Member Author

@hugovk I propose we merge this as-is for now, because +1m43s is not even close to how long our regular tests take (around 15m 4s). Moreover, we will save +13m from Azure Pipeline, when duplication is deleted:
Снимок экрана 2023-09-16 в 15 02 35

Later we can:

  • Replace most checks with pre-commit / ruff
  • Refactor the rest checks to use modern alternatives / API
  • Do not build CPython from source here for no reason
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@AA-Turner

AA-Turner commented Sep 17, 2023

Copy link
Copy Markdown
Member

A slightly different approach would be AA-Turner@5ad4faf, which avoids all of the Git chicanery and just runs patchcheck on all files in the source tree. This seems to take ~20s (ex1, ex2).

A

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@hugovk

hugovk commented Sep 17, 2023

Copy link
Copy Markdown
Member

A slightly different approach would be AA-Turner@5ad4faf, which avoids all of the Git chicanery and just runs patchcheck on all files in the source tree. This seems to take ~20s (ex1, ex2).

A

~30s for the whole thing, but that's not too bad and I think I prefer this approach -- we also run on all files for pre-commit on the CI.

AlexWaygood and AA-Turner reacted with thumbs up emoji
All reactions
  • 👍 2 reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@sobolevn

sobolevn commented Sep 17, 2023

Copy link
Copy Markdown
Member Author

Yeap, all files is also fine by me (since they are all correct anyway).

I will rework @AA-Turner's script to make it possible to run on Windows as well, because it might be useful for local testing. Thanks!

hugovk reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@AA-Turner

AA-Turner commented Sep 17, 2023

Copy link
Copy Markdown
Member

I will rework @AA-Turner's script to make it possible to run on Windows as well

I developed it on Windows--though I normalised all paths to forward-slash, for simplicity.

A

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@AA-Turner

AA-Turner commented Sep 26, 2023

Copy link
Copy Markdown
Member

If all three of #109854; #109890; and #109891 are merged then this PR may be closed.

A

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@sobolevn sobolevn closed this Nov 3, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@vstinner vstinner vstinner left review comments
@hugovk hugovk hugovk left review comments
@AA-Turner AA-Turner AA-Turner left review comments
@ezio-melotti ezio-melotti Awaiting requested review from ezio-melotti ezio-melotti is a code owner

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

awaiting review skip news

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

4 participants

@sobolevn @AA-Turner @hugovk @vstinner
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.