GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-100238: Use setuptools in peg-generator and reenable tests by lysnikolaou · Pull Request #104798 · python/cpython

Conversation

@lysnikolaou

@lysnikolaou lysnikolaou commented May 23, 2023 •
edited by bedevere-bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member
  • Issue: Update test_peg_generator to not need distutils & reenable it #100238

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@lysnikolaou
pythongh-100238: Use setuptools in peg-generator and reenable tests
88edfa7
@lysnikolaou
lysnikolaou requested a review from pablogsal as a code owner May 23, 2023 14:26
@bedevere-bot bedevere-bot added the awaiting core review label May 23, 2023
@bedevere-bot bedevere-bot mentioned this pull request May 23, 2023
Update test_peg_generator to not need distutils & reenable it #100238
Closed
@lysnikolaou lysnikolaou mentioned this pull request May 24, 2023
Token lines in the tokenize module always end in new line #104825
Closed
@pablogsal

pablogsal commented May 24, 2023

Copy link
Copy Markdown
Member

@lysnikolaou you need to probably rebase the changes in the PR that removes the endings from lines (also two windows tests are failing).

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

lysnikolaou added 4 commits May 24, 2023 12:24
@lysnikolaou
Merge branch 'main' into setuptools-peg-generator
c750365
@lysnikolaou
Roll back changes re. newline in tokenizer
907eae8
@lysnikolaou
Fix test_pegen tests after changes in the Python tokenizer
3e8c0aa
@lysnikolaou
Take fixup_build_ext from 3.11 to use in the peg-generator
1c905d7
@pablogsal

pablogsal commented May 25, 2023

Copy link
Copy Markdown
Member

We still have the failing windows tests. This may be something regarding the name of the binary perhaps?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@lysnikolaou

lysnikolaou commented May 25, 2023

Copy link
Copy Markdown
Member Author

It appears so and I'm not sure why. The only other test that uses this (test_cppext) is skipped on Windows, so we don't really have something to compare.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@pablogsal

pablogsal commented May 25, 2023

Copy link
Copy Markdown
Member

It appears so and I'm not sure why

Do you know if python_d.exe is correct on Windows? I think that otherwise this line is kind of wrong:

Run: D:\a\cpython\cpython\PCbuild\win32\python_d.exe -X dev -m venv venv

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@pablogsal

pablogsal commented May 25, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

Also that looks odd because there should be a temporary directory there no (as in /tmp/.../venv)?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@lysnikolaou

lysnikolaou commented May 25, 2023

Copy link
Copy Markdown
Member Author

Also that looks odd because there should be a temporary directory there no (as in /tmp/.../venv)?

That's okay, since that's the command that creates the venv.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

lysnikolaou added 4 commits May 25, 2023 16:41
@lysnikolaou
Use full path in venv python executable
339e592
@lysnikolaou
Add another verbose print
672e463
@lysnikolaou
Use sys.executable to find name of venv executable
dc89e7b
@lysnikolaou
Check sysconfig for site-packages path
b3ab1a9
pablogsal
pablogsal approved these changes May 26, 2023
View reviewed changes

@pablogsal pablogsal left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great job @lysnikolaou ❤️

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@bedevere-bot bedevere-bot added awaiting merge and removed awaiting core review labels May 26, 2023
@pablogsal pablogsal added the skip news label May 26, 2023
@lysnikolaou

lysnikolaou commented May 26, 2023

Copy link
Copy Markdown
Member Author

Probabaly no backport for this, right?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@pablogsal

pablogsal commented May 26, 2023

Copy link
Copy Markdown
Member

Probabaly no backport for this, right?

I don't think so, but I also don't think it would hurt

lysnikolaou reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@lysnikolaou
lysnikolaou merged commit afa759f into python:main May 26, 2023
@bedevere-bot bedevere-bot removed the awaiting merge label May 26, 2023
This was referenced May 26, 2023
PEP 632: Remove the distutils package (Python 3.12) #92584
Closed
gh-104972: Ensure that line attributes in tokens in the tokenize module are correct #104975
Merged
buildbots fail because of test_peg_generator #104983
Closed
@Eclips4 Eclips4 mentioned this pull request May 29, 2023
test_peg_generator leaks references #105063
Open
@hroncok hroncok mentioned this pull request May 30, 2023
gh-105084: Tests: Use setuptools+wheel from sysconfig.get_config_var('WHEEL_PKG_DIR') if set #105056
Merged
@hroncok

hroncok commented May 30, 2023

Copy link
Copy Markdown
Contributor

@pablogsal I built #105056 on top. I think it would be reasonable to backport this to 3.12, so I can backport mine as is. WDYT?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@sunmy2019 sunmy2019 mentioned this pull request May 30, 2023
Ref leaks in test_peg_generator #105087
Closed
@encukou encukou added the needs backport to 3.12 only security fixes label May 31, 2023
@miss-islington

miss-islington commented May 31, 2023

Copy link
Copy Markdown
Contributor

Thanks @lysnikolaou for the PR 🌮🎉.. I'm working now to backport this PR to: 3.12.
🐍🍒⛏🤖

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request May 31, 2023
@lysnikolaou @miss-islington
pythongh-100238: Use setuptools in peg-generator and reenable tests (p…
4e73547
…ythonGH-104798)

(cherry picked from commit afa759f)

Co-authored-by: Lysandros Nikolaou <[email protected]>
@bedevere-bot

bedevere-bot commented May 31, 2023

Copy link
Copy Markdown

GH-105135 is a backport of this pull request to the 3.12 branch.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot bedevere-bot removed the needs backport to 3.12 only security fixes label May 31, 2023
@hroncok

hroncok commented May 31, 2023

Copy link
Copy Markdown
Contributor

I've asked Petr to label this with the 3.12 label so we know the CI results before a decision is made.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@encukou

encukou commented May 31, 2023

Copy link
Copy Markdown
Member

(I pushed the button to open the 3.12 PR for Miro; I'm not implying it should be merged.)

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@lysnikolaou

lysnikolaou commented May 31, 2023

Copy link
Copy Markdown
Member Author

I'm okay with backporting, since this is just re-enabling a test, so it really wouldn't hurt. I'll leave it up to @pablogsal though.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@pablogsal

pablogsal commented May 31, 2023

Copy link
Copy Markdown
Member

I'm okay with backporting, since this is just re-enabling a test, so it really wouldn't hurt. I'll leave it up to @pablogsal though.

The problem with the back port is that we have ref leaks with this PR on main no? If we back port without fixing them we are going to have them also on 3.12

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@lysnikolaou

lysnikolaou commented May 31, 2023

Copy link
Copy Markdown
Member Author

I'm okay with backporting, since this is just re-enabling a test, so it really wouldn't hurt. I'll leave it up to @pablogsal though.

The problem with the back port is that we have ref leaks with this PR on main no? If we back port without fixing them we are going to have them also on 3.12

That's right, I meant after fixing the refleaks.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@pablogsal

pablogsal commented May 31, 2023

Copy link
Copy Markdown
Member

I'm okay with backporting, since this is just re-enabling a test, so it really wouldn't hurt. I'll leave it up to @pablogsal though.

The problem with the back port is that we have ref leaks with this PR on main no? If we back port without fixing them we are going to have them also on 3.12

That's right, I meant after fixing the refleaks.

In that case I am ok back porting 👍

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@lysnikolaou lysnikolaou mentioned this pull request May 31, 2023
[BUG] setuptools.Distribution may be leaking references pypa/setuptools#3938
Open
@erlend-aasland erlend-aasland mentioned this pull request Jun 23, 2023
gh-105063: Disable test_peg_generator.TestCParser bco. ref leaks #106024
Merged
erlend-aasland added a commit to erlend-aasland/cpython that referenced this pull request Jun 23, 2023
@erlend-aasland
pythongh-105063: Disable test_peg_generator.TestCParser bco. ref leaks
97a988b
Since pythongh-104798 (Use setuptools in peg-generator and reenable
tests), the TestCParser test case has been producing ref leaks.
erlend-aasland added a commit that referenced this pull request Jun 23, 2023
@erlend-aasland
gh-105063: Disable test_peg_generator.TestCParser bco. ref leaks (#10…
41ad4df
…6024)

Since gh-104798 (Use setuptools in peg-generator and reenable
tests), the TestCParser test case has been producing ref leaks.
lysnikolaou added a commit that referenced this pull request Jul 5, 2023
@miss-islington @lysnikolaou
[3.12] gh-100238: Use setuptools in peg-generator and reenable tests (G…
6371029
…H-104798) (#105135)

(cherry picked from commit afa759f)

Co-authored-by: Lysandros Nikolaou <[email protected]>
@miss-islington miss-islington mentioned this pull request Jul 5, 2023
[3.12] gh-105063: Disable test_peg_generator.TestCParser bco. ref leaks (GH-106024) #106450
Merged
miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Jul 5, 2023
@erlend-aasland @miss-islington
pythongh-105063: Disable test_peg_generator.TestCParser bco. ref leaks (
c8b2a49
pythonGH-106024)

Since pythongh-104798 (Use setuptools in peg-generator and reenable
tests), the TestCParser test case has been producing ref leaks.
(cherry picked from commit 41ad4df)

Co-authored-by: Erlend E. Aasland <[email protected]>
ambv pushed a commit that referenced this pull request Jul 5, 2023
@miss-islington @erlend-aasland
[3.12] gh-105063: Disable test_peg_generator.TestCParser bco. ref lea…
b314194
…ks (GH-106024) (#106450)

Since gh-104798 (Use setuptools in peg-generator and reenable
tests), the TestCParser test case has been producing ref leaks.
(cherry picked from commit 41ad4df)

Co-authored-by: Erlend E. Aasland <[email protected]>
@bedevere-bot

bedevere-bot commented Jul 5, 2023

Copy link
Copy Markdown

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Windows11 Bigmem 3.12 has failed when building commit 6371029.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/1117/builds/131) and take a look at the build logs.
  4. Check if the failure is related to this commit (6371029) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/1117/builds/131

Summary of the results of the build (if available):

== Tests result: ENV CHANGED ==

431 tests OK.

10 slowest tests:

  • test_bigmem: 43 min 23 sec
  • test_lzma: 34 min 18 sec
  • test_bz2: 20 min 50 sec
  • test_array: 7 min 23 sec
  • test_zlib: 5 min 5 sec
  • test_hashlib: 4 min 3 sec
  • test_multiprocessing_spawn: 2 min 1 sec
  • test_math: 1 min 32 sec
  • test_pickle: 1 min 18 sec
  • test_concurrent_futures: 1 min 14 sec

1 test altered the execution environment:
test_peg_generator

35 tests skipped:
test.test_asyncio.test_unix_events test_curses test_dbm_gnu
test_dbm_ndbm test_devpoll test_epoll test_fcntl test_fork1
test_gdb test_grp test_ioctl test_kqueue test_multiprocessing_fork
test_multiprocessing_forkserver test_nis test_openpty
test_ossaudiodev test_perf_profiler test_perfmaps test_pipes
test_poll test_posix test_pty test_pwd test_readline test_resource
test_spwd test_syslog test_threadsignals test_wait3 test_wait4
test_xxlimited test_xxtestfuzz test_zipfile64 test_zoneinfo

Total duration: 43 min 51 sec

Click to see traceback logs
remote: Enumerating objects: 13, done.        
remote: Counting objects:   8% (1/12)        
remote: Counting objects:  16% (2/12)        
remote: Counting objects:  25% (3/12)        
remote: Counting objects:  33% (4/12)        
remote: Counting objects:  41% (5/12)        
remote: Counting objects:  50% (6/12)        
remote: Counting objects:  58% (7/12)        
remote: Counting objects:  66% (8/12)        
remote: Counting objects:  75% (9/12)        
remote: Counting objects:  83% (10/12)        
remote: Counting objects:  91% (11/12)        
remote: Counting objects: 100% (12/12)        
remote: Counting objects: 100% (12/12), done.        
remote: Compressing objects:  11% (1/9)        
remote: Compressing objects:  22% (2/9)        
remote: Compressing objects:  33% (3/9)        
remote: Compressing objects:  44% (4/9)        
remote: Compressing objects:  55% (5/9)        
remote: Compressing objects:  66% (6/9)        
remote: Compressing objects:  77% (7/9)        
remote: Compressing objects:  88% (8/9)        
remote: Compressing objects: 100% (9/9)        
remote: Compressing objects: 100% (9/9), done.        
remote: Total 13 (delta 3), reused 8 (delta 3), pack-reused 1        
From https://github.com/python/cpython
 * branch                  3.12       -> FETCH_HEAD
Note: switching to '637102980d12e5ff99b69ddffd5f05ecfe0adeb6'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 637102980d [3.12] gh-100238: Use setuptools in peg-generator and reenable tests (GH-104798) (#105135)
Switched to and reset branch '3.12'

Could Not Find R:\buildarea\3.12.ambv-bb-win11.bigmem\build\Lib\*.pyc
The system cannot find the file specified.
Could Not Find R:\buildarea\3.12.ambv-bb-win11.bigmem\build\PCbuild\python*.zip

Could Not Find R:\buildarea\3.12.ambv-bb-win11.bigmem\build\PCbuild\python*.zip
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pablogsal pablogsal pablogsal approved these changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

skip news

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

6 participants

@lysnikolaou @pablogsal @hroncok @miss-islington @bedevere-bot @encukou
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.