GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — bpo-40334: Rewrite test_c_parser to avoid memory leaks by lysnikolaou · Pull Request #19694 · python/cpython

Conversation

@lysnikolaou

@lysnikolaou lysnikolaou commented Apr 23, 2020 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

Previously every test was building an extension module and
loading it into sys.modules. The tearDown function was thus
not able to clean up correctly, resulting in memory leaks.

With this PR every test function now builds the extension
module and runs the actual test code in a new process
(using assert_python_ok), so that sys.modules stays intact
and no memory gets leaked.

https://bugs.python.org/issue40334

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@the-knights-who-say-ni the-knights-who-say-ni added the CLA signed label Apr 23, 2020
@bedevere-bot bedevere-bot added tests Tests in the Lib/test dir awaiting review labels Apr 23, 2020
@lysnikolaou lysnikolaou added the skip news label Apr 23, 2020
@lysnikolaou
lysnikolaou requested a review from gvanrossum April 23, 2020 23:08
@vstinner

vstinner commented Apr 23, 2020

Copy link
Copy Markdown
Member

Why do you remove tests? It may be worth it to explain it in the commit message.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
gvanrossum
gvanrossum approved these changes Apr 23, 2020
View reviewed changes

@gvanrossum gvanrossum left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will land once tests pass.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@bedevere-bot bedevere-bot added awaiting merge and removed awaiting review labels Apr 23, 2020
@lysnikolaou
lysnikolaou force-pushed the remove-test-cparser branch from 5731030 to 5f04be3 Compare April 23, 2020 23:16
@lysnikolaou

lysnikolaou commented Apr 23, 2020

Copy link
Copy Markdown
Member Author

Why do you remove tests? It may be worth it to explain it in the commit message.

Commit message updated. Thanks for the advice.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@gvanrossum

gvanrossum commented Apr 23, 2020

Copy link
Copy Markdown
Member

(Hmm, there's a plan to skip the tests rather than delete the file.)
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@lysnikolaou

lysnikolaou commented Apr 23, 2020

Copy link
Copy Markdown
Member Author

(Hmm, there's a plan to skip the tests rather than delete the file.)

Or maybe even fix them..
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@lysnikolaou
lysnikolaou marked this pull request as draft April 23, 2020 23:45
@lysnikolaou
bpo-40334: Rewrite test_c_parser to avoid memory leaks
4c02d3b 
Previously every test was building an extension module and
loading it into sys.modules. The tearDown function was thus
not able to clean up correctly, resulting in memory leaks.

With this PR every test function now builds the extension
module and runs the actual test code in a new process
(using assert_python_ok), so that sys.modules stays intact
and no memory gets leaked.
@lysnikolaou
lysnikolaou force-pushed the remove-test-cparser branch from 5f04be3 to 4c02d3b Compare April 24, 2020 10:45
@lysnikolaou lysnikolaou changed the title bpo-40334: Remove test_c_parser from test_peg_generator bpo-40334: Rewrite test_c_parser to avoid memory leaks Apr 24, 2020
@lysnikolaou
lysnikolaou requested review from gvanrossum and pablogsal April 24, 2020 10:46
@lysnikolaou
lysnikolaou marked this pull request as ready for review April 24, 2020 10:47
@lysnikolaou

lysnikolaou commented Apr 24, 2020

Copy link
Copy Markdown
Member Author

PR updated! It now fixes the tests that were causing the memory leaks, instead of removing them. (I didn't open a new one, because we needed to preserve the PR number)
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@lysnikolaou
Run 'make patchcheck' and black
831e3c6
pablogsal
pablogsal reviewed Apr 24, 2020
View reviewed changes
Comment thread Lib/test/test_peg_generator/test_c_parser.py Resolved

Uh oh!

There was an error while loading. Please reload this page.
@lysnikolaou
Remove suppression of PermissionError, that shouldn't occur anymore
fcd1ada
@lysnikolaou lysnikolaou mentioned this pull request Apr 24, 2020
Changing labels when a review gets requested python/bedevere#219
Closed
@pablogsal pablogsal added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Apr 24, 2020
@bedevere-bot

bedevere-bot commented Apr 24, 2020

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @pablogsal for commit fcd1ada 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Apr 24, 2020
@pablogsal

pablogsal commented Apr 24, 2020

Copy link
Copy Markdown
Member

Checking with the buildbots to confirm that there were no more leaks.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@lysnikolaou

lysnikolaou commented Apr 24, 2020

Copy link
Copy Markdown
Member Author

On RHEL 7 it's only test_threading that leaks now.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@pablogsal

pablogsal commented Apr 24, 2020

Copy link
Copy Markdown
Member

On RHEL 7 it's only test_threading that leaks now.

Yup, all the buildbot failures are unrelated to this PR (and the refleaks ones do not leak anymore because of this) 🎉
lysnikolaou reacted with hooray emoji
All reactions
  • 🎉 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
pablogsal
pablogsal approved these changes Apr 24, 2020
View reviewed changes
@pablogsal
pablogsal merged commit 24ffe70 into python:master Apr 24, 2020
@bedevere-bot bedevere-bot removed the awaiting merge label Apr 24, 2020
@lysnikolaou
lysnikolaou deleted the remove-test-cparser branch April 24, 2020 13:52
CuriousLearner added a commit to CuriousLearner/cpython that referenced this pull request Apr 27, 2020
@CuriousLearner
Merge branch 'master' of github.com:python/cpython into fix-issue21914
42c39a3 
* 'master' of github.com:python/cpython: (2949 commits)
  Add files in tests/test_peg_generator to the install target lists (pythonGH-19723)
  bpo-40398: Fix typing.get_args() for special generic aliases. (pythonGH-19720)
  bpo-40348: Fix typos in the programming FAQ (pythonGH-19729)
  bpo-38387: Formally document PyDoc_STRVAR and PyDoc_STR macros (pythonGH-16607)
  bpo-40401: Remove duplicate pyhash.h include from pythoncore.vcxproj (pythonGH-19725)
  bpo-40387: Improve queue join() example. (pythonGH-19724)
  bpo-40396: Support GenericAlias in the typing functions. (pythonGH-19718)
  Fix typo in Lib/typing.py (pythonGH-19717)
  Fix typo in object.__format__ docs (pythonGH-19504)
  bpo-40275: Avoid importing logging in test.support (pythonGH-19601)
  bpo-40275: Avoid importing socket in test.support (pythonGH-19603)
  bpo-40275: Avoid importing asyncio in test.support (pythonGH-19600)
  bpo-40279: Add some error-handling to the module initialisation docs example (pythonGH-19705)
  closes bpo-40385: Remove Tools/scripts/checkpyc.py (pythonGH-19709)
  bpo-40334: Add What's New sections for PEP 617 and PEP 585 (pythonGH-19704)
  bpo-40340: Separate examples more clearly in the programming FAQ (pythonGH-19688)
  bpo-40360: Deprecate lib2to3 module in light of PEP 617 (pythonGH-19663)
  bpo-40334: Rewrite test_c_parser to avoid memory leaks (pythonGH-19694)
  bpo-38061: subprocess uses closefrom() on FreeBSD (pythonGH-19697)
  bpo-38061: os.closerange() uses closefrom() on FreeBSD (pythonGH-19696)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pablogsal pablogsal pablogsal approved these changes
@gvanrossum gvanrossum Awaiting requested review from gvanrossum

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

skip news tests Tests in the Lib/test dir

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

6 participants

@lysnikolaou @vstinner @gvanrossum @bedevere-bot @pablogsal @the-knights-who-say-ni
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.