GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-91054: make code watcher tests resilient to other watchers by carljm · Pull Request #107821 · python/cpython

Conversation

@carljm

@carljm carljm commented Aug 9, 2023 •
edited by bedevere-bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

The code watcher tests implicitly assume that the watchers they will register will be numbered 0 and 1. But this assumption may not hold true if the tests are run with another watcher already active.

This PR fixes the tests to record the actual watcher ID for each "test watcher index" and use that to ensure we clear the correct recorded data when clearing a watcher.

  • Issue: Add API to allow extensions to set callback function on creation and destruction of PyCodeObject #91054

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@carljm carljm added the skip news label Aug 9, 2023
@carljm
carljm requested a review from itamaro August 9, 2023 20:20
@bedevere-bot bedevere-bot mentioned this pull request Aug 9, 2023
Add API to allow extensions to set callback function on creation and destruction of PyCodeObject #91054
Closed
@bedevere-bot bedevere-bot added the awaiting core review label Aug 9, 2023
@carljm

carljm commented Aug 9, 2023

Copy link
Copy Markdown
Member Author

Skipping news since this is purely a fix to CPython's own test suite.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@carljm

carljm commented Aug 9, 2023

Copy link
Copy Markdown
Member Author

@Yhg1s how would you feel about backporting this test-suite-only fix to 3.12, to reduce spurious failures in 3.12 if anyone runs the 3.12 tests with a code watcher active, and reduce conflicts on future backports in these tests?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@carljm
pythongh-91054: make code watcher tests resilient to other watchers
b40b696
@carljm
carljm force-pushed the codewatchertests branch from f32672c to b40b696 Compare August 9, 2023 20:27
@carljm
update c-analyzer ignores
9feccba
@itamaro

itamaro commented Aug 9, 2023

Copy link
Copy Markdown
Contributor

in what scenario is this test run with other code watchers registered?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

itamaro
itamaro approved these changes Aug 9, 2023
View reviewed changes
@carljm

carljm commented Aug 9, 2023

Copy link
Copy Markdown
Member Author

in what scenario is this test run with other code watchers registered?

In practice today, probably it isn't. I found this bug when backporting code watchers to Cinder 3.10 so that the Cinder JIT can use them; in that case the Cinder JIT installs a code watcher, and the test suite should pass with the JIT active. In future, in principle, any Python optimizer or debug/monitoring tool that installs a code watcher may want to validate that the Python test suite passes with it active, and I think the tests should work in such cases. (The Cinder JIT for Python 3.12 will certainly be one such case.)

There is the possible edge case in which someone tries to run these tests with more than six code watchers active, in which case they'd fail because there aren't enough code watcher slots left. I doubt that case will ever be encountered unless artificially constructed, but a possible safeguard could be to skip the tests if too many code watchers are already installed.

itamaro reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@carljm
carljm merged commit 2ec16fe into python:main Aug 9, 2023
@bedevere-bot bedevere-bot removed the awaiting core review label Aug 9, 2023
@Yhg1s

Yhg1s commented Aug 10, 2023

Copy link
Copy Markdown
Member

This makes sense to me to backport to 3.12. I think it's worth realising that the Python test suite does get run in situations different from the default one. (At work we run the whole testsuite with an embedded interpreter and custom importers, for example, and I could totally see us adding something that uses watchers for something or other.)

carljm reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@Yhg1s Yhg1s added the needs backport to 3.12 only security fixes label Aug 10, 2023
@miss-islington

miss-islington commented Aug 10, 2023

Copy link
Copy Markdown
Contributor

Thanks @carljm for the PR 🌮🎉.. I'm working now to backport this PR to: 3.12.
🐍🍒⛏🤖 I'm not a witch! I'm not a witch!

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Aug 10, 2023
@carljm @miss-islington
pythongh-91054: make code watcher tests resilient to other watchers (p…
f53bd4d
…ythonGH-107821)

(cherry picked from commit 2ec16fe)

Co-authored-by: Carl Meyer <[email protected]>
@bedevere-bot

bedevere-bot commented Aug 10, 2023

Copy link
Copy Markdown

GH-107835 is a backport of this pull request to the 3.12 branch.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot bedevere-bot removed the needs backport to 3.12 only security fixes label Aug 10, 2023
facebook-github-bot pushed a commit to facebookincubator/MetaPython that referenced this pull request Aug 10, 2023
Carl Meyer @facebook-github-bot
use code watcher in JIT
eacfd94
Summary:
This reverts the revert commit 94d0dbf6891cff0324fdb2d75e9d0fcc9529675f
(D47448305), re-landing D47201503, but with a key change: in the first version I'd missed the line that actually initializes the code watcher on JIT startup, so the code watcher never took effect. This meant that in a JIT-list scenario, we didn't ever remove code objects from the to-be-compiled registry when they were destroyed, so we ended up with dangling pointers on the registry.

This situation should be better tested, and I think it would have been likely caught by the multithreaded-compile test, but that test has been broken and disabled for quite a while. I created T158334137 to track fixing and re-enabling it.

This diff also adds a small improvement to the debug message on a JIT_CHECK.

I also had to fix the code watchers test suite; it didn't work correctly when another code watcher was already enabled. I upstreamed this fix in python/cpython#107821

Reviewed By: alexmalyshev

Differential Revision: D47459869

fbshipit-source-id: c53c9847a300892d1e0b01a6a75f524ed730fd48
Yhg1s pushed a commit that referenced this pull request Aug 11, 2023
@miss-islington @carljm
[3.12] gh-91054: make code watcher tests resilient to other watchers (G…
98dd9d9
…H-107821) (#107835)

gh-91054: make code watcher tests resilient to other watchers (GH-107821)
(cherry picked from commit 2ec16fe)

Co-authored-by: Carl Meyer <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@itamaro itamaro itamaro approved these changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

skip news

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

5 participants

@carljm @itamaro @Yhg1s @miss-islington @bedevere-bot
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.