GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-104615: don't make unsafe swaps in apply_static_swaps by carljm · Pull Request #104620 · python/cpython

Conversation

@carljm

@carljm carljm commented May 18, 2023 •
edited by bedevere-bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

It's not safe to apply SWAP statically if it would reorder two instructions that store to the same location.

  • Issue: compiler can incorrectly optimize a run of stores to the same name preceded by a SWAP #104615

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@carljm
pythongh-104615: don't make unsafe swaps in apply_static_swaps
22533ba
@bedevere-bot bedevere-bot mentioned this pull request May 18, 2023
compiler can incorrectly optimize a run of stores to the same name preceded by a SWAP #104615
Closed
carljm added 2 commits May 18, 2023 13:00
@carljm
add NEWS entry
3f5c4e6
@carljm
fix NEWS entry rst
3a9be8c
@carljm

carljm commented May 18, 2023

Copy link
Copy Markdown
Member Author

One thing I'm not clear on is whether we want to bump bytecode magic number for a fix like this. The old bytecode still works as well it ever did (no incompatible changes in the interpreter), so maybe no? But OTOH, nobody will see this bugfix until their pycs are invalidated.

Basically: do we bump bytecode magic number on any change to the compiler that results in different compiler output, or only when we change the interpreter in a way that makes it incompatible with previous bytecode?
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@carljm
regenerate global objects
6a9f167
@JelleZijlstra

JelleZijlstra commented May 18, 2023

Copy link
Copy Markdown
Member

One thing I'm not clear on is whether we want to bump bytecode magic number for a fix like this. The old bytecode still works as well it ever did (no incompatible changes in the interpreter), so maybe no? But OTOH, nobody will see this bugfix until their pycs are invalidated.

Basically: do we bump bytecode magic number on any change to the compiler that results in different compiler output, or only when we change the interpreter in a way that makes it incompatible with previous bytecode?

I was wondering a bit about that too. Several other recent fixes around comprehensions would also change the compiler output, so theoretically we should have bumped the magic number a few times. On the other hand, the only people who are really affected would be people who built from main directly after the PEP 695 change was merged (which did bump the magic number), so realistically there's not much of an issue. It's probably enough if we bump the magic number one last time before the beta is finalized.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@carljm
carljm marked this pull request as ready for review May 18, 2023 20:35
@carljm
carljm requested review from brandtbucher, iritkatriel and markshannon as code owners May 18, 2023 20:35
@bedevere-bot bedevere-bot added the awaiting core review label May 18, 2023
@brandtbucher

brandtbucher commented May 18, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

My thought has mostly been that magic numbers should only be bumped if the bytecode itself is incompatible (meaning opcodes or the meanings of opargs have changed). I don't think we've bumped it for miscompilations before (but I could be wrong).

(We also bump it when changing how code objects are unmarshalled, which is arguably an abuse of the magic number and should be a marshal version bump instead.)
carljm reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
iritkatriel
iritkatriel reviewed May 18, 2023
View reviewed changes
Comment thread Python/flowgraph.c
}
for (int idx = j + 1; idx < k; idx++) {
int store_idx = STORES_TO(block->b_instr[idx]);
if (store_idx >= 0 && (store_idx == store_j || store_idx == store_k)) {

@iritkatriel iritkatriel May 18, 2023

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How would it happen that there's a STORE_FAST* between j and k?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions

@carljm carljm May 18, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can happen with code like a, a, b = x, y, z as in some of the added tests. This compiles to LOAD_FAST x; LOAD_FAST y; LOAD_FAST z; SWAP 3; STORE_FAST a; STORE_FAST a; STORE_FAST b. Before this PR, apply_static_swaps would optimize that (ignoring the loads) to STORE_FAST b; STORE_FAST a; STORE_FAST a (swapping the first and third store), which is invalid because it reorders the two stores to a.

k - j == n - 1 here, where n is the oparg to the SWAP. So for a SWAP 2, j and k will be adjacent, but for SWAP with oparg > 2, there will be intervening instructions. And STORE_FAST is a SWAPPABLE instruction, so those intervening instructions can be STORE_FAST.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
iritkatriel reacted with thumbs up emoji
All reactions
  • 👍 1 reaction
brandtbucher
brandtbucher approved these changes May 18, 2023
View reviewed changes

@brandtbucher brandtbucher left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the quick fix!

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@bedevere-bot bedevere-bot added awaiting merge and removed awaiting core review labels May 18, 2023
iritkatriel
iritkatriel approved these changes May 18, 2023
View reviewed changes
@carljm
carljm enabled auto-merge (squash) May 18, 2023 21:00
@carljm
carljm merged commit 0589c6a into python:main May 18, 2023
@bedevere-bot bedevere-bot removed the awaiting merge label May 18, 2023
@carljm
carljm deleted the staticswaps branch May 18, 2023 21:31
@carljm

carljm commented May 18, 2023

Copy link
Copy Markdown
Member Author

Oh, I forgot: this should be backported to 3.11, since it's a bug that exists there as well.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@carljm carljm added the needs backport to 3.11 only security fixes label May 18, 2023
@miss-islington

miss-islington commented May 18, 2023

Copy link
Copy Markdown
Contributor

Thanks @carljm for the PR 🌮🎉.. I'm working now to backport this PR to: 3.11.
🐍🍒⛏🤖
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@miss-islington

miss-islington commented May 18, 2023

Copy link
Copy Markdown
Contributor

Sorry @carljm, I had trouble checking out the 3.11 backport branch.
Please retry by removing and re-adding the "needs backport to 3.11" label.
Alternatively, you can backport using cherry_picker on the command line.
cherry_picker 0589c6a4d3d822cace42050198cb9a5e99c879ad 3.11
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@miss-islington miss-islington assigned carljm May 18, 2023
@carljm carljm added needs backport to 3.11 only security fixes and removed needs backport to 3.11 only security fixes labels May 18, 2023
@miss-islington

miss-islington commented May 18, 2023

Copy link
Copy Markdown
Contributor

Thanks @carljm for the PR 🌮🎉.. I'm working now to backport this PR to: 3.11.
🐍🍒⛏🤖
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@miss-islington

miss-islington commented May 18, 2023

Copy link
Copy Markdown
Contributor

Sorry, @carljm, I could not cleanly backport this to 3.11 due to a conflict.
Please backport using cherry_picker on command line.
cherry_picker 0589c6a4d3d822cace42050198cb9a5e99c879ad 3.11
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
carljm added a commit to carljm/cpython that referenced this pull request May 18, 2023
@carljm
Merge branch 'main' into nestedcomps
54095f0 
* main:
  pythongh-74690: Don't set special protocol attributes on non-protocol subclasses of protocols (python#104622)
  pythongh-104623: Update Windows installer to use SQLite 3.42.0 (python#104625)
  pythongh-104050: Add more type annotations to Argument Clinic (python#104628)
  pythongh-104629: Don't skip test_clinic if _testclinic is missing (python#104630)
  pythongh-104549: Set __module__ on TypeAliasType (python#104550)
  pythongh-104050: Improve some typing around `default`s and sentinel values (python#104626)
  pythongh-104146: Remove unused vars from Argument Clinic (python#104627)
  pythongh-104615: don't make unsafe swaps in apply_static_swaps (python#104620)
  pythonGH-104484: Add case_sensitive argument to `pathlib.PurePath.match()` (pythonGH-104565)
  pythonGH-96803: Document and test new unstable internal frame API functions (pythonGH-104211)
  pythonGH-104580: Don't cache eval breaker in interpreter (pythonGH-104581)
carljm added a commit to carljm/cpython that referenced this pull request May 19, 2023
@carljm
[3.11] pythongh-104615: don't make unsafe swaps in apply_static_swaps (…
eb83a26 
…pythonGH-104620).

(cherry picked from commit 0589c6a)

Co-authored-by: Carl Meyer <[email protected]>
@bedevere-bot

bedevere-bot commented May 19, 2023

Copy link
Copy Markdown

GH-104636 is a backport of this pull request to the 3.11 branch.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-bot bedevere-bot removed the needs backport to 3.11 only security fixes label May 19, 2023
carljm added a commit that referenced this pull request May 19, 2023
@carljm
[3.11] gh-104615: don't make unsafe swaps in apply_static_swaps (GH-1…
667e4ec 
…04620). (#104636)

(cherry picked from commit 0589c6a)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@iritkatriel iritkatriel iritkatriel approved these changes
@brandtbucher brandtbucher brandtbucher approved these changes
@markshannon markshannon Awaiting requested review from markshannon markshannon is a code owner

Uh oh!

There was an error while loading. Please reload this page.

Assignees

@carljm carljm

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

6 participants

@carljm @JelleZijlstra @brandtbucher @miss-islington @bedevere-bot @iritkatriel
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.