GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-104357: fix inlined comprehensions that close over iteration var by carljm · Pull Request #104368 · python/cpython

Conversation

@carljm

@carljm carljm commented May 10, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

This fixes the case where an inlined comprehension has an internal cellvar (i.e. it creates lambdas that close over the comprehension iteration variable), but the same name is also used a fast-local in the outer scope, and is read but never written within the body of the function (so e.g. an argument).

In code like this:

def f(x):
    items = [lambda: x for x in range(2)]
    return x

because x is a cell inside the comprehension, after inlining it would appear in u_cellvars for f, and thus the compilation of f would insert a MAKE_CELL for x at the start of the function. But we would still compile the non-comprehension part of the function treating x as a fast-local (emitting LOAD_FAST for it), so f would return a cell object containing the value of x, instead of returning x directly.

Similar cases were already tested, but only with a write (compiled as STORE_FAST) to x before it was referenced, which stored the value directly and overwrote the wrongly-created cell.

One approach to fix this would be to track which names are cells only inside an inlined comprehension, and prevent the compiler from emitting MAKE_CELL for those names (the inlined comprehension will itself emit MAKE_CELL where it should, within the comprehension's isolated scope.) But this fix would add additional complexity to the compiler.

A simpler fix (implemented here) is to ensure that we always treat such a variable as a cell throughout the outer function too (emitting LOAD_DEREF and STORE_DEREF for it). This maintains clearer semantics for co_cellvars (if a name is in there, it is a cell in the main function) and keeps the compiler simpler. The downside is that cells are slightly slower than fast locals, but comprehensions creating lambdas that close over the iteration variable are rare in real code (they don't behave as people usually want them to, given that all the created lambda functions share the same final value in their closure), so I don't think performance is a concern in this edge case.

Fixes #104357

  • Issue: Comprehension inlining: Bug if comprehension contains a lambda #104357

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@carljm
pythongh-104357: fix inlined comprehensions with cells
a9c5a4f
@bedevere-bot bedevere-bot added the awaiting core review label May 10, 2023
@bedevere-bot bedevere-bot mentioned this pull request May 10, 2023
Comprehension inlining: Bug if comprehension contains a lambda #104357
Closed
@carljm carljm changed the title gh-104357: fix inlined comprehensions with cells gh-104357: fix inlined comprehensions that close over iteration var May 10, 2023
@carljm carljm added the skip news label May 10, 2023
JelleZijlstra
JelleZijlstra reviewed May 10, 2023
View reviewed changes
Comment thread Lib/test/test_listcomps.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

@carljm
update test to also verify isolation
b5c8c80
JelleZijlstra
JelleZijlstra approved these changes May 11, 2023
View reviewed changes
@bedevere-bot bedevere-bot added awaiting merge and removed awaiting core review labels May 11, 2023
@JelleZijlstra
JelleZijlstra merged commit fcd5fb4 into python:main May 11, 2023
@bedevere-bot bedevere-bot removed the awaiting merge label May 11, 2023
carljm added a commit to carljm/cpython that referenced this pull request May 11, 2023
@carljm
Merge branch 'main' into supermon
19b3d9e
* main: (27 commits)
  pythongh-87849: fix SEND specialization family definition (pythonGH-104268)
  pythongh-101819: Adapt _io.IOBase.seek and _io.IOBase.truncate to Argument Clinic (python#104384)
  pythongh-101819: Adapt _io._Buffered* methods to Argument Clinic (python#104367)
  pythongh-101819: Refactor `_io` futher in preparation for module isolation (python#104369)
  pythongh-101819: Adapt _io.TextIOBase methods to Argument Clinic (python#104383)
  pythongh-101117: Improve accuracy of sqlite3.Cursor.rowcount docs (python#104287)
  pythonGH-92184: Convert os.altsep to '/' in filenames when creating ZipInfo objects (python#92185)
  pythongh-104357: fix inlined comprehensions that close over iteration var (python#104368)
  pythonGH-90208: Suppress OSError exceptions from `pathlib.Path.glob()` (pythonGH-104141)
  pythonGH-102181: Improve specialization stats for SEND (pythonGH-102182)
  pythongh-103000: Optimise `dataclasses.asdict` for the common case (python#104364)
  pythongh-103538: Remove unused TK_AQUA code (pythonGH-103539)
  pythonGH-87695: Fix OSError from `pathlib.Path.glob()` (pythonGH-104292)
  pythongh-104263: Rely on Py_NAN and introduce Py_INFINITY (pythonGH-104202)
  pythongh-104010: Separate and improve docs for `typing.get_origin` and `typing.get_args` (python#104013)
  pythongh-101819: Adapt _io._BufferedIOBase_Type methods to Argument Clinic (python#104355)
  pythongh-103960: Dark mode: invert image brightness (python#103983)
  pythongh-104252: Immortalize Py_EMPTY_KEYS (pythongh-104253)
  pythongh-101819: Clean up _io windows console io after pythongh-104197 (python#104354)
  pythongh-101819: Harden _io init (python#104352)
  ...
carljm added a commit to carljm/cpython that referenced this pull request May 11, 2023
@carljm
Merge branch 'main' into fixloadsuperspec
4456dd6
* main:
  pythongh-87849: fix SEND specialization family definition (pythonGH-104268)
  pythongh-101819: Adapt _io.IOBase.seek and _io.IOBase.truncate to Argument Clinic (python#104384)
  pythongh-101819: Adapt _io._Buffered* methods to Argument Clinic (python#104367)
  pythongh-101819: Refactor `_io` futher in preparation for module isolation (python#104369)
  pythongh-101819: Adapt _io.TextIOBase methods to Argument Clinic (python#104383)
  pythongh-101117: Improve accuracy of sqlite3.Cursor.rowcount docs (python#104287)
  pythonGH-92184: Convert os.altsep to '/' in filenames when creating ZipInfo objects (python#92185)
  pythongh-104357: fix inlined comprehensions that close over iteration var (python#104368)
  pythonGH-90208: Suppress OSError exceptions from `pathlib.Path.glob()` (pythonGH-104141)
@carljm carljm mentioned this pull request May 12, 2023
gh-104404: fix crasher with nested comprehensions plus lambdas #104442
Merged
@JelleZijlstra JelleZijlstra mentioned this pull request May 22, 2023
PPC64LE Fedora Stable Clang 3.x fails test_gdb #104736
Closed
@carljm
carljm deleted the argcell branch June 16, 2023 17:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@JelleZijlstra JelleZijlstra JelleZijlstra approved these changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

skip news

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Comprehension inlining: Bug if comprehension contains a lambda

Uh oh!

There was an error while loading. Please reload this page.

3 participants

@carljm @JelleZijlstra @bedevere-bot
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.