GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — GH-98831: Implement array support in cases generator by gvanrossum · Pull Request #100912 · python/cpython

Conversation

@gvanrossum

@gvanrossum gvanrossum commented Jan 10, 2023 •
edited by bedevere-bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

There are some issues, but as Mark said we needn't attempt to get the generator to do everything (at least not right now).

In particular, as I mentioned in the meeting, there's no full support for output arrays (but maybe we'll never need that).

While I don't envision super-instructions with array operands, once macros become more popular they may well need arrays -- that's currently not supported.

We might want to consider a macro to DECREF an array of object pointers; this is currently done in an ad-hoc fashion.

  • Issue: Generate the interpreter #98831

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
gvanrossum added 16 commits January 6, 2023 09:07
@gvanrossum
Tweak 'This file is generated...' header
4b17476
@gvanrossum
Start on array stack effects
69f04e4
@gvanrossum
Simple input array effects work
b9bd7e3
@gvanrossum
Ignore Context when comparing Nodes
bb53c44
@gvanrossum
Improve stack effect counting (TODO: use everywhere)
4a15c53
@gvanrossum
Fix stack effect counting in most places
0a1e9a2
@gvanrossum
Fix TODO
b023236
@gvanrossum
Fix opcode metadata for array stack effects
9ec91e1
@gvanrossum
Array-ize LIST_APPEND
af785de
@gvanrossum
Array-ize SET_ADD
4fddf2b
@gvanrossum
Array-ize LIST_EXTEND
ac22115
@gvanrossum
Array-ize SET_UPDATE
6c83be0
@gvanrossum
Array-ize BUILD_TUPLE
c7999dd
@gvanrossum
Array-ize BUILD_LIST
a7aa425
@gvanrossum
Use ERROR_IF() in BUILD_STRING
11f6a58
@gvanrossum
Merge remote-tracking branch 'origin/main' into cases-array
597fd69
@gvanrossum gvanrossum added skip news 🔨 test-with-refleak-buildbots Test PR w/ refleak buildbots; report in status section labels Jan 10, 2023
@gvanrossum
gvanrossum requested a review from brandtbucher January 10, 2023 05:35
@bedevere-bot

bedevere-bot commented Jan 10, 2023

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @gvanrossum for commit 597fd69 🤖

If you want to schedule another build, you need to add the :hammer: test-with-refleak-buildbots label again.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot bedevere-bot added awaiting core review and removed 🔨 test-with-refleak-buildbots Test PR w/ refleak buildbots; report in status section labels Jan 10, 2023
@bedevere-bot bedevere-bot mentioned this pull request Jan 10, 2023
Generate the interpreter #98831
Closed
brandtbucher
brandtbucher reviewed Jan 12, 2023
View reviewed changes

@brandtbucher brandtbucher left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My computer is forcing a restart on me in two minutes, so here's what I've got. I still need to finish looking at test_generator.py and generate_cases.py tomorrow (there's a lot of new logic that I don't quite follow yet).

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Python/bytecodes.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Python/bytecodes.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Tools/cases_generator/generate_cases.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Tools/cases_generator/generate_cases.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Tools/cases_generator/generate_cases.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Tools/cases_generator/parser.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Tools/cases_generator/parser.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

gvanrossum added 6 commits January 11, 2023 21:57
@gvanrossum
Apply most suggestions from code review
abff4ae
@gvanrossum
Accept more general dimensions
a02a9eb
@gvanrossum
Fix LIST_EXTEND
2bcd90f
@gvanrossum
Fix LIST_APPEND, SET_ADD, SET_UPDATE
c579a40
@gvanrossum
Merge remote-tracking branch 'origin/main' into cases-array
21ac256
@gvanrossum
Array-ize BUILD_SET
70e983d
gvanrossum added 3 commits January 13, 2023 17:42
@gvanrossum
Array-ize BUILD_MAP
827481e
@gvanrossum
Array-ize BUILD_CONST_KEY_MAP
aa69c55
@gvanrossum
Add comment warning that RAISE_VARARGS needs to stay legacy
6cbabc7
@gvanrossum

gvanrossum commented Jan 14, 2023

Copy link
Copy Markdown
Member Author

@brandtbucher This should be ready for your review now. I don't intend to convert the remaining opcodes that have an array stack effect -- they either have an output array (for which this logic doesn't really work) or they are one of the main CALL specializations (wow there are a lot of those).

Let me know if I can explicate something for you.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

brandtbucher
brandtbucher reviewed Jan 17, 2023
View reviewed changes

@brandtbucher brandtbucher left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your patience. A few more things (mostly minor):

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Python/opcode_metadata.h Outdated
Comment on lines +48 to +49
[LIST_APPEND] = { -1, -1, DIR_NONE, DIR_NONE, DIR_NONE, true, INSTR_FMT_IX },
[SET_ADD] = { -1, -1, DIR_NONE, DIR_NONE, DIR_NONE, true, INSTR_FMT_IX },

@brandtbucher brandtbucher Jan 17, 2023

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All four of the changed opcodes in this file should be INSTR_FMT_IB, not INSTR_FMT_IX.

I'm guessing you search for "oparg" in the body of the instruction, but I think we need to now search for it in array dimensions too.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@gvanrossum gvanrossum Jan 17, 2023

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch. Fixed.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Tools/cases_generator/parser.py
Comment on lines +240 to +253
if not (dim := self.dimension()):
raise self.make_syntax_error("Expected dimension")
self.require(lx.RBRACKET)
return StackEffect(tkn.text, "PyObject **", dim.text.strip())
else:
return StackEffect(tkn.text)

@contextual
def dimension(self) -> Dimension | None:
tokens: list[lx.Token] = []
while (tkn := self.peek()) and tkn.kind != lx.RBRACKET:
tokens.append(tkn)
self.next()
return Dimension(lx.to_text(tokens).strip())

@brandtbucher brandtbucher Jan 17, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nitpicks:

dimension can't return None, right?

Suggested change
if not (dim := self.dimension()):
raise self.make_syntax_error("Expected dimension")
self.require(lx.RBRACKET)
return StackEffect(tkn.text, "PyObject **", dim.text.strip())
else:
return StackEffect(tkn.text)
@contextual
def dimension(self) -> Dimension | None:
tokens: list[lx.Token] = []
while (tkn := self.peek()) and tkn.kind != lx.RBRACKET:
tokens.append(tkn)
self.next()
return Dimension(lx.to_text(tokens).strip())
dim = self.dimension()
self.require(lx.RBRACKET)
return StackEffect(tkn.text, "PyObject **", dim.text.strip())
else:
return StackEffect(tkn.text)
@contextual
def dimension(self) -> Dimension:
tokens: list[lx.Token] = []
while (tkn := self.peek()) and tkn.kind != lx.RBRACKET:
tokens.append(tkn)
self.next()
return Dimension(lx.to_text(tokens).strip())

Also, I think the current definition of dimension allows an empty dimension (like stuff[]). Probably worth guarding against.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@gvanrossum gvanrossum Jan 17, 2023

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, but the correct fix to the code is for dimension() to check that it's got at least one token and return None otherwise.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Tools/cases_generator/test_generator.py
import tempfile

import generate_cases
from parser import StackEffect

@brandtbucher brandtbucher Jan 17, 2023

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Question about this file: is it supposed to run as part of our test suite, or is the workflow just to use pytest or something locally?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@gvanrossum gvanrossum Jan 17, 2023

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For now I just run it locally after I make changes.

I don't think our test suite supports pytest, and I don't want to make it a separate GitHub workflow. Eventually we'll have to rewrite it to use unittest. But not today.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

brandtbucher reacted with thumbs up emoji
All reactions
  • 👍 1 reaction
Comment thread Tools/cases_generator/test_generator.py
Comment on lines +365 to +366
STACK_SHRINK(oparg*2);
STACK_SHRINK(2);

@brandtbucher brandtbucher Jan 17, 2023

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm sort of surprised we don't combine these. Not that it matters much, though.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Tools/cases_generator/generate_cases.py Outdated
# The only supported output array forms are:
# - unused[...]
# - X[...] where X[...] matches an i99nput array form
self.emit(f"MOVE_ITEMS({src.name}, {dst.name}, {src.size});")

@brandtbucher brandtbucher Jan 17, 2023

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This feels backwards to me. My mental model is the memcpy argument order:

Suggested change
self.emit(f"MOVE_ITEMS({src.name}, {dst.name}, {src.size});")
self.emit(f"MOVE_ITEMS({dst.name}, {src.name}, {src.size});")

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Tools/cases_generator/generate_cases.py
Comment on lines +119 to +126
if isym and isym != osym:
self.emit(f"STACK_SHRINK({isym});")
if diff < 0:
self.emit(f"STACK_SHRINK({-diff});")
if diff > 0:
self.emit(f"STACK_GROW({diff});")
elif diff < 0:
self.emit(f"STACK_SHRINK({-diff});")
if osym and osym != isym:
self.emit(f"STACK_GROW({osym});")

@brandtbucher brandtbucher Jan 17, 2023

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See my comment earlier. It probably doesn't matter much, but it feels like we should be combining these into a single STACK_GROW/STACK_SHRINK per instruction.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@gvanrossum gvanrossum Jan 17, 2023

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought about that. There are some problematic theoretical cases where the generator would need to understand the symbolic sizes to be able to tell if it's a net grow or shrink operation, and it can't first do a grow/shrink of the numeric part only, since you could have e.g.

inst(FOO, (a, b[oparg] -- c, d))

which shrinks if oparg == 0 but grows for oparg >= 2. We cannot safely do this as grow(1); shrink(oparg) or shrink(oparg); grow(1), nor as grow(1 - oparg) nor as shrink(oparg - 1). It must be done as shrink(1 + oparg); grow(2), or use STACK_ADJUST(), but the latter doesn't do bounds checking in debug mode. The best way is shrink(oparg); grow(1), but I left it even simpler, as shrink(1); shrink(oparg); grow(2).

Honestly, the case analysis got hairy enough that I decided that this was okay to leave to future generations; the compiler will probably do a fine job optimizing shrink(a); shrink(b) to shrink(a + b).

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

brandtbucher reacted with thumbs up emoji
All reactions
  • 👍 1 reaction
Comment thread Tools/cases_generator/generate_cases.py Outdated
# NOTE: MOVE_ITEMS() does not actually exist.
# The only supported output array forms are:
# - unused[...]
# - X[...] where X[...] matches an i99nput array form

@brandtbucher brandtbucher Jan 17, 2023

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At first I thought this was something like "i18n" or "a11y". ;)

Suggested change
# - X[...] where X[...] matches an i99nput array form
# - X[...] where X[...] matches an input array form

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Tools/cases_generator/generate_cases.py Outdated
Comment on lines +257 to +258
for i in range(len(ieffects)):
ieffect = ieffects[i]

@brandtbucher brandtbucher Jan 17, 2023

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

;)

Suggested change
for i in range(len(ieffects)):
ieffect = ieffects[i]
for i, ieffect in enumerate(ieffects):

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Tools/cases_generator/generate_cases.py Outdated
Comment on lines +291 to +292
for i in range(len(oeffects)):
oeffect = oeffects[i]

@brandtbucher brandtbucher Jan 17, 2023

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
for i in range(len(oeffects)):
oeffect = oeffects[i]
for i, oeffect in enumerate(oeffects):

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Tools/cases_generator/generate_cases.py
Comment on lines +290 to +299
oeffects = list(reversed(self.output_effects))
for i in range(len(oeffects)):
oeffect = oeffects[i]
if oeffect.name in self.unmoved_names:
continue
osize = string_effect_size(list_effect_size(oeffects[:i+1]))
if oeffect.size:
dst = StackEffect(f"&PEEK({osize})", "PyObject **")
else:
dst = StackEffect(f"PEEK({osize})", "")

@brandtbucher brandtbucher Jan 17, 2023

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems like this whole block is identical to the logic for input effects. Maybe factor out into a function?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@gvanrossum gvanrossum Jan 17, 2023

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not excited about having to pick a name for a helper method that's five lines and takes three parameters (i, ieffect, ieffects).

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@brandtbucher brandtbucher Jan 17, 2023

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I missed the differences at the top and bottom of the for loops (I thought the entire loop could be factored out). No need to do this then.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
gvanrossum added 7 commits January 17, 2023 12:22
@gvanrossum
Look for 'oparg' in the whole instruction
c7cfe9a
@gvanrossum
Add docstring for effect_size()
a1a4688
@gvanrossum
Parenthesize certain symbolic effects
16a2d99
Anything that's not 'oparg' or 'oparg*2' will be parenthesized.
@gvanrossum
Fix typo in comment
1ef085d
@gvanrossum
Swap args of imaginary MOVE_ITEMS() macro
1bf42ce
@gvanrossum
Tune for loops
c2be1ab
@gvanrossum
Disallow empty dimension
c7edea2
@gvanrossum

gvanrossum commented Jan 17, 2023

Copy link
Copy Markdown
Member Author

If the buildbot tests pass I'll just merge it, unless you strenuously object.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@gvanrossum gvanrossum added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Jan 17, 2023
@bedevere-bot

bedevere-bot commented Jan 17, 2023

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @gvanrossum for commit c7edea2 🤖

If you want to schedule another build, you need to add the :hammer: test-with-buildbots label again.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Jan 17, 2023
brandtbucher
brandtbucher approved these changes Jan 17, 2023
View reviewed changes
@bedevere-bot bedevere-bot added awaiting merge and removed awaiting core review labels Jan 17, 2023
@gvanrossum

gvanrossum commented Jan 17, 2023

Copy link
Copy Markdown
Member Author

I'm just gonna land now. The "ARM64 Windows PR" buildbot is read, but it's been red for a week.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@gvanrossum
gvanrossum merged commit 80e3e34 into python:main Jan 17, 2023
@bedevere-bot bedevere-bot removed the awaiting merge label Jan 17, 2023
@gvanrossum
gvanrossum deleted the cases-array branch January 17, 2023 23:59
@kumaraditya303 kumaraditya303 mentioned this pull request Feb 16, 2023
Set: BUILD_SET opcode can be failed with segfault #101952
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@brandtbucher brandtbucher brandtbucher approved these changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

skip news

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

3 participants

@gvanrossum @bedevere-bot @brandtbucher
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.