Description

gvanrossum

opened on Oct 28, 2022

Issue body actions

Overview

Over in faster-cpython/ideas we've been exploring the idea of generating the interpreter from a set of instruction definitions. This will eventually enable having multiple versions of the interpreter (e.g. with and without tracing enabled), and it will allow us to automatically combine instructions using a powerful notation (e.g. super(LOAD_FAST__LOAD_FAST) = LOAD_FAST + LOAD_FAST;). It will also let us auto-generate things like stack_effect().

We are planning to land at least an early version of this work in 3.12. We have a tentative grammar for instruction definition DSL -- which will undoubtedly undergo several iterations before we've settled. We have a first draft of the tooling ready for review (which currently reproduces the status quo) done.

Once the first version of the tooling has landed we expect to iterate quickly, using this issue an umbrella issue for our PRs to link to.

References

• Generating code for the ceval loop/switch faster-cpython/ideas#5
• Super-instructions faster-cpython/ideas#16
• Define opcodes in terms of smaller "micro-ops" faster-cpython/ideas#454
• Change the interpreter definition DSL to be legal C syntax faster-cpython/ideas#477
• Mechanics of interpreter generation from DSL faster-cpython/ideas#479
• Stack effect of jumping -- how to express in DSL faster-cpython/ideas#481

• PR: GH-98831: "Generate" the interpreter #98830

• PR: GH-98831: Remove redundant extract_cases.py script #99065

• PR: GH-98831: Implement super-instruction generation #99084

• PR: GH-98831: Add some macros definitions to bytecodes.c to reduce IDE warnings. #99093

• PR: GH-98831: Auto-generate PREDICTED() macro calls #99102

• PR: GH-98831: Add regen-cases to regen-all #99107

• PR: GH-98831: Simple input-output stack effects #99120

• PR: GH-98831: Remove all remaining DISPATCH() calls from bytecodes.c #99271

• PR: GH-98831: Implement basic cache effects #99313

• PR: GH-98831: Refactor generate_cases.py #99408

• PR: GH-98831: Add macro and op and their implementation to DSL #99495

• PR: GH-98831: Refactor and fix cases generator #99526

• PR: GH-98831: Support cache effects in super- and macro instructions #99601

• PR: GH-98831: Typed stack effects, and more instructions converted #99764

• PR: GH-98831: Get rid of super(), just use macro() #100095

• PR: GH-98831: Generate things in the input order #100123

• PR: GH-98831: Remove super-instruction definitions, use macro instructions instead #100124

• PR: GH-98831: Add DECREF_INPUTS(), expanding to DECREF() each stack input #100205

• PR: GH-98831: Modernize a ton of simpler instructions #100545

• PR: GH-98831: Add some tests for generate_cases.py #100763

• PR: GH-98831: Update generate_cases.py: register inst, opcode_metadata.h #100735

• PR: GH-98831: Revert "GH-98831: Update generate_cases.py: register inst, opcode_metadata.h (#100735)" #100777

• PR: gh-98831: Regenerate Python/opcode_metadata.h #100778

• PR: GH-98831: Refactor instr format code and change to enum #100895

• PR: GH-98831: Implement array support in cases generator #100912

• PR: GH-98831: Identify instructions that don't use oparg #100957

• PR: GH-98831: Move assorted macros from ceval.h to a new header #101116

• PR: gh-98831: register instructions have 0 pushes and pops #101163

• PR: gh-98831: rewrite CHECK_EG_MATCH opcode in the instruction definition DSL #101269

• PR: gh-98831: rewrite pattern matching opcodes in the instruction definition DSL #101287

• PR: GH-98831: Elaborate some cases_generator tests #101299

• PR: gh-98831: rewrite RAISE_VARARGS in the instruction definition DSL #101306

• PR: gh-98831: add variable stack effect support to cases generator #101309

• PR: gh-98831: Support conditional effects; use for LOAD_ATTR #101333

• PR: gh-98831: Fix two bugs in case generator #101349

• PR: gh-98831: Clean up and add cache size static_assert to macro #101442

• PR: gh-98831: rewrite GET_LEN, GET_ITER, BEFORE_WITH and a few simple opcodes in the instruction definition DSL #101443

• PR: gh-98831: rewrite BEFORE_ASYNC_WITH and END_ASYNC_FOR in the instruction definition DSL #101458

• PR: gh-98831: rewrite PUSH_EXC_INFO and conditional jumps in the instruction definition DSL #101481

• PR: gh-98831: Modernize the LOAD_ATTR family #101488

• PR: gh-98831: Modernize the LOAD_GLOBAL family #101502

• PR: gh-98831: rewrite RERAISE and CLEANUP_THROW in the instruction definition DSL #101511

• PR: gh-98831: Modernize CALL and family #101508

• PR: gh-98831: rewrite SEND, GET_YIELD_FROM_ITER, RETURN_GENERATOR in the instruction definition DSL #101516

• PR: gh-98831: rewrite MAKE_FUNCTION and BUILD_SLICE in the instruction definition DSL #101529

• PR: gh-98831: rewrite COPY and SWAP in the instruction definition DSL #101620

• PR: gh-98831: Modernize the FOR_ITER family of instructions #101626

• PR: gh-98831: Modernize CALL_FUNCTION_EX #101627

• PR: gh-98831: Modernize FORMAT_VALUE #101628

• PR: gh-98831: Move DSL documentation here from ideas repo #101629

• PR: gh-98831: rewrite UNPACK_EX, UNPACK_SEQUENCE, UNPACK_SEQUENCE_TWO_TUPLE in the instruction definition DSL  #101641

• PR: gh-98831: Finish the UNPACK_SEQUENCE family #101666

• PR: gh-98831: Use opcode metadata for stack_effect() #101704

• PR: gh-98831: Use DECREF_INPUTS() more #102409

Reacted by Donghee Na, Kumar Aditya, Leonard Mosescu, Jaime R., Brandon T. Willard, Sakti Dwi Cahyono, Koopa, An Long, FredslundMagnus, Erlend E. Aasland and 2 more

Reacted by CF Bolz-Tereick, Łukasz Langa, Kumar Aditya, Shahriyar Rzayev, Jaime R., Sakti Dwi Cahyono, Antonio Hickey, Johan Dahlin, Nyuan Zhang, ProgramRipper and 3 more

Reacted by Dave, flow, Mickaël Schoentgen, Wilson Josix Wang, Lee Dogeon, 93578237, Alex Waygood, Dave Burkholder, Rodolfo De Nadai, Hoang Tran and 10 more

Activity

1. gvanrossum

   added

   type-featureA feature request or enhancementA feature request or enhancement

   on Oct 28, 2022
2. kumaraditya303

   added

   interpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)

   3.12only security fixesonly security fixes

   on Oct 29, 2022
3. gvanrossum

   moved this to Todo in Fancy CPython Boardon Oct 31, 2022
4. gvanrossum

   added this to Fancy CPython Boardon Oct 31, 2022
5. gvanrossum

   self-assigned this

   on Oct 31, 2022
6. gvanrossum

   changed the title [-]Generating the interpreter[/-] [+]Generate the interpreter[/+] on Oct 31, 2022
7. gvanrossum

   added 8 commits that reference this issue on Nov 3, 2022

   GH-98831: "Generate" the interpreter (#98830)

   41bc101

   GH-98831: Remove redundant extract_cases.py script (GH-99065)

   916af11

   GH-98831: Add `regen-cases` to `regen-all` (#99107)

   e99380c

   GH-98831: Auto-generate PREDICTED() macro calls (#99102)

   c885623

   GH-98831: Add some macros definitions to bytecodes.c to reduce IDE wa…

   7a020b8

   GH-98831: Implement super-instruction generation (#99084)

   7dcd28e

   GH-98831: Simple input-output stack effects for bytecodes.c (#99120)

   f1a6546

   GH-98831: Remove all remaining DISPATCH() calls from bytecodes.c (#99271

   694cdb2
8. 61 remaining items
9. iritkatriel

   added a commit that references this issue on Feb 7, 2023

   gh-98831: rewrite UNPACK_EX, UNPACK_SEQUENCE, UNPACK_SEQUENCE_TWO_TUP…

   dec1ab0
10. gvanrossum commented on Feb 7, 2023

    

    gvanrossum

    on Feb 7, 2023

    MemberAuthor

    More actions

    Once these PRs land, every opcode has cache/stack effect specifications, and I will disable the legacy syntax from the parser.

    • gh-98831: Modernize CALL and family #101508 CALL family
    • gh-98831: Modernize CALL_FUNCTION_EX #101627 CALL_FUNCTION_EX
    • gh-98831: Modernize FORMAT_VALUE #101628 FORMAT_VALUE
    • gh-98831: Finish the UNPACK_SEQUENCE family #101666 UNPACK_SEQUENCE family
11. gvanrossum

    added 5 commits that reference this issue on Feb 7, 2023

    gh-98831: Finish the UNPACK_SEQUENCE family (#101666)

    aacbdb0

    gh-98831: Modernize FORMAT_VALUE (#101628)

    b2b85b5

    gh-98831: Modernize CALL_FUNCTION_EX (#101627)

    a9f0144

    gh-98831: Modernize CALL and family (#101508)

    616aec1

    gh-98831: Use opcode metadata for stack_effect() (#101704)

    65b7b6b
12. carljm

    added a commit that references this issue on Feb 9, 2023

    Merge branch 'main' into inlinecomp2

    be3becc
13. gvanrossum

    added a commit that references this issue on Mar 13, 2023

    gh-98831: Use DECREF_INPUTS() more (#102409)

    392f2ad
14. carljm

    added a commit that references this issue on Mar 14, 2023

    Merge branch 'main' into inlinecomp2

    6c5f269
15. gvanrossum commented on Mar 15, 2023

    

    gvanrossum

    on Mar 15, 2023

    MemberAuthor

    More actions

    Let's close this issue, the interpreter is now definitely being generated. Subsequent changes may reference this (if small) or have their own issue.
16. gvanrossum

    closed this as completedon Mar 15, 2023
17. github-project-automation

    moved this from Todo to Done in Fancy CPython Boardon Mar 15, 2023
18. Fidget-Spinner

    added a commit that references this issue on Mar 27, 2023

    pythongh-98831: Use DECREF_INPUTS() more (python#102409)

    b99132d
19. warsaw

    added a commit that references this issue on Apr 11, 2023

    pythongh-98831: Use DECREF_INPUTS() more (python#102409)

    9cf44ab

Issue actions