GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-127833: Docs: Add a `grammar-snippet` directive & replace `productionlist` by encukou · Pull Request #127835 · python/cpython

Conversation

@encukou

@encukou encukou commented Dec 11, 2024 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

As a first step to what's proposed in the issue, this overrides the ReST productionlist directive to:

  • use : instead of the ::= symbol
  • add syntax highlighting for strings (the implementation is preliminary; I intend to later add a proper class to the theme instead of borrowing a Pygments class.)

All links and link targets should be preserved. (Unfortunately, this reaches into some Sphinx internals; I don't see a better way to do exactly what Sphinx does.)

This also adds a new directive, grammar-snippet, which formats the snippet almost exactly like what's in the source, modulo syntax highlighting and keeping the backtick character to mark links to other rules.
This will allow formatting the snippets as in the grammar file (most importantly: to remove the the indentation that Sphinx adds to align the text).

I'd appreciate if a Sphinx expert could take a look to see if there's a better way to do something :) cc @AA-Turner

  • Issue: Align the grammar documentation with Python's actual grammar #127833

📚 Documentation preview 📚: https://cpython-previews--127835.org.readthedocs.build/

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

willingc reacted with hooray emoji
All reactions
  • 🎉 1 reaction
encukou and others added 6 commits November 20, 2024 17:55
@encukou @blaisep @wqferr
First version of GrammarSnippetDirective, copied from an earlier branch
86a0ecc
Co-authored-by: Blaise Pabon <[email protected]>
Co-authored-by: William Ferreira <[email protected]>
@blaisep
Add a grammar-snippet directive
f6ffb21
@blaisep
Remove monkey patch for production list and Add CompatProductionList
5cfc701
to conserve the productionlist usage in the docs.
@encukou
Fix ReST grammar
f0cbed8
@encukou @johnslavik
Complete the CompatProductionList class
9139e25
Co-Authored-By: bswck <[email protected]>
@encukou
Adjust docstrings & move GrammarSnippetDirective class up
b10d78b
@encukou
encukou requested a review from willingc as a code owner December 11, 2024 16:59
@bedevere-app bedevere-app Bot mentioned this pull request Dec 11, 2024
Align the grammar documentation with Python's actual grammar #127833
Open
@bedevere-app bedevere-app Bot added the awaiting core review label Dec 11, 2024
encukou added 2 commits December 11, 2024 18:04
@encukou
Merge in the main branch
c18a4d3
@encukou
Run Ruff
a403fb6
@encukou encukou added the skip news label Dec 11, 2024
@encukou
Run Ruff again with different flags
074c189
@encukou
encukou requested a review from AA-Turner December 11, 2024 17:11
AA-Turner
AA-Turner reviewed Jan 15, 2025
View reviewed changes
Comment thread Doc/reference/toplevel_components.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

AA-Turner
AA-Turner reviewed Jan 15, 2025
View reviewed changes

@AA-Turner AA-Turner left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few initial notes. The targets problem is annoying, Sphinx doesn't make it easy...

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

encukou reacted with thumbs up emoji
All reactions
  • 👍 1 reaction
Comment thread Doc/tools/extensions/grammar_snippet.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Doc/tools/extensions/grammar_snippet.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Doc/tools/extensions/grammar_snippet.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Doc/tools/extensions/grammar_snippet.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

encukou added 2 commits January 22, 2025 16:39
@encukou
Merge in the main branch
b2fb819
@encukou
Turn the common function into a class with method; break make_link_to…
ea9e2f0
…_token out
@encukou
encukou requested a review from hugovk as a code owner January 22, 2025 15:47
encukou and others added 9 commits January 22, 2025 16:50
@encukou
Remove blank line
b2205a8
@encukou
Run ruff format
829e4f0
@encukou
Add forgotten arguments
e57a37e
@encukou
Merge in the main branch
fca04c7
@encukou
Use Sphinx's token_xrefs function for formatting the tokens
2732143
@encukou
Improve the highlight class injection
ef3c552
@encukou
Appease Ruff
c0c8432
@encukou
Appase Ruff again
e2359b7
@AA-Turner
Add type annotations, minor clean-ups
3fcb730
- Add a module docstring
- Use the snake_case convention for node classes
- Make :group: a required option
- declare parallel_write_safe = True
AA-Turner
AA-Turner reviewed Jan 29, 2025
View reviewed changes
Comment thread Doc/tools/extensions/grammar_snippet.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Doc/tools/extensions/grammar_snippet.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Doc/tools/extensions/grammar_snippet.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

AA-Turner
AA-Turner approved these changes Jan 29, 2025
View reviewed changes

@AA-Turner AA-Turner left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm happy with this as it stands, thank you @encukou for the discussion. As discussed, I've pushed a commit to add type annotations. I've included some other minor changes and left a self-review for a brief explanation -- feel free to revert any you disagree with.

A

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

encukou reacted with heart emoji
All reactions
  • ❤️ 1 reaction
@bedevere-app bedevere-app Bot added awaiting merge and removed awaiting core review labels Jan 29, 2025
@AA-Turner AA-Turner added docs Documentation in the Doc dir needs backport to 3.12 only security fixes needs backport to 3.13 only security fixes labels Jan 29, 2025
@encukou
encukou merged commit 58a4357 into python:main Feb 5, 2025
@miss-islington-app

miss-islington-app Bot commented Feb 5, 2025

Copy link
Copy Markdown

Thanks @encukou for the PR 🌮🎉.. I'm working now to backport this PR to: 3.12, 3.13.
🐍🍒⛏🤖

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-app bedevere-app Bot removed the awaiting merge label Feb 5, 2025
@encukou
encukou deleted the doc-grammar-snippets branch February 5, 2025 15:12
miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Feb 5, 2025
@encukou @blaisep
@wqferr @johnslavik @AA-Turner @miss-islington
pythongh-127833: Docs: Add a grammar-snippet directive & replace `p…
0f80ab2
…roductionlist` (pythonGH-127835)

As a first step toward aligning the grammar documentation with Python's actual
grammar, this overrides the ReST `productionlist` directive to:
- use `:` instead of the `::=` symbol
- add syntax highlighting for strings (using a Pygments highlighting class)

All links and link targets should be preserved. (Unfortunately, this reaches
into some Sphinx internals; I don't see a better way to do exactly what
Sphinx does.)

This also adds a new directive, `grammar-snippet`, which formats the snippet
almost exactly like what's in the source, modulo syntax highlighting and
keeping the backtick character to mark links to other rules.
This will allow formatting the snippets as in the grammar file
(file:///home/encukou/dev/cpython/Doc/build/html/reference/grammar.html).

The new directive is applied to two simple rules in toplevel_components.rst

---------
(cherry picked from commit 58a4357)

Co-authored-by: Petr Viktorin <[email protected]>
Co-authored-by: Blaise Pabon <[email protected]>
Co-authored-by: William Ferreira <[email protected]>
Co-authored-by: bswck <[email protected]>
Co-authored-by: Adam Turner <[email protected]>
@bedevere-app

bedevere-app Bot commented Feb 5, 2025

Copy link
Copy Markdown

GH-129689 is a backport of this pull request to the 3.13 branch.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-app bedevere-app Bot removed the needs backport to 3.13 only security fixes label Feb 5, 2025
miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Feb 5, 2025
@encukou @blaisep
@wqferr @johnslavik @AA-Turner @miss-islington
pythongh-127833: Docs: Add a grammar-snippet directive & replace `p…
14991a4
…roductionlist` (pythonGH-127835)

As a first step toward aligning the grammar documentation with Python's actual
grammar, this overrides the ReST `productionlist` directive to:
- use `:` instead of the `::=` symbol
- add syntax highlighting for strings (using a Pygments highlighting class)

All links and link targets should be preserved. (Unfortunately, this reaches
into some Sphinx internals; I don't see a better way to do exactly what
Sphinx does.)

This also adds a new directive, `grammar-snippet`, which formats the snippet
almost exactly like what's in the source, modulo syntax highlighting and
keeping the backtick character to mark links to other rules.
This will allow formatting the snippets as in the grammar file
(file:///home/encukou/dev/cpython/Doc/build/html/reference/grammar.html).

The new directive is applied to two simple rules in toplevel_components.rst

---------
(cherry picked from commit 58a4357)

Co-authored-by: Petr Viktorin <[email protected]>
Co-authored-by: Blaise Pabon <[email protected]>
Co-authored-by: William Ferreira <[email protected]>
Co-authored-by: bswck <[email protected]>
Co-authored-by: Adam Turner <[email protected]>
@bedevere-app

bedevere-app Bot commented Feb 5, 2025

Copy link
Copy Markdown

GH-129690 is a backport of this pull request to the 3.12 branch.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-app bedevere-app Bot removed the needs backport to 3.12 only security fixes label Feb 5, 2025
srinivasreddy pushed a commit to srinivasreddy/cpython that referenced this pull request Feb 7, 2025
@encukou @blaisep
@wqferr @johnslavik @AA-Turner @srinivasreddy
pythongh-127833: Docs: Add a grammar-snippet directive & replace `p…
9b37336
…roductionlist` (pythonGH-127835)

As a first step toward aligning the grammar documentation with Python's actual
grammar, this overrides the ReST `productionlist` directive to:
- use `:` instead of the `::=` symbol
- add syntax highlighting for strings (using a Pygments highlighting class)

All links and link targets should be preserved. (Unfortunately, this reaches
into some Sphinx internals; I don't see a better way to do exactly what
Sphinx does.)

This also adds a new directive, `grammar-snippet`, which formats the snippet
almost exactly like what's in the source, modulo syntax highlighting and
keeping the backtick character to mark links to other rules.
This will allow formatting the snippets as in the grammar file
(file:///home/encukou/dev/cpython/Doc/build/html/reference/grammar.html).

The new directive is applied to two simple rules in toplevel_components.rst

---------

Co-authored-by: Blaise Pabon <[email protected]>
Co-authored-by: William Ferreira <[email protected]>
Co-authored-by: bswck <[email protected]>
Co-authored-by: Adam Turner <[email protected]>
cmaloney pushed a commit to cmaloney/cpython that referenced this pull request Feb 8, 2025
@encukou @blaisep
@wqferr @johnslavik @AA-Turner @cmaloney
pythongh-127833: Docs: Add a grammar-snippet directive & replace `p…
9fd0c7c
…roductionlist` (pythonGH-127835)

As a first step toward aligning the grammar documentation with Python's actual
grammar, this overrides the ReST `productionlist` directive to:
- use `:` instead of the `::=` symbol
- add syntax highlighting for strings (using a Pygments highlighting class)

All links and link targets should be preserved. (Unfortunately, this reaches
into some Sphinx internals; I don't see a better way to do exactly what
Sphinx does.)

This also adds a new directive, `grammar-snippet`, which formats the snippet
almost exactly like what's in the source, modulo syntax highlighting and
keeping the backtick character to mark links to other rules.
This will allow formatting the snippets as in the grammar file
(file:///home/encukou/dev/cpython/Doc/build/html/reference/grammar.html).

The new directive is applied to two simple rules in toplevel_components.rst

---------

Co-authored-by: Blaise Pabon <[email protected]>
Co-authored-by: William Ferreira <[email protected]>
Co-authored-by: bswck <[email protected]>
Co-authored-by: Adam Turner <[email protected]>
@AA-Turner AA-Turner mentioned this pull request Feb 10, 2025
Remove productionlist hard-coding in translators sphinx-doc/sphinx#13326
Merged
@encukou encukou mentioned this pull request May 13, 2026
Link to existing rules in compound_stmts.rst #149811
Merged
encukou added a commit that referenced this pull request May 14, 2026
@encukou
Link to existing rules in compound_stmts.rst (GH-149811)
c375292
In gh-138418, `!` was added to links to rules that don't exist in
the docs, in order to silence broken link warnings.
However, productionlist doesn't parse the `!`, which ends up in
the rendered documentation. (It's possible that gh-127835 broke
the `!` support.)

Replace the names with ones that appear in docs:

- `star_named_expression` in the grammar corresponds to
  `flexible_expression` in the docs
- `star_named_expressions` in the grammar corresponds to
  `flexible_expression_list` in the docs
- `named_expression` in the grammar corresponds to
  `assignment_expression` in the docs

Having two sets of names isn't great of course. Consolidating them
is tracked in (subissues of) gh-127833.
This was referenced May 14, 2026
[3.15] Link to existing rules in compound_stmts.rst (GH-149811) #149836
Merged
[3.14] Link to existing rules in compound_stmts.rst (GH-149811) #149837
Merged
[3.13] Link to existing rules in compound_stmts.rst (GH-149811) #149838
Merged
encukou added a commit that referenced this pull request May 14, 2026
@miss-islington @encukou
[3.15] Link to existing rules in compound_stmts.rst (GH-149811) (GH-1…
451f06b
…49836)

Link to existing rules in compound_stmts.rst (GH-149811)

In gh-138418, `!` was added to links to rules that don't exist in
the docs, in order to silence broken link warnings.
However, productionlist doesn't parse the `!`, which ends up in
the rendered documentation. (It's possible that gh-127835 broke
the `!` support.)

Replace the names with ones that appear in docs:

- `star_named_expression` in the grammar corresponds to
  `flexible_expression` in the docs
- `star_named_expressions` in the grammar corresponds to
  `flexible_expression_list` in the docs
- `named_expression` in the grammar corresponds to
  `assignment_expression` in the docs

Having two sets of names isn't great of course. Consolidating them
is tracked in (subissues of) gh-127833.
(cherry picked from commit c375292)

Co-authored-by: Petr Viktorin <[email protected]>
encukou added a commit that referenced this pull request May 14, 2026
@miss-islington @encukou
[3.14] Link to existing rules in compound_stmts.rst (GH-149811) (GH-1…
9a1f30f
…49837)

Link to existing rules in compound_stmts.rst (GH-149811)

In gh-138418, `!` was added to links to rules that don't exist in
the docs, in order to silence broken link warnings.
However, productionlist doesn't parse the `!`, which ends up in
the rendered documentation. (It's possible that gh-127835 broke
the `!` support.)

Replace the names with ones that appear in docs:

- `star_named_expression` in the grammar corresponds to
  `flexible_expression` in the docs
- `star_named_expressions` in the grammar corresponds to
  `flexible_expression_list` in the docs
- `named_expression` in the grammar corresponds to
  `assignment_expression` in the docs

Having two sets of names isn't great of course. Consolidating them
is tracked in (subissues of) gh-127833.
(cherry picked from commit c375292)

Co-authored-by: Petr Viktorin <[email protected]>
encukou added a commit that referenced this pull request May 14, 2026
@miss-islington @encukou
[3.13] Link to existing rules in compound_stmts.rst (GH-149811) (GH-1…
4da3e91
…49838)

Link to existing rules in compound_stmts.rst (GH-149811)

In gh-138418, `!` was added to links to rules that don't exist in
the docs, in order to silence broken link warnings.
However, productionlist doesn't parse the `!`, which ends up in
the rendered documentation. (It's possible that gh-127835 broke
the `!` support.)

Replace the names with ones that appear in docs:

- `star_named_expression` in the grammar corresponds to
  `flexible_expression` in the docs
- `star_named_expressions` in the grammar corresponds to
  `flexible_expression_list` in the docs
- `named_expression` in the grammar corresponds to
  `assignment_expression` in the docs

Having two sets of names isn't great of course. Consolidating them
is tracked in (subissues of) gh-127833.
(cherry picked from commit c375292)

Co-authored-by: Petr Viktorin <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@AA-Turner AA-Turner AA-Turner approved these changes
@willingc willingc Awaiting requested review from willingc willingc is a code owner
@hugovk hugovk Awaiting requested review from hugovk hugovk is a code owner

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

docs Documentation in the Doc dir skip news

Projects

Docs PRs
Status: Done

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

4 participants

@encukou @hugovk @AA-Turner @blaisep
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.