GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — GH-94694: Fix column offsets for multi-line method lookups by brandtbucher · Pull Request #94697 · python/cpython

Conversation

@brandtbucher

@brandtbucher brandtbucher commented Jul 8, 2022 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

If we're given an AST that doesn't make much sense here, just wipe out the column info instead of crashing later.

Also, don't try to include the dot in the column span.

  • Issue: 3.11 branch: column >= -1 assertion failure with pytest rewriting #94694

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
brandtbucher added 5 commits July 8, 2022 16:16
@brandtbucher
Add failing regression test
937a3db
@brandtbucher
Fix invalid column math
dd5a391
@brandtbucher
Add additional tests for column locations
ffd3413
@brandtbucher
blurb add
53ba444
@brandtbucher
Update comment
33dc4d8
@brandtbucher brandtbucher added interpreter-core (Objects, Python, Grammar, and Parser dirs) needs backport to 3.11 only security fixes labels Jul 8, 2022
@brandtbucher
brandtbucher requested a review from pablogsal July 8, 2022 23:51
@brandtbucher brandtbucher self-assigned this Jul 8, 2022
@brandtbucher
brandtbucher requested review from iritkatriel and markshannon as code owners July 8, 2022 23:51
@bedevere-bot bedevere-bot added the awaiting core review label Jul 8, 2022
@brandtbucher brandtbucher added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Jul 9, 2022
@bedevere-bot

bedevere-bot commented Jul 9, 2022

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @brandtbucher for commit 33dc4d8 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Jul 9, 2022
@brandtbucher

brandtbucher commented Jul 9, 2022

Copy link
Copy Markdown
Member Author

wasm32 buildbot failures are unrelated: #94549 (comment)

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

iritkatriel
iritkatriel reviewed Jul 9, 2022
View reviewed changes
Comment thread Lib/test/test_compile.py Outdated

def test_multiline_assert_rewritten_as_method_call(self):
# GH-94694: Copying location information from a "real" node to a
# handwritten one should always be valid!

@iritkatriel iritkatriel Jul 9, 2022

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should it really always be valid?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@brandtbucher brandtbucher Jul 9, 2022

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can remove the comment if it's too controversial. It just seems to me that the utility/safety of ast.copy_location and ast.fix_missing_locations is dramatically reduced if they can't at least promise crash-free behavior.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Python/compile.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Python/compile.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

@pablogsal

pablogsal commented Jul 9, 2022 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

Is a bit unfortunate that we cannot just propagate correctly the offsets from the AST nodes. The disadvantage of a method relying on inspecting the source (here through the method name) is that ASTs created from abstract sources (or no source at all) may fall through the cracks in these kind of checks.

What's preventing us from correctly grab the original column offsets from the multi line method call? (I know this has more to be with the original issue, but I would like to fully understand this before adding more band aids :) )

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@iritkatriel

iritkatriel commented Jul 9, 2022

Copy link
Copy Markdown
Member

The issue here is that we are flattening a multi line location into a single line. So we are trying to construct the location info for the attribute name only (which is not available directly in the AST, it seems).

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@pablogsal

pablogsal commented Jul 9, 2022

Copy link
Copy Markdown
Member

The issue here is that we are flattening a multi line location into a single line. So we are trying to construct the location info for the attribute name only (which is not available directly in the AST, it seems).

For our own nodes, I can fix that in the parser so our nodes make those available. For random AST nodes it may still be a problem.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@iritkatriel

iritkatriel commented Jul 9, 2022

Copy link
Copy Markdown
Member

The issue here is that we are flattening a multi line location into a single line. So we are trying to construct the location info for the attribute name only (which is not available directly in the AST, it seems).

For our own nodes, I can fix that in the parser so our nodes make those available. For random AST nodes it may still be a problem.

I think we almost have it. The location of meth->v.Attribute seems to be the dot plus the attr name. Is that right? Can you exclude the dot?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@iritkatriel

iritkatriel commented Jul 9, 2022

Copy link
Copy Markdown
Member

For random AST nodes it may still be a problem.

The pytest nodes are not completely random, if we copy to them the location of the attr name from the source code the traceback will probably look right - the source line from the file will appear with the right thing hilighted.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@brandtbucher

brandtbucher commented Jul 9, 2022

Copy link
Copy Markdown
Member Author

What's preventing us from correctly grab the original column offsets from the multi line method call?

The root of the issue is that attr is stored as an identifier (string) in the AST. If it was a full Name node, this whole hack could just be replaced with a single SET_LOC.

I think we almost have it. The location of meth->v.Attribute seems to be the dot plus the attr name. Is that right? Can you exclude the dot?

Unfortunately, Attribute nodes include the entire span of the attribute access, including everything to the left of the dot.

This also creates a weird (but probably fine) inconsistency. If the attribute access is all on one line, the whole thing is underlined in the traceback. If it spans multiple lines, only the attribute name is underlined.

All this just comes from our need to trace the line that the attribute name is on, and trying (after the fact) to figure out what columns that should include. :(

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@pablogsal

pablogsal commented Jul 9, 2022

Copy link
Copy Markdown
Member

The root of the issue is that attr is stored as an identifier (string) in the AST. If it was a full Name node, this whole hack could just be replaced with a single SET_LOC.

This is exactly what I'm referring to. We could modify the AST to add this information so we don't need to guess. This way, it should also have work with arbitrary AST nodes.

The price is that although this is something we can do, it will force everyone to adapt.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@brandtbucher

brandtbucher commented Jul 9, 2022

Copy link
Copy Markdown
Member Author

Yeah. I think it would be an acceptable change for 3.12. While some consumers of the AST would need to change their code slightly, it wouldn't be removing any functionality that's currently present. If anything, it's adding functionality.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@brandtbucher
Feedback from code review
13c07ff
@brandtbucher
brandtbucher merged commit 264b3dd into python:main Jul 10, 2022
@miss-islington

miss-islington commented Jul 10, 2022

Copy link
Copy Markdown
Contributor

Thanks @brandtbucher for the PR 🌮🎉.. I'm working now to backport this PR to: 3.11.
🐍🍒⛏🤖

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot bedevere-bot removed the awaiting core review label Jul 10, 2022
@miss-islington

miss-islington commented Jul 10, 2022

Copy link
Copy Markdown
Contributor

Sorry, @brandtbucher, I could not cleanly backport this to 3.11 due to a conflict.
Please backport using cherry_picker on command line.
cherry_picker 264b3ddfd561d97204ffb30be6a7d1fb0555e560 3.11

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@brandtbucher

brandtbucher commented Jul 10, 2022

Copy link
Copy Markdown
Member Author

I think this one's gonna need a manual backport.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

brandtbucher added a commit to brandtbucher/cpython that referenced this pull request Jul 10, 2022
@brandtbucher
[3.11] pythonGH-94694: Fix column offsets for multi-line method looku…
eaa2391
…ps (pythonGH-94697).

(cherry picked from commit 264b3dd)

Co-authored-by: Brandt Bucher <[email protected]>
@bedevere-bot bedevere-bot removed the needs backport to 3.11 only security fixes label Jul 10, 2022
@bedevere-bot

bedevere-bot commented Jul 10, 2022

Copy link
Copy Markdown

GH-94721 is a backport of this pull request to the 3.11 branch.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@brandtbucher
brandtbucher deleted the fix-method-cols branch July 21, 2022 20:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@iritkatriel iritkatriel iritkatriel left review comments
@pablogsal pablogsal Awaiting requested review from pablogsal
@markshannon markshannon Awaiting requested review from markshannon markshannon is a code owner

Uh oh!

There was an error while loading. Please reload this page.

Assignees

@brandtbucher brandtbucher

Labels

interpreter-core (Objects, Python, Grammar, and Parser dirs)

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

5 participants

@brandtbucher @bedevere-bot @pablogsal @iritkatriel @miss-islington
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402520 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.