GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — bpo-43950: Specialize tracebacks for subscripts/binary ops by isidentical · Pull Request #27037 · python/cpython

Conversation

@isidentical

@isidentical isidentical commented Jul 5, 2021 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

Examples;

 $ ./python t.py
Traceback (most recent call last):
  File "/home/isidentical/cpython/cpython/t.py", line 10, in <module>
    add_values(1, 2, 'x', 3, 4)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/isidentical/cpython/cpython/t.py", line 2, in add_values
    return a + b + c + d + e
           ~~~~~~^~~
TypeError: unsupported operand type(s) for +: 'int' and 'str'
Traceback (most recent call last):
  File "/home/isidentical/cpython/cpython/t2.py", line 10, in <module>
    response['data']['segment1']['segment2']['segment3']
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
TypeError: 'NoneType' object is not subscriptable

https://bugs.python.org/issue43950

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@the-knights-who-say-ni the-knights-who-say-ni added the CLA signed label Jul 5, 2021
@bedevere-bot bedevere-bot added the awaiting core review label Jul 5, 2021
@isidentical

isidentical commented Jul 5, 2021

Copy link
Copy Markdown
Member Author

For the traceback.py changes (and tests) I guess we should wait for @ammaraskar's changes on the traceback.py module.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@isidentical isidentical added the skip news label Jul 5, 2021
@isidentical
isidentical force-pushed the traceback-specialization branch from cc31c2d to 413d601 Compare July 5, 2021 18:06
iritkatriel
iritkatriel reviewed Jul 6, 2021
View reviewed changes
Comment thread Python/traceback.c Outdated

@iritkatriel iritkatriel Jul 6, 2021

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought from the title of the PR that it's related to Mark's bytecode specializations. Are you settled on using this word? Seems a bit overloaded now.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@isidentical isidentical Jul 6, 2021

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They are not relevant at all, but I can see the confusion. Any suggestions on alternative renaming?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@ammaraskar ammaraskar Jul 7, 2021

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here's how clang's documentation talks about their carets: https://clang.llvm.org/diagnostics.html

So following from there maybe something like Precision traceback or Pointing traceback because we're pointing to a specific part in the expression or trying to make it more precise?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@pablogsal

pablogsal commented Jul 7, 2021

Copy link
Copy Markdown
Member

@isidentical this may need rebasing now that the other PR is merged

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@pablogsal

pablogsal commented Jul 7, 2021

Copy link
Copy Markdown
Member

Let's focus on land this one next @ammaraskar @isidentical

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@isidentical

isidentical commented Jul 7, 2021

Copy link
Copy Markdown
Member Author

@ammaraskar after your _format_traceback (or even before) can you try to port this to the traceback.py? I think it is going to look something like this but haven't integrated printing logic / tests yet.

+def _extract_anchors_from_segment(segment):
+    import ast
+
+    tree = ast.parse(segment)
+    if len(tree.body) == 1:
+        statement = tree.body[0]
+        match statement:
+            case ast.Expr(expr):
+                match expr:
+                    case ast.BinOp():
+                        operator_str = segment[expr.left.end_col_offset:expr.right.col_offset]
+                        operator_offset = len(operator_str) - len(operator_str.lstrip())
+                        return operator_offset, operator_offset + 1
+                    case ast.Subscript():
+                        return expr.value.end_col_offset, expr.slice.col_offset
+    return -1, -1
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

ammaraskar
ammaraskar reviewed Jul 8, 2021
View reviewed changes
Comment thread Python/traceback.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

@ammaraskar
ammaraskar force-pushed the traceback-specialization branch from 94801f4 to 26e900e Compare July 8, 2021 18:11
@isidentical
isidentical marked this pull request as ready for review July 8, 2021 19:12
ammaraskar
ammaraskar reviewed Jul 8, 2021
View reviewed changes
Comment thread Lib/traceback.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

ammaraskar
ammaraskar reviewed Jul 8, 2021
View reviewed changes
Comment thread Python/traceback.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

isidentical and others added 2 commits July 10, 2021 12:05
@isidentical
bpo-43950: Specialize tracebacks for subscripts/binary ops
0a825ea
@ammaraskar @isidentical
Port new specializations to traceback.py
d79d365
Co-authored-by: Batuhan Taskaya <[email protected]>
@isidentical
isidentical force-pushed the traceback-specialization branch from d77011d to d79d365 Compare July 10, 2021 09:07
@isidentical
isidentical requested review from ammaraskar and pablogsal July 10, 2021 09:08
ammaraskar
ammaraskar approved these changes Jul 10, 2021
View reviewed changes

@ammaraskar ammaraskar left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just some basic stylistic suggestions mostly, looks good to me.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Python/traceback.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Python/traceback.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Python/traceback.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Python/traceback.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Python/traceback.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Python/traceback.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Python/traceback.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

@isidentical
pass segmen_str around
26430d4
@isidentical
isidentical force-pushed the traceback-specialization branch from 4a9a967 to 26430d4 Compare July 10, 2021 17:54
@isidentical isidentical added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Jul 10, 2021
@bedevere-bot

bedevere-bot commented Jul 10, 2021

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @isidentical for commit 26430d4 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Jul 10, 2021
@ammaraskar

ammaraskar commented Jul 11, 2021

Copy link
Copy Markdown
Member

Ping @pablogsal

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@isidentical

isidentical commented Jul 11, 2021

Copy link
Copy Markdown
Member Author

Apparently ASAN is failing, though not sure if it is relevant.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@isidentical isidentical closed this Jul 11, 2021
@isidentical isidentical reopened this Jul 11, 2021
pablogsal
pablogsal reviewed Jul 12, 2021
View reviewed changes
Comment thread Python/traceback.c Outdated
err = PyFile_WriteString(" ", f);
if (err < 0) {
goto done;
if (source_line) {

@pablogsal pablogsal Jul 12, 2021

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this guarded by source_line? How we can correctly calculate the offsets if we cannot get the line to calculate the characters? How can tHe code path when this is false and we continue with the bytes be corrent?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@pablogsal pablogsal Jul 12, 2021

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@isidentical This still applies, could you please check out what's going on here?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@pablogsal

pablogsal commented Jul 12, 2021

Copy link
Copy Markdown
Member

@isidentical I have pushed 081738d to greatly simplify the code structure. I think it reads better now. Please, check it out

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@isidentical

isidentical commented Jul 12, 2021

Copy link
Copy Markdown
Member Author

Thanks @pablogsal, it LGTM. Feel free to merge!

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@pablogsal

pablogsal commented Jul 12, 2021

Copy link
Copy Markdown
Member

@isidentical Please, check out #27037 (comment)

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@isidentical

isidentical commented Jul 12, 2021

Copy link
Copy Markdown
Member Author

Ah, I think that is not relevant. I just checked again, and it seems like source_line should be always non-NULL, though since we are dealing with value propagation over functions it might be still wiser to just replace the if (source_line) { ... } with assert(source_line); ....

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@isidentical

isidentical commented Jul 12, 2021

Copy link
Copy Markdown
Member Author

Also I am not really sure if we get any cases where primary/secondary will be different, but if so I think it would be nicer to make amendments to the traceback.py logic to do the same.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@pablogsal

pablogsal commented Jul 12, 2021

Copy link
Copy Markdown
Member

Also I am not really sure if we get any cases where primary/secondary will be different, but if so I think it would be nicer to make amendments to the traceback.py logic to do the same.

In this PR or in a different one?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@pablogsal

pablogsal commented Jul 12, 2021

Copy link
Copy Markdown
Member

Ah, I think that is not relevant. I just checked again, and it seems like source_line should be always non-NULL, though since we are dealing with value propagation over functions it might be still wiser to just replace the if (source_line) { ... } with assert(source_line); ....

We should also add some tests where we fail to parse the source. Is this covered currently?

isidentical reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@isidentical

isidentical commented Jul 12, 2021

Copy link
Copy Markdown
Member Author

In this PR or in a different one?

This PR.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@pablogsal

pablogsal commented Jul 12, 2021

Copy link
Copy Markdown
Member

Also I am not really sure if we get any cases where primary/secondary will be different, but if so I think it would be nicer to make amendments to the traceback.py logic to do the same.

Also, I am not very sure what you refer to with "will be different". Can you elaborate?

We should also add some tests where we fail to parse the source. Is this covered currently?

I don't see this covered. Could you add a test for this?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@isidentical

isidentical commented Jul 12, 2021 via email

Copy link
Copy Markdown
Member Author
Also, I am not very sure what you refer to with "will be different". Can
you elaborate? With the latest commit you pass primary/secondary (^/~) characters to the AST visitors. But for both of the cases they get assigned the same values so that is what I meant by 'will be different' (are there any other specializations that you plan would change the values for these 2 variables). I am currently unavailable to add a test, but can try to write something off tonight.
…
On Mon, Jul 12, 2021, 3:38 PM Pablo Galindo Salgado < ***@***.***> wrote: Also I am not really sure if we get any cases where primary/secondary will be different, but if so I think it would be nicer to make amendments to the traceback.py logic to do the same. Also, I am not very sure what you refer to with "will be different". Can you elaborate? We should also add some tests where we fail to parse the source. Is this covered currently? I don't see this covered. Could you add a test for this? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#27037 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALJKHQJO6MPLFWJU5R75MZ3TXLO37ANCNFSM473CEDAA> .
pablogsal reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

ammaraskar
ammaraskar reviewed Jul 12, 2021
View reviewed changes
Comment thread Lib/traceback.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

@isidentical
isidentical force-pushed the traceback-specialization branch from 5d2036a to a67c8aa Compare July 12, 2021 17:28
@isidentical
make amendments
045c491
@isidentical
isidentical force-pushed the traceback-specialization branch from a67c8aa to 045c491 Compare July 12, 2021 18:03
ammaraskar
ammaraskar reviewed Jul 12, 2021
View reviewed changes
Comment thread Lib/test/test_traceback.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

ammaraskar added 2 commits July 12, 2021 14:14
@ammaraskar
Use TESTFN instead of NamedTemporaryFile
012ae63
@ammaraskar
Remove tempfile import
825e3b0
@ammaraskar

ammaraskar commented Jul 12, 2021

Copy link
Copy Markdown
Member

We now have a test for when parsing fails and the source_line conditional is gone. Anything else to do here?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

pablogsal
pablogsal approved these changes Jul 12, 2021
View reviewed changes
@bedevere-bot bedevere-bot added awaiting merge and removed awaiting core review labels Jul 12, 2021
@pablogsal
pablogsal merged commit 1890dd2 into python:main Jul 12, 2021
@bedevere-bot bedevere-bot removed the awaiting merge label Jul 12, 2021
@pablogsal

pablogsal commented Jul 12, 2021

Copy link
Copy Markdown
Member

We now have a test for when parsing fails and the source_line conditional is gone. Anything else to do here?

Yes, the most important step: celebrate 🎉

(also pray for the buildbots to not fail 😉 )

ammaraskar reacted with laugh emoji
All reactions
  • 😄 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@pablogsal
pablogsal deleted the traceback-specialization branch July 12, 2021 19:33
@ammaraskar ammaraskar mentioned this pull request Jul 12, 2021
bpo-43950: Add documentation for PEP-657 #27047
Merged
jeanas added a commit to jeanas/pygments that referenced this pull request Dec 28, 2021
@jeanas
Update for new Python traceback locations
17ae881
Since python/cpython#27037, they can include
tildes in addition to the carets.
@jeanas jeanas mentioned this pull request Dec 28, 2021
Update for new Python traceback locations pygments/pygments#2004
Merged
Anteru pushed a commit to pygments/pygments that referenced this pull request Dec 29, 2021
@jeanas
Update for new Python traceback locations (#2004)
4d0a7d7
Since python/cpython#27037, they can include
tildes in addition to the carets.
@pablogsal pablogsal mentioned this pull request Apr 13, 2022
Include column offsets for bytecode instructions #88116
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ammaraskar ammaraskar ammaraskar approved these changes
@iritkatriel iritkatriel iritkatriel left review comments
@pablogsal pablogsal pablogsal approved these changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

skip news

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

6 participants

@isidentical @pablogsal @bedevere-bot @ammaraskar @iritkatriel @the-knights-who-say-ni
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.