GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-97973: Return all necessary information from the tokenizer by lysnikolaou · Pull Request #97984 · python/cpython

Conversation

@lysnikolaou

@lysnikolaou lysnikolaou commented Oct 6, 2022 •
edited by bedevere-bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

Right now, the tokenizer only returns type and two pointers to the start and end of the token. This PR modifies the tokenizer to return the type and set all of the necessary information, so that the parser does not have to this.

  • Issue: Tokenizer returnings pointers make calculation of position info more complicated #97973

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@lysnikolaou
pythongh-97973: Return all necessary information from the tokenizer
0a7d0f6 
Right now, the tokenizer only returns type and two pointers to the start and end of the token.
This PR modifies the tokenizer to return the type and set all of the necessary information,
so that the parser does not have to this.
@lysnikolaou
lysnikolaou requested a review from pablogsal as a code owner October 6, 2022 18:44
@bedevere-bot bedevere-bot added the awaiting core review label Oct 6, 2022
@lysnikolaou lysnikolaou linked an issue Oct 6, 2022 that may be closed by this pull request
Tokenizer returnings pointers make calculation of position info more complicated #97973
Closed
lysnikolaou and others added 6 commits October 6, 2022 12:35
@lysnikolaou
Pass pointers so that struct does not get copied
c2320ba
@lysnikolaou
Merge branch 'main' into tokenizer-return-token-2
dee3359
@lysnikolaou
Remove news blurb for now
b28afed
@blurb-it
📜🤖 Added by blurb_it.
1e52242
@lysnikolaou
Merge branch 'main' into tokenizer-return-token-2
0aea48d
@lysnikolaou
Add assertion
689c670
pablogsal
pablogsal approved these changes Oct 6, 2022
View reviewed changes

@pablogsal pablogsal left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Great work 🚀

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@bedevere-bot bedevere-bot added awaiting merge and removed awaiting core review labels Oct 6, 2022
@lysnikolaou
lysnikolaou merged commit cbf0afd into python:main Oct 6, 2022
@bedevere-bot bedevere-bot removed the awaiting merge label Oct 6, 2022
@lysnikolaou lysnikolaou mentioned this pull request Oct 7, 2022
Use new tokenizer column offset to fix fstring ast locations we-like-parsers/cpython#195
Merged
carljm added a commit to carljm/cpython that referenced this pull request Oct 8, 2022
@carljm
Merge branch 'main' into typewatch
415ed49 
* main:
  pythonGH-97002: Prevent `_PyInterpreterFrame`s from backing more than one `PyFrameObject` (pythonGH-97996)
  pythongh-97973: Return all necessary information from the tokenizer (pythonGH-97984)
  fixes pythongh-96078: os.sched_yield release the GIL while calling sched_yield(2). (pythongh-97965)
  pythongh-65961: Do not rely solely on `__cached__` (pythonGH-97990)
  pythongh-97850: Remove the open issues section from the import reference (python#97935)
  Docs: pin sphinx-lint (pythonGH-97992)
  pythongh-94590: add signatures to operator itemgetter, attrgetter, methodcaller (python#94591)
  Add Pynche's move to the What's new in 3.11 (python#97974)
  pythongh-97781: Apply changes from importlib_metadata 5. (pythonGH-97785)
  pythongh-86482: Document assignment expression need for ()s (python#23291)
  pythongh-97943: PyFunction_GetAnnotations should return a borrowed reference. (python#97949)
mpage pushed a commit to mpage/cpython that referenced this pull request Oct 11, 2022
@lysnikolaou @mpage
pythongh-97973: Return all necessary information from the tokenizer (p…
078e3ed 
…ythonGH-97984)

Right now, the tokenizer only returns type and two pointers to the start and end of the token.
This PR modifies the tokenizer to return the type and set all of the necessary information,
so that the parser does not have to this.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pablogsal pablogsal pablogsal approved these changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Tokenizer returnings pointers make calculation of position info more complicated

Uh oh!

There was an error while loading. Please reload this page.

3 participants

@lysnikolaou @pablogsal @bedevere-bot
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.