GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — GH-97973: Return all necessary information from the tokenizer by lysnikolaou · Pull Request #97975 · python/cpython

Conversation

@lysnikolaou

@lysnikolaou lysnikolaou commented Oct 6, 2022 •
edited by bedevere-bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

Right now, the tokenizer only returns type and two pointers to the start and end of the token. This PR modifies the tokenizer to return the type and set all of the necessary information, so that the parser does not have to this.

  • Issue: Tokenizer returnings pointers make calculation of position info more complicated #97973

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@lysnikolaou
pythonGH-97973: Return all necessary information from the tokenizer
5f05ce9 
Right now, the tokenizer only returns type and two pointers to the
start and end of the token. This PR modifies the tokenizer to return
the type and set all of the necessary information, so that the parser
does not have to this.
@lysnikolaou
lysnikolaou requested a review from pablogsal as a code owner October 6, 2022 17:10
@bedevere-bot bedevere-bot added the awaiting core review label Oct 6, 2022
blurb-it Bot and others added 2 commits October 6, 2022 17:13
@blurb-it
📜🤖 Added by blurb_it.
f645664
@lysnikolaou
Merge branch 'main' into tokenizer-return-token
e53c4ea
@lysnikolaou lysnikolaou linked an issue Oct 6, 2022 that may be closed by this pull request
Tokenizer returnings pointers make calculation of position info more complicated #97973
Closed
lysnikolaou and others added 4 commits October 6, 2022 10:55
@lysnikolaou
Fix tests by using the correct defaults for line number
e892553
@lysnikolaou
Fix token_setup
0261165
@kumaraditya303 @lysnikolaou
pythonGH-88050: fix race in closing subprocess pipe in asyncio (pytho…
25c1004 
…n#97951)

Check for None when iterating over `self._pipes.values()`.
@mdboom @lysnikolaou
pythongh-94808: Coverage: Test that maximum indentation level is hand…
b3b300e 
…led (python#95926)

* pythongh-94808: Coverage: Test that maximum indentation level is handled

* Use "compile" rather than "exec"
@lysnikolaou
lysnikolaou force-pushed the tokenizer-return-token branch from cdfc30a to b3b300e Compare October 6, 2022 18:33
@lysnikolaou
lysnikolaou requested review from 1st1, asvetlov and gvanrossum as code owners October 6, 2022 18:33
@lysnikolaou

lysnikolaou commented Oct 6, 2022

Copy link
Copy Markdown
Member Author

Messed up my rebase. Sorry for the noise.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@lysnikolaou lysnikolaou closed this Oct 6, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pablogsal pablogsal Awaiting requested review from pablogsal pablogsal is a code owner
@1st1 1st1 Awaiting requested review from 1st1 1st1 is a code owner
@asvetlov asvetlov Awaiting requested review from asvetlov asvetlov is a code owner
@gvanrossum gvanrossum Awaiting requested review from gvanrossum

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

awaiting core review

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Tokenizer returnings pointers make calculation of position info more complicated

Uh oh!

There was an error while loading. Please reload this page.

4 participants

@lysnikolaou @bedevere-bot @kumaraditya303 @mdboom
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.