GeezerVison

The World Wide Web, restored to 800 pixels of glory.

GeezerVison is showing github.com — Implement _tokenize and update tokenize from v3.14.3 by youknowone · Pull Request #7392 · RustPython/RustPython

Implement _tokenize and update tokenize from v3.14.3 #7392

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

youknowone merged 3 commits into RustPython:main from youknowone:update-tokenize

Mar 10, 2026

Uh oh!

There was an error while loading. Please reload this page.

Merged

Implement _tokenize and update tokenize from v3.14.3 #7392

Show file tree

Hide file tree

Changes from 1 commit

Commits

Show all changes

3 commits

Select commit Hold shift + click to select a range

5c5aff8

Base implementation of _tokenize module

ShaharNaveh Mar 6, 2026

bf2b993

Update tokenize from v3.14.3

Mar 6, 2026

55737ed

Rewrite _tokenize with 2-phase model

youknowone Mar 9, 2026

File filter

Filter by extension .py (2) .rs (3) All 2 file types selected

Deleted files Viewed files

Clear filters

Conversations

Failed to load comments.

Uh oh!

There was an error while loading. Please reload this page.

Jump to

Jump to file

Failed to load files.

Uh oh!

There was an error while loading. Please reload this page.

Diff view

Unified

Split

Hide whitespace

Diff view

Unified

Split

Hide whitespace

Previous commit

Rewrite _tokenize with 2-phase model

Replace per-line reparsing with single-pass tokenization:
- Read all lines via readline, parse once, yield tokens
- Fix token type values (COMMENT=65, NL=66, OP=55)
- Fix NEWLINE/NL end positions and implicit newline handling
- Fix DEDENT positions via look-ahead to next non-DEDENT token
- Handle FSTRING_MIDDLE brace unescaping ({{ → {, }} → })
- Emit implicit NL before ENDMARKER when source lacks trailing newline
- Raise IndentationError from lexer errors
- Remove 13 expectedFailure marks for now-passing tests

Loading branch information

Uh oh!

There was an error while loading. Please reload this page.

youknowone committed Mar 10, 2026

commit 55737ede635ba35bcc60e61420bb3a5a0b5295ec

There are no files selected for viewing

1 change: 0 additions & 1 deletion 1 Lib/test/test_tabnanny.py

Show comments

View file

Open in desktop

Original file line number	Diff line number	Diff line change
Expand Up		@@ -316,7 +316,6 @@ def validate_cmd(self, *args, stdout="", stderr="", partial=False, expect_failur
		self.assertListEqual(out.splitlines(), stdout.splitlines())
		self.assertListEqual(err.splitlines(), stderr.splitlines())

		@unittest.expectedFailure # TODO: RUSTPYTHON; Should displays error when errored python file is given.
		def test_with_errored_file(self):
		"""Should displays error when errored python file is given."""
		with TemporaryPyFile(SOURCE_CODES["wrong_indented"]) as file_path:
Expand Down

13 changes: 0 additions & 13 deletions 13 Lib/test/test_tokenize.py

Show comments

View file

Open in desktop

Original file line number	Diff line number	Diff line change
Expand Up		@@ -1167,7 +1167,6 @@ async def bar(): pass
		DEDENT '' (7, 0) (7, 0)
		""")

		@unittest.expectedFailure # TODO: RUSTPYTHON; + " NEWLINE '\\n' (4, 1) (4, 2)"]
		def test_newline_after_parenthesized_block_with_comment(self):
		self.check_tokenize('''\
		[
Expand All		@@ -1192,7 +1191,6 @@ def test_closing_parenthesis_from_different_line(self):
		NAME 'x' (1, 3) (1, 4)
		""")

		@unittest.expectedFailure # TODO: RUSTPYTHON; ' FSTRING_END "\'\'\'" (2, 68) (2, 71)']
		def test_multiline_non_ascii_fstring(self):
		self.check_tokenize("""\
		a = f'''
Expand All		@@ -1204,7 +1202,6 @@ def test_multiline_non_ascii_fstring(self):
		FSTRING_END "\'\'\'" (2, 68) (2, 71)
		""")

		@unittest.expectedFailure # TODO: RUSTPYTHON; Diff is 696 characters long. Set self.maxDiff to None to see it.
		def test_multiline_non_ascii_fstring_with_expr(self):
		self.check_tokenize("""\
		f'''
Expand Down Expand Up		@@ -2176,7 +2173,6 @@ def test_string_concatenation(self):
		# Two string literals on the same line
		self.check_roundtrip("'' ''")

		@unittest.expectedFailure # TODO: RUSTPYTHON
		def test_random_files(self):
		# Test roundtrip on random python modules.
		# pass the '-ucpu' option to process the full directory.
Expand Down Expand Up		@@ -2214,7 +2210,6 @@ def test_indentation_semantics_retained(self):


		class InvalidPythonTests(TestCase):
		@unittest.expectedFailure # TODO: RUSTPYTHON; Diff is 1046 characters long. Set self.maxDiff to None to see it.
		def test_number_followed_by_name(self):
		# See issue #gh-105549
		source = "2sin(x)"
Expand Down Expand Up		@@ -2254,7 +2249,6 @@ def check_tokenize(self, s, expected):
		)
		self.assertEqual(result, expected.rstrip().splitlines())

		@unittest.expectedFailure # TODO: RUSTPYTHON
		def test_encoding(self):
		def readline(encoding):
		yield "1+1".encode(encoding)
Expand Down Expand Up		@@ -2386,7 +2380,6 @@ def test_float(self):
		NUMBER '3.14e159' (1, 4) (1, 12)
		""")

		@unittest.expectedFailure # TODO: RUSTPYTHON
		def test_string(self):

		self.check_tokenize('x = \'\'; y = ""', """\
Expand Down Expand Up		@@ -2818,7 +2811,6 @@ def test_unary(self):
		NUMBER '1' (1, 22) (1, 23)
		""")

		@unittest.expectedFailure # TODO: RUSTPYTHON
		def test_selector(self):

		self.check_tokenize("import sys, time\nx = sys.modules['time'].time()", """\
Expand All		@@ -2841,7 +2833,6 @@ def test_selector(self):
		RPAR ')' (2, 29) (2, 30)
		""")

		@unittest.expectedFailure # TODO: RUSTPYTHON
		def test_method(self):

		self.check_tokenize('@staticmethod\ndef foo(x,y): pass', """\
Expand All		@@ -2859,7 +2850,6 @@ def test_method(self):
		NAME 'pass' (2, 14) (2, 18)
		""")

		@unittest.expectedFailure # TODO: RUSTPYTHON
		def test_tabs(self):

		self.check_tokenize('@staticmethod\ndef foo(x,y): pass', """\
Expand Down Expand Up		@@ -3144,7 +3134,6 @@ async def bar(): pass
		DEDENT '' (6, -1) (6, -1)
		""")

		@unittest.expectedFailure # TODO: RUSTPYTHON
		def test_unicode(self):

		self.check_tokenize("Örter = u'places'\ngrün = U'green'", """\
Expand Down Expand Up		@@ -3394,7 +3383,6 @@ def f():
		with contextlib.redirect_stderr(StringIO()):
		_ = self.invoke_tokenize('--unknown')

		@unittest.expectedFailure # TODO: RUSTPYTHON
		def test_without_flag(self):
		# test 'python -m tokenize source.py'
		source = 'a = 1'
Expand All		@@ -3408,7 +3396,6 @@ def test_without_flag(self):
		'''
		self.check_output(source, expect)

		@unittest.expectedFailure # TODO: RUSTPYTHON
		def test_exact_flag(self):
		# test 'python -m tokenize -e/--exact source.py'
		source = 'a = 1'
Expand Down

Oops, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

Oops, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments. Suggestions cannot be applied while the pull request is queued to merge. Suggestion cannot be applied right now. Please check back later.

You are visitor 01402520 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.