GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-105017: Include CRLF lines in strings and column numbers by mgmacias95 · Pull Request #105030 · python/cpython

Conversation

@mgmacias95

@mgmacias95 mgmacias95 commented May 27, 2023 •
edited by bedevere-bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor
  • Issue: Tokenizer produces different output on Windows on py312 for ends of files #105017

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@mgmacias95
fix(tokenizer): Include CRLF lines in strings and column numbers
ad130da
@mgmacias95
mgmacias95 requested review from lysnikolaou and pablogsal as code owners May 27, 2023 21:46
@bedevere-bot bedevere-bot added the awaiting review label May 27, 2023
@bedevere-bot bedevere-bot mentioned this pull request May 27, 2023
Tokenizer produces different output on Windows on py312 for ends of files #105017
Closed
mgmacias95 and others added 2 commits May 27, 2023 22:48
@mgmacias95
Merge branch 'main' into crlf_strings
6688c36
@blurb-it
📜🤖 Added by blurb_it.
e5293bc
@pablogsal pablogsal added the needs backport to 3.12 only security fixes label May 27, 2023
@pablogsal
Only preserve crlf in the Python-tokenize extension
37f77ad
pablogsal
pablogsal reviewed May 27, 2023
View reviewed changes
Comment thread Python/Python-tokenize.c Resolved

Uh oh!

There was an error while loading. Please reload this page.
@pablogsal pablogsal changed the title gh-105017: [Tokenizer] Include CRLF lines in strings and column numbers gh-105017: Include CRLF lines in strings and column numbers May 27, 2023
@pablogsal

pablogsal commented May 27, 2023

Copy link
Copy Markdown
Member

There were a bunch of failing tests because we are removing CRLF handling in the regular path that expects the source to be normalized. I pushed a commit that only preserves CRLF in our tokenize extension
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
pablogsal added 4 commits May 28, 2023 12:32
@pablogsal
Handle \r\n in continuation lines
68e7b36
@pablogsal
fixup! Handle \r\n in continuation lines
969060f
@pablogsal
fixup! fixup! Handle \r\n in continuation lines
559487c
@pablogsal
fixup! fixup! fixup! Handle \r\n in continuation lines
363de28
pablogsal
pablogsal reviewed May 28, 2023
View reviewed changes
Comment thread Lib/test/test_tokenize.py Resolved

Uh oh!

There was an error while loading. Please reload this page.
@pablogsal pablogsal added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label May 28, 2023
@bedevere-bot

bedevere-bot commented May 28, 2023

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @pablogsal for commit 363de28 🤖

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label May 28, 2023
mgmacias95 and others added 2 commits May 28, 2023 14:55
@mgmacias95 @pablogsal
Update Lib/test/test_tokenize.py
67b3a9c 
Co-authored-by: Pablo Galindo Salgado <[email protected]>
@mgmacias95
Prevent leak
d3d4ff5
@pablogsal pablogsal added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label May 28, 2023
@bedevere-bot

bedevere-bot commented May 28, 2023

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @pablogsal for commit d3d4ff5 🤖

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label May 28, 2023
@pablogsal
pablogsal merged commit 96fff35 into python:main May 28, 2023
@bedevere-bot bedevere-bot removed the awaiting review label May 28, 2023
@miss-islington

miss-islington commented May 28, 2023

Copy link
Copy Markdown
Contributor

Thanks @mgmacias95 for the PR, and @pablogsal for merging it 🌮🎉.. I'm working now to backport this PR to: 3.12.
🐍🍒⛏🤖
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
miss-islington pushed a commit to miss-islington/cpython that referenced this pull request May 28, 2023
@mgmacias95 @pablogsal @miss-islington
pythongh-105017: Include CRLF lines in strings and column numbers (py…
c220d12 
…thonGH-105030)

(cherry picked from commit 96fff35)

Co-authored-by: Marta Gómez Macías <[email protected]>
Co-authored-by: Pablo Galindo <[email protected]>
@bedevere-bot

bedevere-bot commented May 28, 2023

Copy link
Copy Markdown

GH-105041 is a backport of this pull request to the 3.12 branch.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-bot bedevere-bot removed the needs backport to 3.12 only security fixes label May 28, 2023
pablogsal added a commit that referenced this pull request May 28, 2023
@miss-islington @mgmacias95 @pablogsal
[3.12] gh-105017: Include CRLF lines in strings and column numbers (G…
3f8d5d9 
…H-105030) (#105041)

gh-105017: Include CRLF lines in strings and column numbers (GH-105030)
(cherry picked from commit 96fff35)

Co-authored-by: Marta Gómez Macías <[email protected]>
Co-authored-by: Pablo Galindo <[email protected]>
pablogsal
pablogsal reviewed May 28, 2023
View reviewed changes

@pablogsal pablogsal left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Great job! ✨

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@bedevere-bot

bedevere-bot commented May 28, 2023

Copy link
Copy Markdown

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Debian root 3.12 has failed when building commit 3f8d5d9.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/1102/builds/33) and take a look at the build logs.
  4. Check if the failure is related to this commit (3f8d5d9) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/1102/builds/33

Failed tests:

  • test_threading
  • test__xxsubinterpreters

Summary of the results of the build (if available):

==

Click to see traceback logs 
remote: Enumerating objects: 16, done.        
remote: Counting objects:   6% (1/16)        
remote: Counting objects:  12% (2/16)        
remote: Counting objects:  18% (3/16)        
remote: Counting objects:  25% (4/16)        
remote: Counting objects:  31% (5/16)        
remote: Counting objects:  37% (6/16)        
remote: Counting objects:  43% (7/16)        
remote: Counting objects:  50% (8/16)        
remote: Counting objects:  56% (9/16)        
remote: Counting objects:  62% (10/16)        
remote: Counting objects:  68% (11/16)        
remote: Counting objects:  75% (12/16)        
remote: Counting objects:  81% (13/16)        
remote: Counting objects:  87% (14/16)        
remote: Counting objects:  93% (15/16)        
remote: Counting objects: 100% (16/16)        
remote: Counting objects: 100% (16/16), done.        
remote: Compressing objects:   6% (1/15)        
remote: Compressing objects:  13% (2/15)        
remote: Compressing objects:  20% (3/15)        
remote: Compressing objects:  26% (4/15)        
remote: Compressing objects:  33% (5/15)        
remote: Compressing objects:  40% (6/15)        
remote: Compressing objects:  46% (7/15)        
remote: Compressing objects:  53% (8/15)        
remote: Compressing objects:  60% (9/15)        
remote: Compressing objects:  66% (10/15)        
remote: Compressing objects:  73% (11/15)        
remote: Compressing objects:  80% (12/15)        
remote: Compressing objects:  86% (13/15)        
remote: Compressing objects:  93% (14/15)        
remote: Compressing objects: 100% (15/15)        
remote: Compressing objects: 100% (15/15), done.        
remote: Total 16 (delta 1), reused 11 (delta 1), pack-reused 0        
From https://github.com/python/cpython
 * branch                  3.12       -> FETCH_HEAD
Note: switching to '3f8d5d9ed60ae0f70c92dc29587cc6367eda80c3'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 3f8d5d9ed6 [3.12] gh-105017: Include CRLF lines in strings and column numbers (GH-105030) (#105041)
Switched to and reset branch '3.12'

configure: WARNING: pkg-config is missing. Some dependencies may not be detected correctly.

Fatal Python error: Segmentation fault

Thread 0x00007f253095a280 (most recent call first):
  File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/unittest/runner.py", line 74 in _write_status
  File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/unittest/runner.py", line 95 in addSuccess
  File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/test/support/testresult.py", line 122 in addSuccess
  File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/unittest/case.py", line 648 in run
  File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/unittest/case.py", line 690 in __call__
  File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/unittest/suite.py", line 122 in run
  File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/unittest/suite.py", line 84 in __call__
  File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/unittest/suite.py", line 122 in run
  File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/unittest/suite.py", line 84 in __call__
  File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/unittest/suite.py", line 122 in run
  File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/unittest/suite.py", line 84 in __call__
  File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/unittest/runner.py", line 240 in run
  File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/test/support/__init__.py", line 1115 in _run_suite
  File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/test/support/__init__.py", line 1241 in run_unittest
  File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/test/libregrtest/runtest.py", line 294 in _test_module
  File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/test/libregrtest/runtest.py", line 330 in _runtest_inner2
  File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/test/libregrtest/runtest.py", line 373 in _runtest_inner
  File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/test/libregrtest/runtest.py", line 248 in _runtest
  File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/test/libregrtest/runtest.py", line 278 in runtest
  File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/test/libregrtest/main.py", line 365 in rerun_failed_tests
  File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/test/libregrtest/main.py", line 783 in _main
  File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/test/libregrtest/main.py", line 738 in main
  File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/test/libregrtest/main.py", line 802 in main
  File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/test/__main__.py", line 2 in <module>
  File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/runpy.py", line 88 in _run_code
  File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/runpy.py", line 198 in _run_module_as_main

Extension modules: _testcapi (total: 1)
Segmentation fault
make: *** [Makefile:2009: buildbottest] Error 139

Cannot open file '/root/buildarea/3.12.angelico-debian-amd64/build/test-results.xml' for upload
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@picnixz picnixz mentioned this pull request May 28, 2023
Compatibility: pycode parser: adjust dedent-token handling to maintain py3.12+ compatibility sphinx-doc/sphinx#11440
Closed
@erlend-aasland erlend-aasland mentioned this pull request Jun 6, 2023
3.12 backport gh 105236 #105358
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pablogsal pablogsal pablogsal left review comments
@lysnikolaou lysnikolaou Awaiting requested review from lysnikolaou lysnikolaou is a code owner

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

4 participants

@mgmacias95 @pablogsal @bedevere-bot @miss-islington
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.