GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-109638: Fix for significant backtracking in csv.Sniffer by sg3-141-592 · Pull Request #109639 · python/cpython

Conversation

@sg3-141-592

@sg3-141-592 sg3-141-592 commented Sep 21, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown

#109638 it is possible to get significant backtracking in csv.Sniffer() inside the doublequote checking regex. This change introduces a zero-length lookahead assertion to reduce the amount of backtracking.

This yields a significant improvement in testing

import csv
import time

for NUM_ITERATIONS in range(10,70,10):

    test_str = '"",'*NUM_ITERATIONS + '"'*NUM_ITERATIONS + '0' + '"'*NUM_ITERATIONS + '0'

    t0 = time.time()

    dialect = csv.Sniffer().sniff(test_str)

    t1 = time.time()
    print(f"{NUM_ITERATIONS}, {t1-t0}")
NUM_ITERATIONS | Before           | After Regex Fix
10  | 0.002374649 | 0.000935555
20  | 0.051596165 | 0.003250599
30  | 0.371201515 | 0.017635345
40  | 1.477169752 | 0.054927588
50  | 4.845417738 | 0.120140076
60  | 11.52993703 | 0.252950668

image

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@sg3-141-592
Performance improvements for csv doublequote regex
35154f3
@ghost

This comment was marked as resolved.

Sign in to view
@bedevere-app

This comment was marked as resolved.

Sign in to view
@bedevere-app bedevere-app Bot mentioned this pull request Sep 21, 2023
csv.Sniffer regular expression has significant backtracing #109638
Closed
@blurb-it @sg3-141-592
📜🤖 Added by blurb_it.
16d9612
@sg3-141-592
sg3-141-592 force-pushed the main branch from 86e8342 to 16d9612 Compare September 21, 2023 10:45
@sg3-141-592
sg3-141-592 marked this pull request as ready for review September 22, 2023 15:01
@bedevere-app bedevere-app Bot added the awaiting review label Sep 22, 2023
@AA-Turner AA-Turner added the skip news label Sep 22, 2023
@AA-Turner
Revert "📜🤖 Added by blurb_it."
2b29051 
This reverts commit 16d9612.
AA-Turner
AA-Turner reviewed Sep 22, 2023
View reviewed changes
Comment thread Lib/csv.py Outdated

# if we see an extra quote between delimiters, we've got a
# double quoted format
# in future Python versions this zero width look-ahead assert can be replaced with atomic groups

@AA-Turner AA-Turner Sep 22, 2023

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please may you explain this comment?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions

@sg3-141-592 sg3-141-592 Sep 23, 2023

Copy link
Copy Markdown
Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, this zero-width lookahead assertion change in the Regex can be done with an atomic group which is cleaner and more concise.

# Current change
(,|^)\W*"(?=(?P<zero>[^,|"\n]*))(?P=zero)"[^,|\n]*"\W*(,|$)
# Atomic Group
(,|^)\W*"(?>[^,|"\n]*)"[^,|\n]*"\W*(,|$)

But atomic groups are only supported in Python 3.11 onwards so I avoided using them here.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions

@AA-Turner AA-Turner Sep 23, 2023

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Sean. This PR (if merged) would be part of Python 3.13, so let's use the better atomic group method.

A

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions

@sg3-141-592 sg3-141-592 Sep 23, 2023

Copy link
Copy Markdown
Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, I've switched us to the simpler atomic group setup. Performance is identical to the previous fix.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@sg3-141-592
Converting zero-length lookahead assertions to atomic group
0265641
@sg3-141-592
sg3-141-592 force-pushed the main branch from c2a3522 to 0265641 Compare September 23, 2023 10:35
aterrel
aterrel approved these changes May 20, 2025
View reviewed changes

@aterrel aterrel left a comment

Copy link
Copy Markdown

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks like a good quick fix for the problem.

Ultimately though, these regexs are hard to read and cause a few problems with lists and other items. I think we should be thinking about how to replace the sniffer to have a higher accuracy. (See https://github.com/ws-garcia/CSVsniffer which shows that only 67.54% accuracy). I've posted to dpo on this topic here: https://discuss.python.org/t/rewrite-csv-sniffer

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@bedevere-app bedevere-app Bot added awaiting core review and removed awaiting review labels May 20, 2025
@github-actions

github-actions Bot commented Apr 14, 2026

Copy link
Copy Markdown

This PR is stale because it has been open for 30 days with no activity.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@github-actions github-actions Bot added the stale Stale PR or inactive for long period of time. label Apr 14, 2026
@serhiy-storchaka serhiy-storchaka mentioned this pull request Jul 14, 2026
gh-83273: Rewrite csv.Sniffer dialect detection using trial parsing #153694
Merged
@serhiy-storchaka
Merge branch 'main' into main
a9c4adf
@serhiy-storchaka

serhiy-storchaka commented Jul 21, 2026

Copy link
Copy Markdown
Member

Thank you for the patch, but I am afraid it does not work.

[^%(delim)s\n] includes the quote character, so the atomic group consumes everything up to the delimiter, the line break or the end of data, and the quote character which follows it in the pattern can never match there. The regular expression became unmatchable, so doublequote is now always false: it is fast because it no longer checks anything, and test_doublequote fails.

These regular expressions are gone on the main branch now (gh-83273), but a fix is still needed for the maintenance branches. Excluding the quote character from the character class, as the lookahead variant in the issue does, keeps the detection working and is enough to reduce the time for your example with 60 iterations from 11 s to 0.3 s (the growth is still superlinear, but much slower).
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@serhiy-storchaka serhiy-storchaka closed this Jul 21, 2026
@Punisheroot Punisheroot mentioned this pull request Jul 29, 2026
[3.15] gh-109638: Avoid pathological backtracking in csv.Sniffer (GH-153694) #154865
Closed
@serhiy-storchaka serhiy-storchaka mentioned this pull request Jul 29, 2026
[3.15] gh-109638: Fix exponential time in csv.Sniffer for doubled quotes #154868
Merged
@pikammmmm pikammmmm mentioned this pull request Jul 29, 2026
[3.15] gh-109638: Fix excessive backtracking in csv.Sniffer doublequote detection #154869
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@AA-Turner AA-Turner AA-Turner left review comments
+1 more reviewer
@aterrel aterrel aterrel approved these changes
Reviewers whose approvals may not affect merge requirements

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

awaiting core review skip news stale Stale PR or inactive for long period of time.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

4 participants

@sg3-141-592 @serhiy-storchaka @aterrel @AA-Turner
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.