GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — [3.15] gh-109638: Fix excessive backtracking in csv.Sniffer doublequote detection by pikammmmm · Pull Request #154869 · python/cpython

Conversation

@pikammmmm

@pikammmmm pikammmmm commented Jul 29, 2026 •
edited by bedevere-app Bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown

csv.Sniffer detects a doubled-quote dialect with a pattern built from two
greedy character classes, both of which admit the quote character:

%(quote)s[^%(delim)s\n]*%(quote)s[^%(delim)s\n]*%(quote)s

Because either class can absorb a quote, the engine has to try every way of
splitting a run of quotes between them. On quote-heavy input that blows up:
sniffing the sample from the issue with 60 iterations takes about 9.7 s on 3.15.

Excluding the quote from the first class forces the quote that follows it to
match at the first one available, which removes the ambiguity:

%(quote)s[^%(delim)s%(quote)s\n]*%(quote)s[^%(delim)s\n]*%(quote)s

The second class has to keep admitting the quote — that is what lets "a""b"
match, where it consumes "b so the third quote can land on the closing one.
Narrowing both classes, or making the first one atomic while it still admits the
quote (as GH-109639 did), makes the pattern unmatchable and silently disables
detection; test_doublequote catches both.

This is the change @serhiy-storchaka described in #109638. It is not needed on
main, where these regexes no longer exist after the Sniffer rewrite in
gh-83273, so this PR is based directly on 3.15. 3.14 and 3.13 are affected
too and will get the same fix once this is reviewed.

Measurements

csv.Sniffer().sniff() on the sample from the issue body, run on 3.15.0b1:

iterations before after speedup
40 1.11 s 0.042 s 27x
60 9.70 s 0.257 s 38x
80 43.85 s 0.83 s 53x

Growth is still superlinear, as noted in the issue — this narrows the search
rather than replacing the approach — but it is far slower.

Test

test_sniff_regex_backtracking mirrors the test added on main, with one
difference: it uses 60 iterations rather than 100. main can afford 100 because
the rewrite made the cost negligible, whereas with this targeted fix 100
iterations still takes about 1.7 s, which is too slow to put in the suite. 60 is
the size quoted in the issue and costs 0.23 s.

Whole-file test_csv.py on 3.15.0b1: 0.45 s with the fix, 11.6 s
without it, 135 passing either way — so the new test acts as the same kind of
canary it does on main rather than asserting on a timer.

Checking the change is behaviour-preserving

Beyond the suite, I compared sniff() results between patched and unpatched
csv.py across 8726 cases — the delimiter/quote/doublequote/skipinitialspace
combinations, TestSniffer's own corpus, and 4000 randomized quote-dense
strings — with no divergence in the returned dialect or the exceptions raised.


Disclosure: prepared with AI assistance (Claude Code) — used to analyse the
backtracking behaviour, produce the measurements and differential check above,
and draft the patch and this description.

  • Issue: csv.Sniffer regular expression has significant backtracing #109638

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@pikammmmm
pythongh-109638: Fix excessive backtracking in csv.Sniffer doublequot…
dde954e
…e detection

The doublequote-detection pattern used two greedy character classes that both
admitted the quote character, so the engine had to try every way of splitting a
run of quotes between them.

Excluding the quote from the first class forces the quote that follows it to
match at the first one available, which removes the ambiguity while leaving
detection intact.  The second class must keep admitting the quote, since that
is what lets `"a""b"` match -- narrowing both classes, or making the first one
atomic while it still admits the quote, silently disables the detection.
@python-cla-bot

python-cla-bot Bot commented Jul 29, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown

All commit authors signed the Contributor License Agreement.

CLA signed

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-app bedevere-app Bot added the awaiting review label Jul 29, 2026
@bedevere-app bedevere-app Bot mentioned this pull request Jul 29, 2026
csv.Sniffer regular expression has significant backtracing #109638
Closed
@pikammmmm

pikammmmm commented Jul 29, 2026

Copy link
Copy Markdown
Author

Closing this in favour of #154868, which solves the same problem more thoroughly: the possessive-quantifier rewrite also detects a doubled quote inside a field that contains the delimiter or a line break, which this patch does not.

I hadn't spotted #154865 or #154868 when I opened this — sorry for the duplicate noise.

serhiy-storchaka reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@pikammmmm pikammmmm closed this Jul 29, 2026
@serhiy-storchaka serhiy-storchaka mentioned this pull request Jul 29, 2026
[3.15] gh-109638: Avoid pathological backtracking in csv.Sniffer (GH-153694) #154865
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

awaiting review

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

1 participant

@pikammmmm
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.