Description

jackdesert

opened on Oct 28, 2022

Issue body actions

Using single-column quoted csv file of this format:

"one-one"
"two-two"
"three-three"
...

If the file is small (say 3,000 rows), it generates an error: _csv.Error: Could not determine delimiter
But if the file is large (say 30,000 rows) the process appears to hang.

Here is a code sample that reproduces this behavior:

import csv
from uuid import uuid4

items = []

items = [f'"{uuid4()}"' for _ in range(30_000)]
print('1. Created uuids')

csv_data = '\n'.join(items)
print('2. Joined')

with open('output', 'w') as h:
    h.write(csv_data)
print('3. Wrote output')

sniffer = csv.Sniffer()

# This is the line than hangs
dialect = sniffer.sniff(csv_data, delimiters=',:|\t')
print(f'4. Sniffed delimiter "{dialiect.delimiter}"')

Linked PRs

• [3.15] gh-98820: Fix quadratic time in csv.Sniffer for quoted fields #154867
• [3.14] gh-98820: Fix quadratic time in csv.Sniffer for quoted fields (GH-154867) #155117
• [3.13] gh-98820: Fix quadratic time in csv.Sniffer for quoted fields (GH-154867) #155118
• [3.12] gh-98820: Fix quadratic time in csv.Sniffer for quoted fields (GH-154867) #155166
• [3.11] gh-98820: Fix quadratic time in csv.Sniffer for quoted fields (GH-154867) #155538
• [3.10] gh-98820: Fix quadratic time in csv.Sniffer for quoted fields (GH-154867) #155539

Activity

1. mdboom

   added

   performancePerformance or resource usagePerformance or resource usage

   on Oct 28, 2022
2. Tickloop commented on Oct 31, 2022

   

   Tickloop

   on Oct 31, 2022

   More actions

   Can I try and tackle this one?
3. Tickloop commented on Nov 14, 2022

   

   Tickloop

   on Nov 14, 2022

   More actions

   Here are my thoughts after a dive into the implementation of csv module:

   1. Error produced:

      The error is expected as the module can't find a delimiter as there isn't one in csv_data. The code is telling the sniffer to find if the rows in csv_data use one of ',:|\t' as a delimiter, which isn't the case (csv_data does not have a delimiter as there is only one column).

      Now if the data were correctly formatted i.e. if one of the delimiters were present on even one of the lines, then the module is able to detect it as a delimiter.

   2. Hanging on sniffer.sniff():

      The sniffer's implementation to find the quotechar and delimiter in the data uses regex matching.

      If a delimiter isn't found then a probabilistic analysis happens which goes through chunks of data to determine a suitable delimiter.

      If the data isn't consistent and doesn't have a delimiter, then the regex function has n^2 run time which is causing the process to hang in case of 30_000 lines of input.
4. serhiy-storchaka

   added this to CSV issues 📊on Sep 10, 2023
5. iritkatriel

   added

   stdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directory

   on Nov 27, 2023
6. serhiy-storchaka

   self-assigned this

   on Jan 9, 2024
7. serhiy-storchaka

   closed this as completedin #153694on Jul 21, 2026
8. github-project-automation

   moved this to Done in CSV issues 📊on Jul 21, 2026
9. serhiy-storchaka

   reopened this on Jul 21, 2026
10. serhiy-storchaka

    added

    3.13only security fixesonly security fixes

    3.14bugs and security fixesbugs and security fixes

    3.15bugs and security fixesbugs and security fixes

    on Jul 21, 2026
11. serhiy-storchaka commented on Jul 21, 2026

    

    serhiy-storchaka

    on Jul 21, 2026

    Member

    More actions

    Fixed in 3.16 by the csv.Sniffer rewrite (gh-83273): a sample which is a single column of quoted fields is now recognized as such and csv.Error is raised immediately, instead of parsing quadratically slower with every added row.

    The maintenance branches are still affected: 3000 rows already take 2 seconds there, and the time grows quadratically. A targeted fix for them is welcome.
12. serhiy-storchaka

    removed their assignment

    on Jul 21, 2026
13. serhiy-storchaka

    added a commit that references this issue on Aug 3, 2026

    [3.15] gh-98820: Fix quadratic time in csv.Sniffer for quoted fields (G…

    476fb09
14. Yhg1s

    added a commit that references this issue on Aug 4, 2026

    [3.13] gh-98820: Fix quadratic time in csv.Sniffer for quoted fields (G…

    b30c7fa
15. serhiy-storchaka

    added a commit that references this issue on Aug 4, 2026

    [3.14] gh-98820: Fix quadratic time in csv.Sniffer for quoted fields (G…

    89f29c7
16. Yhg1s

    added a commit that references this issue on Aug 4, 2026

    [3.12] gh-98820: Fix quadratic time in csv.Sniffer for quoted fields (G…

    063d455
17. deadlovelll

    added a commit that references this issue on Aug 5, 2026

    [3.14] pythongh-98820: Fix quadratic time in csv.Sniffer for quoted f…

    318cc3d
18. pablogsal

    added 2 commits that reference this issue on Aug 11, 2026

    [3.11] gh-98820: Fix quadratic time in csv.Sniffer for quoted fields (G…

    b09a67a

    [3.10] gh-98820: Fix quadratic time in csv.Sniffer for quoted fields (G…

    fd78b56
19. StanFromIreland

    closed this as completedon Aug 21, 2026
20. mcepl

    added 4 commits that reference this issue on Sep 18, 2026

    [CVE-2026-18503] Fix quadratic time in csv.Sniffer for quoted fields

    9b385b2

    [CVE-2026-18503] Fix quadratic time in csv.Sniffer for quoted fields

    460f114

    [CVE-2026-18503] Fix quadratic time in csv.Sniffer for quoted fields

    761b17e

    [CVE-2026-18503] Fix quadratic time in csv.Sniffer for quoted fields

    4a37ce3
21. mcepl

    added 3 commits that reference this issue on Oct 1, 2026

    [CVE-2026-18503] Fix quadratic time in csv.Sniffer for quoted fields

    847b361

    [CVE-2026-18503] Fix quadratic time in csv.Sniffer for quoted fields

    53beee9

    [CVE-2026-18503] Fix quadratic time in csv.Sniffer for quoted fields

    d6fd9d2
22. icanhasmath

    added a commit that references this issue on Oct 28, 2026

    [3.10] pythongh-98820: Fix quadratic time in csv.Sniffer for quoted f…

    c1c23f0

Issue actions