Description

g-yziquel

opened on Sep 29, 2023

Issue body actions

Bug report

Bug description:

CSV dialect sniffer gets wrong dialect (excel) when tab delimited csv file has long header fields.

This issue is lifted up here as it was initially raised in the jc cli github repo. Here:

kellyjonbrazil/jc#465

CPython versions tested on:

3.10

Operating systems tested on:

Linux

Linked PRs

• [3.15] gh-110128: Ignore a truncated final row in csv.Sniffer #155037

Activity

1. g-yziquel

   added

   type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

   on Sep 29, 2023
2. sunmy2019 commented on Sep 29, 2023

   

   sunmy2019

   on Sep 29, 2023

   Member

   More actions

   You'd better provide a repro code that only utilizes Python stdlib.

   ---

   You raised a problem with csv.Sniffer.sniff. You think one of its sniff results is inaccurate.

   But, it's just a guess. And guess cannot be perfect.

   What new algo will you raise and how will it affect the current use case?
3. sunmy2019

   added

   stdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directory

   on Sep 29, 2023
4. g-yziquel commented on Sep 29, 2023

   

   g-yziquel

   on Sep 29, 2023

   Author

   More actions

   Sure. Any more accurate sniffer is arguably not to be provided out of the box by the stdlib.

   It remains that one of its sniffs is inaccurate, and I could reproduce it with the stdlib. (If that would be genuinely welcome).

   I would however be very wary of making changes to the sniffing algo without a series of tests on a corpus of such CSV files.

   As of today, I have no clue as to how this should be done. But I do believe the CSV files I gave should, in an ideal world, be part of such a testing corpus.

   Feel free to close the issue. I raised it out of personal ethos. If it's just noise to you, please do close it.

   Reacted by W. García
5. serhiy-storchaka

   added this to CSV issues 📊on Sep 30, 2023
6. serhiy-storchaka

   self-assigned this

   on Jan 9, 2024
7. ws-garcia commented on Jan 5, 2025

   

   ws-garcia

   on Jan 5, 2025

   More actions

   Hi @g-yziquel, the research on the subject is on its way! Please take a look to the Table Uniformity Method (TUM), also implemented in Python using the built-in csv module.
8. serhiy-storchaka commented on Jul 21, 2026

   

   serhiy-storchaka

   on Jul 21, 2026

   Member

   More actions

   The problem is not the length of the header fields, but the truncation of the sample: jc sniffs a fixed-size prefix, and the old code counted the incomplete last row as a real one, so the tab was no longer consistent and sniff() raised csv.Error. DummyData.Right.csv is 1024 bytes and fits in the sample entirely, DummyData.Wrong.csv is 1046 bytes and does not; the "right" file fails too if the sample is cut at 512 bytes.

   Fixed in 3.16 by the csv.Sniffer rewrite (gh-83273): the last row is not counted if the sample can be cut off in the middle of it. The maintenance branches are still affected, and a targeted fix for them is welcome.
9. serhiy-storchaka

   added

   3.13only security fixesonly security fixes

   3.14bugs and security fixesbugs and security fixes

   3.15bugs and security fixesbugs and security fixes

   on Jul 21, 2026
10. serhiy-storchaka

    removed their assignment

    on Jul 21, 2026
11. serhiy-storchaka

    changed the title [-]CSV dialect sniffer gets wrong dialect (excel) when tab delimited csv file has long header fields[/-] [+]CSV dialect sniffer gets wrong dialect (excel) when tab delimited csv file has been truncated[/+] on Jul 21, 2026
12. Punisheroot

    added a commit that references this issue on Aug 1, 2026

    [3.15] pythongh-110128: Ignore a truncated final row in csv.Sniffer

    6975587
13. encukou commented on Oct 13, 2026

    

    encukou

    on Oct 13, 2026

    Member

    More actions

    @serhiy-storchaka: Myself, I'd leave long-standing behaviour alone in bugfix releases, rather than change it with fixes that aren't in main.
    @hugovk, what's your opinion?
14. kevinprimeraquecreo-stack

    added this to Project_VKon Oct 18, 2026
15. github-project-automation

    moved this to Todo in Project_VKon Oct 18, 2026
16. kevinprimeraquecreo-stack

    added this to Project_VKon Oct 18, 2026
17. hugovk commented on Oct 29, 2026

    

    hugovk

    on Oct 29, 2026

    Member

    More actions

    Agreed, let's keep the existing behaviour for 3.15 and 3.14.

    The larger #153694 rewrite fixes it for 3.16 (and we decided not to backport that #153694 (comment)).

    Thanks anyway!
18. hugovk

    closed this as not plannedon Oct 29, 2026
19. github-project-automation

    moved this to Done in CSV issues 📊on Oct 29, 2026

Issue actions