GeezerVison

The World Wide Web, restored to 800 pixels of glory.

GeezerVison is showing github.com — bpo-30825: guess lineterminator in csv.Sniffer by vmax · Pull Request #2529 · python/cpython

bpo-30825: guess lineterminator in csv.Sniffer #2529

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed

vmax wants to merge 2 commits into python:master from vmax:fix-issue-30825

Uh oh!

There was an error while loading. Please reload this page.

Closed

bpo-30825: guess lineterminator in csv.Sniffer #2529

Show file tree

Hide file tree

Changes from all commits

Commits

Show all changes

2 commits

Select commit Hold shift + click to select a range

6f46019

bpo-30825: guess lineterminator in csv.Sniffer

vmax Jul 1, 2017

524ff18

Fix unit test to cover weird or no terminator case (os.linesep)

vmax Feb 12, 2018

File filter

Filter by extension .py (2) .rst (1) All 2 file types selected

Viewed files

Clear filters

Conversations

Failed to load comments.

Uh oh!

There was an error while loading. Please reload this page.

Jump to

Jump to file

Failed to load files.

Uh oh!

There was an error while loading. Please reload this page.

Diff view

Unified

Split

Hide whitespace

Diff view

Unified

Split

Hide whitespace

There are no files selected for viewing

24 changes: 19 additions & 5 deletions 24 Lib/csv.py

Show comments

View file

Open in desktop

Original file line number	Diff line number	Diff line change
Expand Up		@@ -13,6 +13,7 @@

		from collections import OrderedDict
		from io import StringIO
		import os

		__all__ = ["QUOTE_MINIMAL", "QUOTE_ALL", "QUOTE_NONNUMERIC", "QUOTE_NONE",
		"Error", "Dialect", "__doc__", "excel", "excel_tab",
Expand Down Expand Up		@@ -165,7 +166,7 @@ def writerows(self, rowdicts):

		class Sniffer:
		'''
		"Sniffs" the format of a CSV file (i.e. delimiter, quotechar)
		"Sniffs" the format of a CSV file (i.e. delimiter, quotechar, lineterminator)
		Returns a Dialect object.
		'''
		def __init__(self):
Expand All		@@ -178,18 +179,19 @@ def sniff(self, sample, delimiters=None):
		Returns a dialect (or None) corresponding to the sample
		"""

		lineterminator = self._guess_lineterminator(sample)
		quotechar, doublequote, delimiter, skipinitialspace = \
		self._guess_quote_and_delimiter(sample, delimiters)
		if not delimiter:
		delimiter, skipinitialspace = self._guess_delimiter(sample,
		delimiters)
		delimiters,
		lineterminator)

		if not delimiter:
		raise Error("Could not determine delimiter")

		class dialect(Dialect):
		_name = "sniffed"
		lineterminator = '\r\n'
		quoting = QUOTE_MINIMAL
		# escapechar = ''

Expand All		@@ -198,6 +200,7 @@ class dialect(Dialect):
		# _csv.reader won't accept a quotechar of ''
		dialect.quotechar = quotechar or '"'
		dialect.skipinitialspace = skipinitialspace
		dialect.lineterminator = lineterminator

		return dialect

Expand Down Expand Up		@@ -278,7 +281,7 @@ def _guess_quote_and_delimiter(self, data, delimiters):
		return (quotechar, doublequote, delim, skipinitialspace)


		def _guess_delimiter(self, data, delimiters):
		def _guess_delimiter(self, data, delimiters, lineterminator):
		"""
		The delimiter /should/ occur the same number of times on
		each row. However, due to malformed data, it may not. We don't want
Expand All		@@ -297,7 +300,7 @@ def _guess_delimiter(self, data, delimiters):
		additional chunks as necessary.
		"""

		data = list(filter(None, data.split('\n')))
		data = list(filter(None, data.split(lineterminator)))

		ascii = [chr(c) for c in range(127)] # 7-bit ASCII

Expand Down Expand Up		@@ -447,3 +450,14 @@ def has_header(self, sample):
		hasHeader -= 1

		return hasHeader > 0


		def _guess_lineterminator(self, sample):
		# Guess line terminator based on presence
		terminators = ('\r\n', '\n', '\r')

		for terminator in terminators:
		if terminator in sample:
		return terminator
Comment thread terryjreedy Feb 7, 2018 Copy link Copy Markdown Member There was a problem hiding this comment. Choose a reason for hiding this comment The reason will be displayed to describe this comment to others. Learn more. This is the core of the patch. The sequence looks right. Sorry, something went wrong. Uh oh! There was an error while loading. Please reload this page. All reactions

		return os.linesep
Comment thread terryjreedy Feb 7, 2018 Copy link Copy Markdown Member There was a problem hiding this comment. Choose a reason for hiding this comment The reason will be displayed to describe this comment to others. Learn more. In case user submits a sample with an unusual or no line terminator. Sorry, something went wrong. Uh oh! There was an error while loading. Please reload this page. All reactions

15 changes: 15 additions & 0 deletions 15 Lib/test/test_csv.py

Show comments

View file

Open in desktop

Original file line number	Diff line number	Diff line change
Expand Up		@@ -9,6 +9,7 @@
		import csv
		import gc
		import pickle
		import os
		from test import support
		from itertools import permutations
		from textwrap import dedent
Expand Down Expand Up		@@ -1018,6 +1019,20 @@ def test_doublequote(self):
		dialect = sniffer.sniff(self.sample9)
		self.assertTrue(dialect.doublequote)

		def test_guess_lineterminator(self):
		sniffer = csv.Sniffer()
		dialect = sniffer.sniff(r"Date;Value\r\n2010-01-01;10")
		self.assertEqual(dialect.lineterminator, '\r\n')
		dialect = sniffer.sniff(r"Date;Value\n2010-01-01;10")
		self.assertEqual(dialect.lineterminator, '\n')
Comment thread terryjreedy Feb 7, 2018 Copy link Copy Markdown Member There was a problem hiding this comment. Choose a reason for hiding this comment The reason will be displayed to describe this comment to others. Learn more. This assumes that the line terminator for multiline triple-quoted string literals is '\n'. We require that '\r\n' in code files be converted to '\n' before merging, but the test should not assume '\n' for all developer and user systems where the test might be run. So `sample8` should be replaced by a literal. Since the sample strings need not be long, they can all be literals defined here. Make all 3 the same except for the terminator. This will make the test both crystal clear and independent of system and optional local settings. Sorry, something went wrong. Uh oh! There was an error while loading. Please reload this page. All reactions
		dialect = sniffer.sniff(r"Date;Value\r2010-01-01;10")
		self.assertEqual(dialect.lineterminator, '\r')
		dialect = sniffer.sniff(r"Date;Value\v2010-01-01;10")
		self.assertEqual(dialect.lineterminator, os.linesep)
		dialect = sniffer.sniff(r"Date;Value")
		self.assertEqual(dialect.lineterminator, os.linesep)

Comment thread terryjreedy Feb 7, 2018 Copy link Copy Markdown Member There was a problem hiding this comment. Choose a reason for hiding this comment The reason will be displayed to describe this comment to others. Learn more. Add tests to cover the last line of _guess_lineterminator: `return os.linesep`. Add a test with the same literal, but with an oddball terminator, such as '\v' (vertical tab), and another with a single line with no control chars. Sorry, something went wrong. Uh oh! There was an error while loading. Please reload this page. All reactions

		class NUL:
		def write(s, *args):
		pass
Expand Down

1 change: 1 addition & 0 deletions 1 Misc/NEWS.d/next/Library/2018-02-12-05-10-28.bpo-30825.t0RXql.rst

Show comments

View file

Open in desktop

Original file line number	Diff line number	Diff line change
		@@ -0,0 +1 @@
		csv.Sniffer now detects line terminator instead of defaulting to '\r\n'

Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments. Suggestions cannot be applied while the pull request is queued to merge. Suggestion cannot be applied right now. Please check back later.

You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.

GeezerVison

bpo-30825: guess lineterminator in csv.Sniffer #2529

Uh oh!

bpo-30825: guess lineterminator in csv.Sniffer #2529

Uh oh!

Uh oh!

Diff view

Diff view

There are no files selected for viewing

terryjreedy Feb 7, 2018

Choose a reason for hiding this comment

Uh oh!

terryjreedy Feb 7, 2018

Choose a reason for hiding this comment

Uh oh!

terryjreedy Feb 7, 2018

Choose a reason for hiding this comment

Uh oh!

terryjreedy Feb 7, 2018

Choose a reason for hiding this comment

Uh oh!