GeezerVison is showing github.com — `ConvertTo-Csv`: Quote fields with quotes and newlines when using `-UseQuotes AsNeeded` by lselden · Pull Request #15765 · PowerShell/PowerShell
Conversation
lselden
commented
Jul 13, 2021
•
edited
Loading
Uh oh!
There was an error while loading. Please reload this page.
Fix for issue #9284 - Escape fields that contain quotes and newlines, not just those that contain the delimiter
PR Summary
This PR fixes the case of CSV fields that include double quotes or newlines in their values. If a Column value contains ",\r,\n or the delimiter then it will be marked as needing to be surrounded by quotes.
This follows RFC-4180 rules as to what characters trigger a field to be quoted (except it allows for different delimiters than the comma). This also follows how ConvertFrom-Csv parses input CSV files.
PR Context
-UseQuotes AsNeeded currently only looks for delimiters, which means that the field Hello,World will be surrounded in quotes, however the field Hello"World would not be surrounded in quotes, causing this kind of embarassing slip-up:
$testObj = [pscustomobject]@{ Greet = "Hello"; Me = "`""; Here = "World" };
# Greet Me Here
# ----- -- ----
# Hello " World
$str = $testObj | convertto-csv -UseQuotes AsNeeded
# Greet,Me,Here
# Hello,",World <-- Should be 'Hello,"""",World'
$str | convertfrom-csv
# Greet Me Here
# ----- -- ----
# Hello ,World
Newlines are another risk, since they're interpreted as completely different rows.
$testObj = [pscustomobject]@{ FirstColumn = "Hello"; SecondColumn = "World`r`nGoodbye" }
FirstColumn,SecondColumn
Hello,World
Goodbye
Performance Caveat: This version is searching for the existence of 4 characters (using string.IndexOfAny(chars) instead of the original 1 (string.Contains(char). Based on my benchmarks this change does have a miniscule increase in time for that single operation. However, in my naïve in-memory benchmarks against a 100MB CSV file, using -UseQuotes AsNeeded was slightly faster than -UseQuotes Always or -QuoteFields. My guess is that I didn't need to allocate as much memory for those unneeded quotes.
PR Checklist
- PR has a meaningful title
- Use the present tense and imperative mood when describing your changes
- Summarized changes
- Make sure all
.h, .cpp, .cs, .ps1 and .psm1 files have the correct copyright header
- This PR is ready to merge and is not Work in Progress.
- If the PR is work in progress, please add the prefix
WIP: or [ WIP ] to the beginning of the title (the WIP bot will keep its status check at Pending while the prefix is present) and remove the prefix when the PR is ready.
- Breaking changes
- User-facing changes
- Testing - New and feature
- Tooling
- I have considered the user experience from a tooling perspective and don't believe tooling will be impacted.
- OR
- I have considered the user experience from a tooling perspective and opened an issue in the relevant tool repository. This may include:
- Impact on PowerShell Editor Services which is used in the PowerShell extension for VSCode
(which runs in a different PS Host).
- Impact on Completions (both in the console and in editors) - one of PowerShell's most powerful features.
- Impact on PSScriptAnalyzer (which provides linting & formatting in the editor extensions).
- Impact on EditorSyntax (which provides syntax highlighting with in VSCode, GitHub, and many other editors).
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
Fix for issue PowerShell#9284 - Escape fields that contain quotes and newlines, not just those that contain the delimiter
|
All CLA requirements met.
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please add a test with new line chars.
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Added additional tests for different character scenarios that should be handled
|
Re: Testing - Added additional tests for detecting delimiters, newlines and quotes. I did put the test objects inside the It block, since they were only used per-test - let me know if they should be hoisted up to the Context's BeforeAll section.
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
Cast string as ReadOnlySpan to avoid allocation of char array
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
|
This pull request has been automatically marked as Review Needed because it has been there has not been any activity for 7 days.
Maintainer, please provide feedback and/or mark it as Waiting on Author
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
Travis Plunk (TravisEz13)
changed the title
ConvertTo-Csv: Quote fields with quotes and newlines when using -UseQuotes AsNeeded
ConvertTo-Csv: Quote fields with quotes and newlines when using -UseQuotes AsNeeded
Aug 4, 2021
|
🎉v7.2.0-preview.9 has been released which incorporates this pull request.:tada:
Handy links:
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
|
🎉v7.2.0-preview.10 has been released which incorporates this pull request.:tada:
Handy links:
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Labels
CL-General
Indicates that a PR should be marked as a general cmdlet change in the Change Log
Development
Successfully merging this pull request may close these issues.
Uh oh!
There was an error while loading. Please reload this page.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
|