GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — `ConvertTo-Csv`: Quote fields with quotes and newlines when using `-UseQuotes AsNeeded` by lselden · Pull Request #15765 · PowerShell/PowerShell

Conversation

@lselden

@lselden lselden commented Jul 13, 2021 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

Fix for issue #9284 - Escape fields that contain quotes and newlines, not just those that contain the delimiter

PR Summary

This PR fixes the case of CSV fields that include double quotes or newlines in their values. If a Column value contains ",\r,\n or the delimiter then it will be marked as needing to be surrounded by quotes.

This follows RFC-4180 rules as to what characters trigger a field to be quoted (except it allows for different delimiters than the comma). This also follows how ConvertFrom-Csv parses input CSV files.

PR Context

-UseQuotes AsNeeded currently only looks for delimiters, which means that the field Hello,World will be surrounded in quotes, however the field Hello"World would not be surrounded in quotes, causing this kind of embarassing slip-up:

$testObj = [pscustomobject]@{ Greet = "Hello"; Me = "`""; Here = "World" };
# Greet Me Here
# ----- -- ----
# Hello "  World

$str = $testObj | convertto-csv -UseQuotes AsNeeded
# Greet,Me,Here
# Hello,",World  <-- Should be 'Hello,"""",World'

$str | convertfrom-csv
# Greet Me     Here
# ----- --     ----
# Hello ,World

Newlines are another risk, since they're interpreted as completely different rows.

$testObj = [pscustomobject]@{ FirstColumn = "Hello"; SecondColumn = "World`r`nGoodbye" }
FirstColumn,SecondColumn
Hello,World
Goodbye

Performance Caveat: This version is searching for the existence of 4 characters (using string.IndexOfAny(chars) instead of the original 1 (string.Contains(char). Based on my benchmarks this change does have a miniscule increase in time for that single operation. However, in my naïve in-memory benchmarks against a 100MB CSV file, using -UseQuotes AsNeeded was slightly faster than -UseQuotes Always or -QuoteFields. My guess is that I didn't need to allocate as much memory for those unneeded quotes.

PR Checklist

  • PR has a meaningful title
    • Use the present tense and imperative mood when describing your changes
  • Summarized changes
  • Make sure all .h, .cpp, .cs, .ps1 and .psm1 files have the correct copyright header
  • This PR is ready to merge and is not Work in Progress.
    • If the PR is work in progress, please add the prefix WIP: or [ WIP ] to the beginning of the title (the WIP bot will keep its status check at Pending while the prefix is present) and remove the prefix when the PR is ready.
  • Breaking changes
    • None
    • OR
    • Experimental feature(s) needed
      • Experimental feature name(s):
  • User-facing changes
    • Not Applicable
    • OR
    • Documentation needed
      • Issue filed: 7796
  • Testing - New and feature
    • N/A or can only be tested interactively
    • OR
    • Make sure you've added a new test if existing tests do not effectively test the code changed
  • Tooling
    • I have considered the user experience from a tooling perspective and don't believe tooling will be impacted.
    • OR
    • I have considered the user experience from a tooling perspective and opened an issue in the relevant tool repository. This may include:
      • Impact on PowerShell Editor Services which is used in the PowerShell extension for VSCode
        (which runs in a different PS Host).
        • Issue filed:
      • Impact on Completions (both in the console and in editors) - one of PowerShell's most powerful features.
        • Issue filed:
      • Impact on PSScriptAnalyzer (which provides linting & formatting in the editor extensions).
        • Issue filed:
      • Impact on EditorSyntax (which provides syntax highlighting with in VSCode, GitHub, and many other editors).
        • Issue filed:

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@lukeselden
Escape quotes and newlines with -UseQuotes AsNeeded
34e3ca6
Fix for issue PowerShell#9284 - Escape fields that contain quotes and newlines, not just those that contain the delimiter
@ghost ghost assigned Travis Plunk (TravisEz13) Jul 13, 2021
@ghost

Deleted user (ghost) commented Jul 13, 2021 •
edited by ghost
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown

CLA assistant check
All CLA requirements met.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

iSazonov
Ilya (iSazonov) reviewed Jul 13, 2021
View reviewed changes

@iSazonov Ilya (iSazonov) left a comment

Copy link
Copy Markdown
Collaborator

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add a test with new line chars.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread src/Microsoft.PowerShell.Commands.Utility/commands/utility/CsvCommands.cs Resolved

Uh oh!

There was an error while loading. Please reload this page.

@lukeselden
add additional -UseQuotes AsNeeded tests
c712df9
Added additional tests for different character scenarios that should be handled
@lselden

lselden commented Jul 13, 2021

Copy link
Copy Markdown
Contributor Author

Re: Testing - Added additional tests for detecting delimiters, newlines and quotes. I did put the test objects inside the It block, since they were only used per-test - let me know if they should be hoisted up to the Context's BeforeAll section.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

Luke Selden (Vbrick) (lukeselden) added 2 commits July 13, 2021 12:24
@lukeselden
Code Quality cleanup per codefactor review
15200e4
@lukeselden
inline AsNeeded character detection
4832f43
Cast string as ReadOnlySpan to avoid allocation of char array
@iSazonov Ilya (iSazonov) added the CL-General Indicates that a PR should be marked as a general cmdlet change in the Change Log label Jul 13, 2021
iSazonov
Ilya (iSazonov) reviewed Jul 14, 2021
View reviewed changes
Comment thread test/powershell/Modules/Microsoft.PowerShell.Utility/ConvertTo-Csv.Tests.ps1 Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread test/powershell/Modules/Microsoft.PowerShell.Utility/ConvertTo-Csv.Tests.ps1 Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread test/powershell/Modules/Microsoft.PowerShell.Utility/ConvertTo-Csv.Tests.ps1 Resolved

Uh oh!

There was an error while loading. Please reload this page.

@lselden @iSazonov
Apply suggestions from code review
a9f296f
Co-authored-by: Ilya <[email protected]>
iSazonov
Ilya (iSazonov) approved these changes Jul 15, 2021
View reviewed changes
@sdwheeler Sean Wheeler (sdwheeler) mentioned this pull request Jul 22, 2021
ConvertTo-Csv / Export-Csv: Update -UseQuotes AsNeeded MicrosoftDocs/PowerShell-Docs#7796
Closed
@ghost ghost added the Review - Needed The PR is being reviewed label Jul 22, 2021
@ghost

Deleted user (ghost) commented Jul 22, 2021

Copy link
Copy Markdown

This pull request has been automatically marked as Review Needed because it has been there has not been any activity for 7 days.
Maintainer, please provide feedback and/or mark it as Waiting on Author

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

TravisEz13
Travis Plunk (TravisEz13) approved these changes Aug 4, 2021
View reviewed changes
@TravisEz13 Travis Plunk (TravisEz13) changed the title ConvertTo-Csv: Quote fields with quotes and newlines when using -UseQuotes AsNeeded ConvertTo-Csv: Quote fields with quotes and newlines when using -UseQuotes AsNeeded Aug 4, 2021
@TravisEz13
Travis Plunk (TravisEz13) merged commit 3e86f9b into PowerShell:master Aug 4, 2021
@ghost ghost removed the Review - Needed The PR is being reviewed label Aug 4, 2021
@TravisEz13 Travis Plunk (TravisEz13) added this to the 7.2.0-preview.9 milestone Aug 4, 2021
@lselden
lselden deleted the csv-usequotes-asneeded-rfc4180-compat branch August 11, 2021 13:19
@ghost

Deleted user (ghost) commented Aug 23, 2021

Copy link
Copy Markdown

🎉v7.2.0-preview.9 has been released which incorporates this pull request.:tada:

Handy links:

  • Release Notes
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@ghost

Deleted user (ghost) commented Sep 28, 2021

Copy link
Copy Markdown

🎉v7.2.0-preview.10 has been released which incorporates this pull request.:tada:

Handy links:

  • Release Notes
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

Thatgfsj (Thatgfsj) pushed a commit to Thatgfsj-contribs/PowerShell that referenced this pull request Aug 6, 2026
@lselden @iSazonov @lukeselden
ConvertTo-Csv: Quote fields with quotes and newlines when using `-U…
4dbaf0a
…seQuotes AsNeeded` (PowerShell#15765)

Co-authored-by: Ilya <[email protected]>
Co-authored-by: lselden <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@TravisEz13 Travis Plunk (TravisEz13) TravisEz13 approved these changes
@iSazonov Ilya (iSazonov) iSazonov approved these changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

@TravisEz13 Travis Plunk (TravisEz13)

Labels

CL-General Indicates that a PR should be marked as a general cmdlet change in the Change Log

Projects

None yet

Milestone

7.2.0-preview.9

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

4 participants

@lselden @TravisEz13 @iSazonov @lukeselden
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.