GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — Handle large history file properly by reading lines in the streaming way by daxian-dbw · Pull Request #3810 · PowerShell/PSReadLine

Conversation

@daxian-dbw

@daxian-dbw Dongbo Wang (daxian-dbw) commented Sep 21, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

PR Summary

Fix #3771, Fix #1360, Fix #537

Handle large history file properly by reading lines in the streaming way.

PR Checklist

  • PR has a meaningful title
    • Use the present tense and imperative mood when describing your changes
  • Summarized changes
  • Make sure you've added one or more new tests
  • Make sure you've tested these changes in terminals that PowerShell is commonly used in (i.e. conhost.exe, Windows Terminal, Visual Studio Code Integrated Terminal, etc.)
  • User-facing changes
    • Not Applicable
    • OR
    • Documentation needed at PowerShell-Docs
      • Doc Issue filed:
Microsoft Reviewers: Open in CodeFlow

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@daxian-dbw
Handle large history file properly by reading lines in the streaming way
238ff97
@daxian-dbw
Dongbo Wang (daxian-dbw) requested a review from Steve Lee (SteveL-MSFT) September 21, 2023 22:08
iSazonov
Ilya (iSazonov) reviewed Sep 22, 2023
View reviewed changes
Comment thread PSReadLine/History.cs Resolved

Uh oh!

There was an error while loading. Please reload this page.

Dongbo Wang (daxian-dbw) added 2 commits September 26, 2023 12:44
@daxian-dbw
Read lines instead of bytes
59d5587
@daxian-dbw
Fix for .net462
d293db1
daxian-dbw
Dongbo Wang (daxian-dbw) commented Sep 26, 2023
View reviewed changes
Comment thread PSReadLine/History.cs
// After seeking, the current position may point at the middle of a history record, or even at a
// byte within a UTF-8 character (history file is saved with UTF-8 encoding). So, let's ignore the
// first line read from that position.
sr.ReadLine();

@daxian-dbw Dongbo Wang (daxian-dbw) Sep 26, 2023

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ilya (@iSazonov) Regarding you comment "I could save the file in Utf16", PSReadLine always saves the history file in UTF-8 encoding. Both File.CreateText and File.AppendText writes UTF-8 encoding text to the file, so if a user saves the file in Unicode, then it may be broken to read later after PSReadLine writes something to the file. So, I will assume it's UTF-8 encoding.

Can you please review again? Thanks!

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@iSazonov Ilya (iSazonov) Sep 27, 2023

Copy link
Copy Markdown

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you see no problem that the history file may be corrupted in very rare cases this code looks good.


For reflection only, I have the following questions and doubts.

  1. The user can actually save the file in a different encoding.
  2. Only .Net Core at some point started using Utf8 by default. Windows PowerShell can write in Utf16 by default.
  3. Although StreamReader detects the encoding automatically, the current code seems to ignore this as it skips the beginning of the file.
  4. Why do we store the beginning of a large file when now we don't use it anyway and the user doesn't even know about it? Probably we can find a way to trim it, although it's not easy to think of.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@daxian-dbw Dongbo Wang (daxian-dbw) Sep 27, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The user can actually save the file in a different encoding.

If the user manually changes encoding, to like UTF-16 LE, the history file will be corrupted once PSReadLine writes updates to the file, because PSReadLine always writes text in UTF-8. I verified this using both Windows PowerShell and PowerShell 7+.

See the code below (Both File.CreateText and File.AppendText writes UTF-8 encoding text to the file, on both .NET and .NET Framework):

PSReadLine/PSReadLine/History.cs

Lines 366 to 379 in d045b50

using (var file = overwritten ? File.CreateText(Options.HistorySavePath) : File.AppendText(Options.HistorySavePath))
{
for (var i = start; i <= end; i++)
{
HistoryItem item = _history[i];
item._saved = true;
// Actually, skip writing sensitive items to file.
if (item._sensitive) { continue; }
var line = item.CommandLine.Replace("\n", "`\n");
file.WriteLine(line);
}
}

Only .Net Core at some point started using Utf8 by default. Windows PowerShell can write in Utf16 by default.

Again, File.CreateText and File.AppendText writes UTF-8 encoding text to the file, on both .NET and .NET Framework

Although StreamReader detects the encoding automatically, the current code seems to ignore this as it skips the beginning of the file.

I tried it out, and it turns out that StreamReader detects encoding when creating the instance.

Why do we store the beginning of a large file when now we don't use it anyway and the user doesn't even know about it? Probably we can find a way to trim it, although it's not easy to think of.

I think you missed something in the code. If the user decides to set max history count to more than 20,000, we will read all content from the file. So, the history is still accessible to the user as long as they want.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
iSazonov
Ilya (iSazonov) reviewed Sep 27, 2023
View reviewed changes
Comment thread PSReadLine/History.cs Resolved

Uh oh!

There was an error while loading. Please reload this page.

@daxian-dbw

Dongbo Wang (daxian-dbw) commented Sep 27, 2023

Copy link
Copy Markdown
Member Author

Ilya (@iSazonov) Can you please approve the PR if all your concerns are resolved? I need an approval to merge the PR.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

iSazonov
Ilya (iSazonov) approved these changes Sep 28, 2023
View reviewed changes
@iSazonov

Ilya (iSazonov) commented Sep 28, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown

Ilya (@iSazonov) Can you please approve the PR if all your concerns are resolved? I need an approval to merge the PR.

Approved with one note but I haven't full permissions :-)

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

iSazonov
Ilya (iSazonov) reviewed Sep 28, 2023
View reviewed changes
Comment thread PSReadLine/History.cs Resolved

Uh oh!

There was an error while loading. Please reload this page.

@daxian-dbw
Dongbo Wang (daxian-dbw) merged commit ac69010 into PowerShell:master Oct 3, 2023
@daxian-dbw
Dongbo Wang (daxian-dbw) deleted the history branch October 3, 2023 01:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@SteveL-MSFT Steve Lee (SteveL-MSFT) Awaiting requested review from SteveL-MSFT
1 more reviewer
@iSazonov Ilya (iSazonov) iSazonov approved these changes
Reviewers whose approvals may not affect merge requirements

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

PSReadLine cannot handle super large history file properly In VS Code getting this System.OutOfMemoryException: Array dimensions exceeded supported range. Limit size of ConsoleHost_history.txt file

Uh oh!

There was an error while loading. Please reload this page.

2 participants

@daxian-dbw @iSazonov
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402520 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.