GeezerVison is showing github.com — Get-Item ./.hidden-file fails with "Could not find item" · Issue #15323 · PowerShell/PowerShell
Get-Item ./.hidden-file fails with "Could not find item" #15323DescriptionThis behavior is on macOS 11.3.
Steps to reproduce
Expected behavior
Directory: /Users/nimazzuc
Mode LastWriteTime Length Name
---- ------------- ------ ----
----- 4/28/2021 12:25 PM 107 .profile
Actual behavior
Get-Item: Could not find item /Users/nimazzuc/.profile.
Environment data
WSManStackVersion = 3.0
PSEdition = Core
Platform = Unix
PSCompatibleVersions = 1.0 2.0 3.0 4.0 5.0 5.1.10032.0 6.0.0 6.1.0 6.2.0 7.0.0 7.1.3
GitCommitId = 7.1.3
OS = Darwin 20.4.0 Darwin Kernel Version 20.4.0: Fri Mar 5 01:14:02 PST 2021; root:xnu-7195.101.1~3/RELEASE_ARM64_T8101
PSRemotingProtocolVersion = 2.3
SerializationVersion = 1.1.0.1
PSVersion = 7.1.3
Additionally, the Mode thing is weird, shouldn't that show the unix file mode? Reacted by Michael Klement Activity237dmitry commented on Apr 29, 2021 Reacted by Michael Klement and James Truher rjmholt commented on Apr 29, 2021 I'm able to reproduce this:
> gi ~/.vimrc
Get-Item: Could not find item /Users/rjmholt/.vimrc.
> gi -force ~/.vimrc
Directory: /Users/rjmholt
Mode LastWriteTime Length Name
---- ------------- ------ ----
---h- 4/22/2021 3:46 PM 1030 .vimrc
> stat ~/.vimrc
16777233 1451798 -rw-r--r-- 1 rjmholt staff 0 1030 "Apr 22 16:28:44 2021" "Apr 22 15:46:55 2021" "Apr 22 15:46:55 2021" "Apr 22 15:46:55 2021" 4096 8 0 /Users/rjmholt/.vimrc
The difference in the modes shown seems to imply that PowerShell lacks permissions in some sense perhaps. rjmholt commented on Apr 29, 2021 rjmholt commented on Apr 29, 2021 Get-Error output:
> get-error
Exception :
Type : System.IO.IOException
Message : Could not find item /Users/rjmholt/.vimrc.
HResult : -2146232800
TargetObject : /Users/rjmholt/.vimrc
CategoryInfo : ObjectNotFound: (/Users/rjmholt/.vimrc:String) [Get-Item], IOException
FullyQualifiedErrorId : ItemNotFound,Microsoft.PowerShell.Commands.GetItemCommand
InvocationInfo :
MyCommand : Get-Item
ScriptLineNumber : 1
OffsetInLine : 1
HistoryId : 169
Line : gi ~/.vimrc
PositionMessage : At line:1 char:1
+ gi ~/.vimrc
+ ~~~~~~~~~~~
InvocationName : gi
CommandOrigin : Internal
ScriptStackTrace : at <ScriptBlock>, <No file>: line 1
PipelineIterationInfo :
237dmitry commented on Apr 29, 2021 This is standart syntax for hidden "items".
> help Get-item -Parameter Force
-Force <System.Management.Automation.SwitchParameter>
Indicates that this cmdlet gets items that can't otherwise be accessed, such as hidden items. Implementation varies from provider to provider.....
vexx32 commented on Apr 29, 2021 This has always been the norm as far as I'm aware, yeah. -Force has always been required when querying for hidden and sometimes system files/folders. Reacted by Patrick Meinecke strega-nil commented on Apr 29, 2021 This is... very weird to me, but fair enough. I still think it's a bug (or at least a thing that should be changed), since hidden files aren't "different" from regular files on unices. Reacted by Michael Klement, vanillaSprinkles, Rain Sallow (/u/ta11ow) and Oscar Aguinagalde Reacted by Philip Meholm iSazonov commented on Apr 30, 2021 It is not bug, it is default PowerShell behavior for years. PowerShell never gets/enumerates objects with hidden and system attributes.
since hidden files aren't "different" from regular files on unices.
It is Unix standard - first dot in name assumes that the file is hidden. .Net maps sign to FileAttribute.Hidden and PowerShell follows the Unix and .Net behavior.
In Get-ChildItem we can use -Hidden and -System switches to get such files in output. mklement0 commented on Apr 30, 2021 Even if it isn't a bug, the behavior is counterintuitive and inconvenient, given that a specific, known-to-exist file is being asked for: If I know that it's there, its hidden-or-not status should be irrelevant.
Contrast that with the arguably more sensible behavior of POSIX-compatible shells such as bash, and the (POSIX-mandated) ls utility:
-
ls behavior (shell-independent):
- Enumeration behavior (no path / only parent path argument, e.g.,
ls ~):
- Parallels
Get-ChildItem's enumeration behavior: hidden items are not included by default, and there's an opt-in to include them (-A, the equivalent of -Force)
- Literal direct-target path-argument behavior (e.g,
ls ~/.bashrc):
- The item is listed, irrespective of whether it is hidden or not.
-
Globbing behavior, which is purely a shell feature (that is, the shell itself resolves the wildcard expression up front and passes the resulting literal paths to the target command):
- Unless the pattern starts with
., hidden items are not included by default; e.g.:
sh -c 'printf %s\\n ~/*bashrc' does not find ~/.bashrc (unfortunately, bash, dash, and ksh then pass the unexpanded pattern through, which is unfortunate, but that's a separate issue)- but
sh -c 'printf %s\\n ~/.*bashrc' does, due to the name-pattern-initial .
- Specific shells may have shell-environment options to include hidden items by default, such as
bash's dotglob option.
- Note that the only reason I'm using
sh -c in the examples above is the need to bypass PowerShell's own native globbing (see below).
The following following changes make sense to me:
-
Make Get-Item / Get-ChildItem find hidden items directly specified by their literal path, without the need for -Force.
-
Make .-initial wildcards such as .*rc match hidden items without the need for -Force.
rjmholt commented on Apr 30, 2021 The hidden dot-files behaviour was originally a bug in ls, so I don't think it gets any credit there. What's more, the dot-file concept is specific to ls as a utility — things like stat don't care what a file is called.
On the other hand, Get-ChildItem and Get-Item (which I think is arguably more like stat than ls) share a common implementation because they are provider traversal commands under which the filesystem provider is just one implementation, so the provider must have a single consistent behaviour underneath both of them. This isn't an implementation detail; PowerShell providers are a user-level concept and each *-Item* cmdlet corresponds to an established provider API.
Incidentally this is actually much closer in spirit to Plan9's walk and reminds me of the fact that Plan9 has no recursive -r flag on utilities because their philosophy was that there should only be one code path for that.
With that said though, PowerShell providers do differentiate between getting an item and getting child items. Get-Item uses the former API:
|
/// <summary> |
|
/// Gets the item at the specified path. |
|
/// </summary> |
|
/// <param name="path"> |
|
/// A fully qualified path representing a file or directory in the |
|
/// file system. |
|
/// </param> |
|
/// <returns> |
|
/// Nothing. FileInfo and DirectoryInfo objects are written to the |
|
/// context's pipeline. |
|
/// </returns> |
|
/// <exception cref="System.ArgumentException"> |
|
/// path is null or empty. |
|
/// </exception> |
|
protected override void GetItem(string path) |
While Get-ChildItem uses the latter:
|
/// <summary> |
|
/// Gets the child items of a given directory. |
|
/// </summary> |
|
/// <param name="path"> |
|
/// The full path of the directory to enumerate. |
|
/// </param> |
|
/// <param name="recurse"> |
|
/// If true, recursively enumerates the child items as well. |
|
/// </param> |
|
/// <param name="depth"> |
|
/// Limits the depth of recursion; uint.MaxValue performs full recursion. |
|
/// </param> |
|
/// <returns> |
|
/// Nothing. FileInfo and DirectoryInfo objects that match the filter are written to the |
|
/// context's pipeline. |
|
/// </returns> |
|
/// <exception cref="System.ArgumentException"> |
|
/// path is null or empty. |
|
/// </exception> |
|
protected override void GetChildItems( |
|
string path, |
|
bool recurse, |
|
uint depth) |
|
{ |
|
GetPathItems(path, recurse, depth, false, ReturnContainers.ReturnMatchingContainers); |
|
} |
So conceivably we could alter the behaviour in only Get-Item without introducing deep inconsistency, here:
|
if (!isContainer) |
|
{ |
|
if (!hidden || Force || showHidden || filterHidden || switchFilterHidden) |
|
{ |
|
s_tracer.WriteLine("Got file info: {0}", result); |
|
return result; |
|
} |
|
} |
It's worth noting that PowerShell itself doesn't decide that files whose names start with . are hidden. We rely on an attribute that .NET gives us:
|
bool hidden = attributes.HasFlag(FileAttributes.Hidden); |
.NET has decided to inherit ls's baggage here and as a result, we have no way of knowing why a file was hidden. So this behaviour would need to apply to all hidden files (e.g. if a file has the BSD hidden flag set and starts with a ., we have no .NET-native way to know about the former). More importantly, this would imply changes on Windows as well, since again, we rely on .NET's abstraction here.
The main problem I see, clearly from the responses here, is that this is an established behaviour — users have learned that this happens with hidden files and that -Force is required much like with ls -a.
Anyway, I don't have a particular horse in this race (I was mainly interested to learn more on the topic). But generally speaking, I'd say you need a pretty compelling argument to change an established behaviour in a core functionality like this. mklement0 commented on Apr 30, 2021 Rob Holt (@rjmholt), after having just posted #10238 (comment), I figured: what the heck, one for the road, for old time's sake:
the dot-file concept is specific to ls as a utility
As shown above, all major POSIX-like shells have adopted this convention.
That this convention may have been a historical accident is an interesting tidbit, but no more (a shell with an ironclad commitment to backward-compatibility should appreciate that).
users have learned that this happens with hidden files and that -Force is required much like with ls -a.
That's great. And it's appropriate for enumeration behavior and for wildcards that don't start with .
But it's absurd otherwise - when you know you're looking for a hidden file, either by literal path or by deliberately formulated wildcard - as evidenced by the following:
# Tab-complete a hidden file:
PS> Get-Item ~/.ba<tab> # tab-completes to '/Users/jdoe/.bashrc'
Get-Item: Could not find item /Users/jdoe/.bashrc. Now, you could argue that the tab-completion behavior should be fixed.
But I hope you won't. strega-nil commented on May 1, 2021 Yeah, the fact that Get-Item requires -Force to "find" a hidden file just means that one should always pass -Force to Get-Item, since:
function Get-JsonElement {
[CmdletBinding()]
Param([Parameter(Mandatory)][String]$Path, [Parameter(Mandatory)][String]$JPointer)
$File = Get-Item $Path
...
}this is entirely reasonable code to write; additionally:
> Get-JsonElement .config.json '.foo.bar' is also entirely reasonable code to write. There is no reason that putting these two bits of code together should fail, assuming .config.json exists, yet the default is that it does fail with a File Not Found error (which is... a very odd error, I must say).
EDIT: I would also argue that the behavior makes no sense on Windows either; why does Get-Item ~/AppData fail. That is very, very weird behavior. Reacted by Michael Klement, Rain Sallow (/u/ta11ow) and vanillaSprinkles jantari commented on May 3, 2021 mklement0 commented on May 4, 2021 Thanks, jantari, but just to spell it out: you workaround makes Get-ChildItem and Get-Item return hidden items unconditionally.
A proper fix would mean:
- Always allow items targeted by literal path, irrespective of their hidden status; never require
-Force in that case.
- As an aside: requiring
-Force for removal is justified, and Remove-Item already behaves correctly in this case: if -Force is absent, instead of pretending that the item doesn't exist (the way Get-Item always and Get-ChildItem situationally does), it acknowledges the existence and states that the hidden nature of the item requires -Force.
- Omit hidden items by default when enumerating items in a directory, which happens in two cases:
- Explicitly, with
Get-ChildItem
- Implicitly, via globbing (resolving wildcard expressions to matching items)
- To include hidden items in these enumeration cases,
-Force is needed.
- Additionally, on Unix only, a wildcard expression (path component) that starts with
. should implicitly act as if -Force had also been passed (on Windows, an item's name doesn't signal its hidden status, so this doesn't apply)
- A related challenge - again on Unix only - is that the globbing applied when calling native utilities provides doesn't allow for a
-Force switch (Native globbing on Unix lacks ability to include hidden items #4683).
Conversely, to summarize what's broken with respect to literal paths to hidden items specified without -Force:
-
Get-Item is broken with respect to both files and directories - it pretends they don't exist.
-
Get-ChildItem is broken with respect to files only - it happily targets hidden directories (see code below) and non-hidden files inside them (by direct path).
- It is in bad company there:
cmd.exe's dir exhibits the same partially broken behavior; additionally, del is broken in the sense that it pretends that hidden items don't exist (cf. the sensible Remove-Item behavior described above).
Furthermore, Rob Holt (@rjmholt):
.NET has decided to inherit ls's baggage here
To recap: .-initial items being considered hidden from enumeration is a decades-old convention observed on all Unix-like platforms.
and as a result, we have no way of knowing why a file was hidden.
We don't need to know, we can rely on .NET's enumeration behavior when exclusion of hidden items is requested (by default, nothing is excluded).
So this behaviour would need to apply to all hidden files (e.g. if a file has the BSD hidden flag set
At least as of .NET 5, the .NET file-system enumeration APIs don't know about this alternative, name-independent BSD/macOS mechanism of hiding files, and, similarly, neither does ls on macOS (where only the platform's file manager, Finder, observes it). [Update: This may change in .NET 6 - see https://github.com/dotnet/runtime/pull/52235]
Again, deferring to .NET should be fine.
To further illustrate the inconsistency in Get-Item's (and partially Get-ChildItem's) behavior with respect to literal paths directly targeting a hidden item: The following demonstrates that other file-processing cmdlets do find hidden items by literal path, by (often invariable) default, and also juxtaposes the behavior with that of the platform-native shell / utilities:
# Switch to a temporary directory.
Push-Location -ea Stop ($tmpDir = (New-Item -Type Directory -Force (Join-Path Temp:/ $PID)).FullName)
try {
# Create a hidden directory...
$null = New-Item -Type Directory -ea Stop .hiddenTmpDir; if ($IsWindows) { attrib +h .hiddenTmpDir }
# ... and a hidden file.
$null = New-Item -Type File -ea Stop .hiddenTmpFile; if ($IsWindows) { attrib +h .hiddenTmpFile }
# Add content to the hidden file.
# Add-Content targets the hidden file just fine, if it is targeted with a literal path.
'hi1' | Add-Content -ea Stop .hiddenTmpFile
# Create a non-hidden file in the hidden directory.
# Note using the hidden directory as part of the file path works just fine.
'hi2' > .hiddenTmpDir/test.txt
# Ditto, with Set-Content
'hi3' | Set-Content -ea Stop .hiddenTmpDir/test.txt
# Get-ChildItem accepts literal paths to hidden directories and to files inside hidden directories,
# but NOT if you directly target a file that is itself hidden, .hiddenTmpFile.
(Get-ChildItem .hiddenTmpDir).Name
(Get-ChildItem .hiddenTmpDir/test.txt).Name
# Test the existence of the hidden file, the hidden dir. and the non-hidden
# file inside the hidden dir.
Test-Path .hiddenTmpFile
Test-Path .hiddenTmpDir
Test-Path .hiddenTmpDir/test.txt
# Get the content of the hidden file and of one of the non-hidden files in
# the hidden directory.
Get-Content .hiddenTmpFile
Get-Content .hiddenTmpDir/test.txt
# Set-Location / Push-Location accepts literal paths to hidden directories.
Push-Location
Set-Location .hiddenTmpDir
Pop-Location
# Removing a non-hidden file in a hidden dir. works as-is
Remove-Item .hiddenTmpDir/test.txt
# By contrast, removing an item that is itself hidden - justifiably -
# requires -Force, but note that Remove-Item does NOT pretend that the
# the item *isn't there* if you neglect to use -Force.
Remove-Item -Force .hiddenTmpFile
Remove-Item -Force -Recurse .hiddenTmpDir
'---- Native shell / external utilities:'
# Create a hidden directory...
$null = New-Item -Type Directory -ea Stop .hiddenTmpDir; if ($IsWindows) { attrib +h .hiddenTmpDir }
# ... and a hidden file.
$null = New-Item -Type File -ea Stop .hiddenTmpFile; if ($IsWindows) { attrib +h .hiddenTmpFile }
# ... and a non-hidden file in the hidden dir.
'hi3' > .hiddenTmpDir/test.txt
if ($isWindows) {
cmd /v /c 'cd .hiddenTmpDir & echo !CD!'
cmd /c 'dir /b .hiddenTmpDir'
cmd /c 'type .hiddenTmpFile'
cmd /c 'type .hiddenTmpDir\test.txt'
cmd /c 'if exist .hiddenTmpFile echo exists'
cmd /c 'if exist .hiddenTmpDir echo exists'
cmd /c 'if exist .hiddenTmpDir\test.txt echo exists'
cmd /c 'del .hiddenTmpDir\test.txt'
cmd /c 'rd /q /s .hiddenTmpDir'
}
else {
sh -c 'cd .hiddenTmpDir; echo $PWD'
ls .hiddenTmpFile
stat .hiddenTmpFile
ls -d .hiddenTmpDir
sh -c 'test -e .hiddenTmpFile && echo exists'
sh -c 'test -e .hiddenTmpDir && echo exists'
sh -c 'test -e .hiddenTmpDir/test.txt && echo exists'
cat .hiddenTmpFile
cat .hiddenTmpDir/test.txt
rm .hiddenTmpFile
rm .hiddenTmpDir/test.txt
rm -r .hiddenTmpDir
}
}
finally {
# Clean up.
Pop-Location; Remove-Item -Force -Recurse $tmpDir
}
I think I've said all I have to say, so I'm unsubscribing from this thread. daxian-dbw commented on May 28, 2021 The Engine Working Group discussed this issue today, and we agree it's an enhancement that we should consider. But for this particular issue, it's more of a question, and it was answered, so we will resolve this issue as "Answered". A separate enhancement issue #15477 is opened and linked to this issue.
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|