GeezerVison is showing github.com — Documenting that the new (3.14) `pathlib` copy functionality uses Copy-on-Write · Issue #124985 · python/cpython
Documenting that the new (3.14) pathlib copy functionality uses Copy-on-Write #124985DescriptionDocumentation
(edited)
The PR 119058 and 122369 added pathlib.Path.copy(), with Copy-on-Write support. CoW should be documented, because it has distinctive, user-requested properties on huge files.
- Copying is instantaneous, does negligible I/O, requires no disk space for the data and negligible disk space for the metadata.
- Reading the original and the copy does half the I/O and requires half the RAM (page cache) than a traditional copy (think booting VM images)
In the context of a Linux VM manager, CoW is an explicit desired property. Disk image copying is the slowest part of snapshotting a VM. Users expect CoW snapshots nowadays, and intentionally set up a CoW filesystem for the disk image directory.
For clarity, I'm not asking to mention FICLONE specifically. I'm not asking to mention copy_file_range, a micro-optimization on the traditional copy algorithm. Instead, my point is that switching from O(file size) to zero is a user-visible feature.
Keywords: reflink copy
Linked PRs
Activitypicnixz commented on Oct 4, 2024 I'm not sure to follow what you want to do. What I understood is that you want us to document how we support CoW filesystems (by the way, the project you linked only support Python 3.7 according to pypi and I don't know why it would be a "keyword").
cc @barneygale as the PR's author changed the title [-]Reflink mention in the `pathlib` doc[/-] [+]Documenting that the new (3.14) `pathlib` copy functionality uses Copy-on-Write[/+] on Oct 5, 2024opk12 commented on Oct 5, 2024 Please document that copies are CoW if the filesystem is CoW and list the supported filesystems, so that one is certain to provide CoW to the end user. Please leave out the implementation details (the exact syscalls).
Sorry for the confusion. reflink copy / shallow copy is jargon for CoW, and the library was named after that. I now reworked the ticket title. barneygale commented on Oct 5, 2024 I'm cautious about touting the speed benefits of copy(), because the implementation of PathBase.copy() makes too many system calls when walking directories. We ought to add an implementation of Path.copy() that uses os.DirEntry.is_symlink() and is_dir(), rather than calling these methods on path objects. Or we could open a large can of worms, and consider whether PathBase.iterdir() might be allowed generate path objects with a special dir_entry attribute that grants public access to a os.DirEntry-like object, and then implement this in Path.iterdir(), and consult the attribute from PathBase.copy(). opk12 commented on Oct 6, 2024 Very good point. Without touching on copy()'s performance, how about only saying that it does CoW and on which filesystems? As said above, CoW is an explicit user request, has much better free space requirement and future read performance. opk12 commented on Oct 6, 2024 (Off-topic: there even was an actual proposal at btrfs, the most used CoW fs on Linux, to CoW-copy an arbitrary directory with a single syscall. The proposal stalled, but showed that this is doable.) barneygale commented on Oct 14, 2024 #125419 will solve the known copy() performance problems. When it lands I'll start working on this issue. added a commit that references this issue on Nov 5, 2024GH-124985: Document that `pathlib.Path.copy()` uses copy-on-write. (#… barneygale commented on Nov 5, 2024 This is now documented! Turns out I didn't need to wait for #125419. Thanks for the report, @opk12
MetadataMetadataStatusShow more project fields
DevelopmentNo branches or pull requestsIssue actions
|