Description

assignee = None
closed_at = None
created_at = <Date 2021-07-07.12:11:29.301>
labels = ['3.8', 'OS-windows', 'performance']
title = 'shutil.copy() inefficient implementation in Windows'
updated_at = <Date 2021-07-07.14:39:41.523>
user = 'https://bugs.python.org/sfmc'

activity = <Date 2021-07-07.14:39:41.523>
actor = 'eryksun'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Windows']
creation = <Date 2021-07-07.12:11:29.301>
creator = 'sfmc'
dependencies = []
files = []
hgrepos = []
issue_num = 44579
keywords = []
message_count = 3.0
messages = ['397076', '397077', '397091']
nosy_count = 6.0
nosy_names = ['paul.moore', 'tim.golden', 'zach.ware', 'eryksun', 'steve.dower', 'sfmc']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'performance'
url = 'https://bugs.python.org/issue44579'
versions = ['Python 3.8']

Activity

1. sfmc commented on Jul 7, 2021

   

   sfmcmannequin

   on Jul 7, 2021

   MannequinAuthor

   More actions

   In Windows shutil.copy() uses _copyfileobj_readinto which copies file in user mode.
   In Windows there is an fast API to copy file in kernel mode: CopyFile (see https://docs.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-copyfile).
2. sfmc

   added

   stdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directory

   3.8 (EOL)end of lifeend of life

   performancePerformance or resource usagePerformance or resource usage

   on Jul 7, 2021
3. tirkarthi

   added

   OS-windows

   on Jul 7, 2021
4. eryksun commented on Jul 7, 2021

   

   eryksun

   on Jul 7, 2021

   Contributor

   More actions

   In Windows there is an fast API to copy file in kernel mode: CopyFile

   The possibility of calling CopyFileEx() for shutil.copy2() is discussed in bpo-30044. Note that CopyFileEx() is a high-level Windows API function, not a "kernel mode" copy. It opens the source and destination files and makes multiple system calls in order to copy file data and metadata (e.g. system calls such as NtOpenFile, NtCreateFile, NtReadFile, NtWriteFile, NtQueryInformationFile, NtSetInformationFile, NtQueryEaFile, NtSetEaFile, NtQuerySecurityObject, etc). This includes copying the primary data stream, alternate data streams, file attributes, extended file attributes, and security resource attributes.
5. eryksun

   removed

   OS-windows

   on Jul 7, 2021
6. zooba commented on Jul 7, 2021

   

   zooba

   on Jul 7, 2021

   Member

   More actions

   Note that CopyFileEx() is a high-level Windows API function, not a "kernel mode" copy.

   This is true today, but could change whenever Windows feels like changing it. If we switch to the native API then we'll get any advantage there automatically.

   The only challenge is in managing changes to semantics (that is, anything extra we do to "match" Unix that isn't normally how copies work on Windows - personally, I'd rather be more native on Windows anyway).
7. eryksun

   added

   OS-windows

   and removed

   stdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directory

   on Jul 7, 2021
8. ezio-melotti

   transferred this issue fromon Apr 10, 2022
9. iritkatriel

   added

   3.12only security fixesonly security fixes

   and removed

   3.8 (EOL)end of lifeend of life

   on Sep 7, 2022
10. zooba

    self-assigned this

    on May 27, 2023
11. zooba commented on May 27, 2023

    

    zooba

    on May 27, 2023

    Member

    More actions

    I think it's time to do this one. I've assigned it to myself, but if someone else gets an implementation together first (it could use a CopyFileEx implementation in _winapi) I'm happy to review and merge.

    Performance enhancement can be backported to 3.12. The behaviour of copy2 should be totally unchanged by this, except in obscure edge cases (where they'll likely be more consistent with every other Windows app as a result of the change).

    We should also clearly document that copy2 may be significantly faster on Windows than the other copy* functions. Probably also worth looking for optimisations on other platforms as well, particularly if they preserve stat, so that we can say that copy2 is fastest on most platforms.
12. zooba

    added 2 commits that reference this issue on May 29, 2023

    pythongh-88745: Add _winapi.CopyFile2 and update shutil.copy2 to use it

    dffb615

    gh-88745: Add _winapi.CopyFile2 and update shutil.copy2 to use it (GH…

    cda1bd3
13. miss-islington

    added a commit that references this issue on May 30, 2023

    pythongh-88745: Add _winapi.CopyFile2 and update shutil.copy2 to use …

    6edc59a
14. zooba

    added a commit that references this issue on May 30, 2023

    gh-88745: Add _winapi.CopyFile2 and update shutil.copy2 to use it (GH…

    9ae49e3
15. zooba

    closed this as completedon May 30, 2023

Issue actions