Description

giampaolo

opened on Jun 5, 2019

Issue body actions

BPO	37159
Nosy	@facundobatista, @ncoghlan, @vstinner, @giampaolo, @encukou, @albertz, @vadmium, @desbma, @pablogsal
Files	patch.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2019-06-05.05:24:51.971>
labels = ['library', '3.9', 'performance']
title = 'Use copy_file_range() in shutil.copyfile() (server-side copy)'
updated_at = <Date 2020-12-29.13:05:56.586>
user = 'https://github.com/giampaolo'

bugs.python.org fields:

activity = <Date 2020-12-29.13:05:56.586>
actor = 'Albert.Zeyer'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2019-06-05.05:24:51.971>
creator = 'giampaolo.rodola'
dependencies = []
files = ['48392']
hgrepos = []
issue_num = 37159
keywords = ['patch']
message_count = 6.0
messages = ['344671', '344679', '344680', '344691', '344693', '383996']
nosy_count = 11.0
nosy_names = ['facundobatista', 'ncoghlan', 'vstinner', 'giampaolo.rodola', 'StyXman', 'petr.viktorin', 'neologix', 'Albert.Zeyer', 'martin.panter', 'desbma', 'pablogsal']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'performance'
url = 'https://bugs.python.org/issue37159'
versions = ['Python 3.9']

Linked PRs

• gh-81340: Use copy_file_range in shutil.copyfile copy functions #93152

Reacted by Daniel Tang and Sam James

Activity

1. giampaolo commented on Jun 5, 2019

   

   giampaolo

   on Jun 5, 2019

   ContributorAuthor

   More actions

   This is a follow up of bpo-33639 (zero-copy via sendfile()) and bpo-26828 (os.copy_file_range()). On [Linux 4.5 / glib 2.27] shutil.copyfile() will use os.copy_file_range() instead of os.sendfile(). According to my benchmarks performances are the same but when dealing with NFS copy_file_range() is supposed to attempt doing a server-side copy, meaning there will be no exchange of data between client and server, making the copy operation an order of magnitude faster.

   Before proceeding unit-tests for big-file support should be added first (bpo-37096). We didn't hit the 3.8 deadline but I actually prefer to land this in 3.9 as I want to experiment with it a bit (copy_file_range() is quite new, bpo-26828 is still a WIP).
2. giampaolo

   added

   3.9 (EOL)end of lifeend of life

   stdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directory

   performancePerformance or resource usagePerformance or resource usage

   on Jun 5, 2019
3. giampaolo

   changed the title [-]Have shutil.copyfile() use copy_file_range()[/-] [+]Use copy_file_range() in shutil.copyfile() (server-side copy)[/+] on Jun 5, 2019
4. vstinner commented on Jun 5, 2019

   

   vstinner

   on Jun 5, 2019

   Member

   More actions

   Oh, I already created https://bugs.python.org/issue37157

   Can we move the discussion there?
5. giampaolo commented on Jun 5, 2019

   

   giampaolo

   on Jun 5, 2019

   ContributorAuthor

   More actions

   bpo-37157 is for reflink / CoW copy, this one is not.
6. vstinner commented on Jun 5, 2019

   

   vstinner

   on Jun 5, 2019

   Member

   More actions

   bpo-37157 is for reflink / CoW copy, this one is not.

   Oh sorry, it seems like I misunderstood copy_file_range(). So it doesn't use/support CoW?
7. giampaolo commented on Jun 5, 2019

   

   giampaolo

   on Jun 5, 2019

   ContributorAuthor

   More actions

   Nope, it doesn't (see man page). We can simply use FICLONE (cp does the same).
8. albertz commented on Dec 29, 2020

   

   albertzmannequin

   on Dec 29, 2020

   Mannequin

   More actions

   According to the man page of copy_file_range (https://man7.org/linux/man-pages/man2/copy_file_range.2.html), copy_file_range also should support copy-on-write:

     copy_file_range() gives filesystems an opportunity to implement
     "copy acceleration" techniques, such as the use of reflinks
     (i.e., two or more inodes that share pointers to the same copy-
     on-write disk blocks) or server-side-copy (in the case of NFS).
   

   Is this wrong?

   However, while researching more about FICLONE vs copy_file_range, I found e.g. this: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=24399

   Which suggests that there are other problems with copy_file_range?
9. ezio-melotti

   transferred this issue fromon Apr 10, 2022
10. illia-v commented on Apr 16, 2022

    

    illia-v

    on Apr 16, 2022

    Contributor

    More actions

    FYI, GNU Coreutils 9.0 (released in September 2021) changed cp to:

    • use copy offload via copy_file_range where available;
    • enable CoW by default.

    https://lists.gnu.org/archive/html/info-gnu/2021-09/msg00010.html
11. iritkatriel

    added

    3.12only security fixesonly security fixes

    and removed

    3.9 (EOL)end of lifeend of life

    on Aug 26, 2022
12. barneygale

    added a commit that references this issue on Jun 14, 2024

    GH-73991: Add `pathlib.Path.copy()` (#119058)

    7c38097
13. mrahtz

    added a commit that references this issue on Jun 30, 2024

    pythonGH-73991: Add `pathlib.Path.copy()` (python#119058)

    7e37fbe
14. estyxx

    added a commit that references this issue on Jul 17, 2024

    pythonGH-73991: Add `pathlib.Path.copy()` (python#119058)

    595b213
15. zooba

    added a commit that references this issue on Feb 3, 2025

    gh-81340: Use copy_file_range in shutil.copyfile copy functions (GH-9…

    a33dcb9
16. zooba commented on Feb 3, 2025

    

    zooba

    on Feb 3, 2025

    Member

    More actions

    I've merged the changes so they get a bit of a run in 3.14 alpha. Nobody (apart from the contributor) seemed willing to step up and say that it's definitely the right thing to do.

    If there are issues, feel free to revert or modify. This isn't "my" code, I don't need to be consulted.
17. zooba

    closed this as completedon Feb 3, 2025
18. srinivasreddy

    added a commit that references this issue on Feb 7, 2025

    pythongh-81340: Use copy_file_range in shutil.copyfile copy functions (…

    e43fa23

Issue actions