GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-78502: Add a trackfd parameter to mmap.mmap() by ZackerySpytz · Pull Request #25425 · python/cpython

Conversation

@ZackerySpytz

@ZackerySpytz ZackerySpytz commented Apr 15, 2021 •
edited by bedevere-app Bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

If trackfd is False, the file descriptor specified by fileno
will not be duplicated.

https://bugs.python.org/issue34321

  • Issue: mmap.mmap() should not necessarily clone the file descriptor #78502

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@the-knights-who-say-ni the-knights-who-say-ni added the CLA signed label Apr 15, 2021
@bedevere-bot bedevere-bot added the awaiting review label Apr 15, 2021
@ZackerySpytz
bpo-34321: Add a trackfd parameter to mmap.mmap()
d42762c
If *trackfd* is False, the file descriptor specified by *fileno*
will not be duplicated.
@ZackerySpytz
ZackerySpytz force-pushed the bpo-34321-mmap-trackfd-parameter branch from f46e71e to d42762c Compare April 15, 2021 17:06
@MojoVampire

MojoVampire commented Apr 15, 2021

Copy link
Copy Markdown
Contributor

Could you make the new trackfd argument keyword-only? When the number of arguments starts getting large, especially when it's an argument that's boolean and not self-describing, it seems like there is a preference for making the new arguments keyword-only both to prevent using the API in needlessly unclear ways and to avoid a long term compatibility requirement that prevents changing the argument order (see evolution of subprocess.Popen and subprocess.run). I suspect, in hindsight, mmap would have liked to keep more arguments keyword-only to avoid the issue with differing argument orders on UNIX-likes and Windows, but that ship has sailed; at least we can avoid making it any worse.

Also, is there a reason you're not doing this for Windows? The same feature seems perfectly reasonable in Windows too.

serhiy-storchaka and encukou reacted with thumbs up emoji
All reactions
  • 👍 2 reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@github-actions

github-actions Bot commented Jun 3, 2021

Copy link
Copy Markdown

This PR is stale because it has been open for 30 days with no activity.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@github-actions github-actions Bot added the stale Stale PR or inactive for long period of time. label Jun 3, 2021
@ezio-melotti ezio-melotti removed the CLA signed label Jul 13, 2022
@github-actions github-actions Bot removed the stale Stale PR or inactive for long period of time. label Aug 6, 2022
@erlend-aasland erlend-aasland changed the title bpo-34321: Add a trackfd parameter to mmap.mmap() gh-78502: Add a trackfd parameter to mmap.mmap() Jan 5, 2024
@manuels manuels mannequin mentioned this pull request Jan 5, 2024
mmap.mmap() should not necessarily clone the file descriptor #78502
Closed
@erlend-aasland
Merge branch 'main' into bpo-34321-mmap-trackfd-parameter
8d141ad
@erlend-aasland

erlend-aasland commented Jan 5, 2024

Copy link
Copy Markdown
Contributor

cc. @encukou / @serhiy-storchaka: would one of you like to take a look at the proposed solution?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@encukou encukou self-assigned this Jan 5, 2024
@encukou

encukou commented Jan 8, 2024

Copy link
Copy Markdown
Member

This is a good start! @ZackerySpytz, do you want to continue or should I finish the feature?

IMO, it's fine to keep it *nix-only, at least initially. It should be documented as such though.
Also, size/resize should be tested to ensure they raise the proper exception rather than crash, and their docs should mention the new error condition. (
I'd also prefer to test other functionality, like __setitem__, flush. If/when Windows support is added, the test should detect any platform differences.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@erlend-aasland

erlend-aasland commented Jan 8, 2024

Copy link
Copy Markdown
Contributor

This is a good start! @ZackerySpytz, do you want to continue or should I finish the feature?

Zackery is not very active these days; I think it is fine to take over the PR.

serhiy-storchaka reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

encukou added 5 commits January 9, 2024 10:53
@encukou
Merge in the main branch
12705a6
@encukou
Make the trackfd argument keyword-only
42302bc
@encukou
Add smoke tests for a mmap with trackfd=False
9443f1f
@encukou
Update documentation
eb8d798
@encukou
Make resize() fail with trackfd=False
8f05c7b
@encukou

encukou commented Jan 9, 2024

Copy link
Copy Markdown
Member

OK! Here are my updates.

I've realized that resize() method works fine without a fd -- it'll resize the map, but not the underlying file, so if the map grows, the “new” part becomes unmapped.
The latest commit here fixes that -- at the cost of an additional flag. (The memory cost in negligible IMO -- and could be optimized away -- but there's a maintenance cost & mental overhead.)
I'm not sure if resize() with trackfd=False would ever be useful. Making it fail is a safer initial implementation; it can always be enabled.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@encukou
encukou marked this pull request as ready for review January 9, 2024 12:47
serhiy-storchaka
serhiy-storchaka reviewed Jan 9, 2024
View reviewed changes

@serhiy-storchaka serhiy-storchaka left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that it would be interesting to add the following tests:

  • Test that it fails on Windows with trackfd=False and trackfd=True.
  • Test with fd=-1.
  • Test with original fd closed after creating a mmap.
  • Test whether mmap can be used after fork.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Doc/library/mmap.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Doc/library/mmap.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Lib/test/test_mmap.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Modules/mmapmodule.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

@encukou

encukou commented Jan 9, 2024

Copy link
Copy Markdown
Member

Test whether mmap can be used after fork.

I think that's too much: it'd be testing the platform, rather than CPython itself.

erlend-aasland reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

encukou added 5 commits January 9, 2024 16:13
@encukou
Test with original fd closed after creating mmap
b207558
@encukou
Document trackfd as keyword-only
7607593
@encukou
Update docstring
2d90229
@encukou
Test with fd=-1, and that this fails on Windows
0a717f8
@encukou
Test the errno
7364a27
serhiy-storchaka
serhiy-storchaka reviewed Jan 9, 2024
View reviewed changes
Comment thread Doc/whatsnew/3.10.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Lib/test/test_mmap.py
Comment on lines +309 to +312
with self.assertRaises(OSError):
m.size()
with self.assertRaises(TypeError):
m.resize(size // 2)

@serhiy-storchaka serhiy-storchaka Jan 9, 2024

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does it work the same for mmap(-1, size, trackfd=True)? I have not found tests for this.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@encukou encukou Jan 10, 2024

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

trackfd=True is the default. It's tested in test_resize_past_pos.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@serhiy-storchaka serhiy-storchaka Jan 10, 2024

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It has different behavior. Should trackfd=False have any effect for fd=-1? Should combination fd=-1 and trackfd=False be allowed?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@encukou encukou Jan 10, 2024

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Conceptually, trackfd=False should be the only possible behaviour for fd=-1 -- the fd is not tracked.
However, I don't think it's worth it to include a third value for “trackfd=default”. As it is, the default (True) works as before. Setting False has the documented effect -- disabling resize. This isn't very useful, but I don't think it's worth handling specially.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Modules/mmapmodule.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

encukou added 2 commits January 10, 2024 14:16
@encukou
Move the What's New entry to 3.13
eca802c
@encukou
Raise ValueEror on resize
51bb4e5
serhiy-storchaka
serhiy-storchaka reviewed Jan 10, 2024
View reviewed changes

@serhiy-storchaka serhiy-storchaka left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By default, the mmap is always created from non-inheritable file descriptor. But if trackfd is false, it can be created from inheritable file descriptor. I wondered what is the difference and how can it be tested.

But if it works after closing the original file descriptor, perhaps there is no difference. cc @vstinner

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Doc/library/mmap.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

@encukou @serhiy-storchaka
Fix Sphinx warning
d274952
Co-authored-by: Serhiy Storchaka <[email protected]>
@encukou

encukou commented Jan 11, 2024

Copy link
Copy Markdown
Member

There's always another test that could be added. I think we're too far into diminishing returns at this point :)

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@serhiy-storchaka

serhiy-storchaka commented Jan 11, 2024

Copy link
Copy Markdown
Member

There is currently a discussion about inheritance of file descriptors in #113817. I just want to be sure that we do not miss some details here (most likely it is all correct). cc @vstinner, @zooba, @eryksun.

encukou reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

zooba
zooba reviewed Jan 11, 2024
View reviewed changes

@zooba zooba left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Our other discussion was specific to Windows, so this issue doesn't apply.

But since I got mentioned, could we get the docs phrased in a way that explains why I might ever want to use the new option? Right now it sounds like a bad thing.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

encukou reacted with thumbs up emoji
All reactions
  • 👍 1 reaction
Comment thread Doc/library/mmap.rst
defaults to 0. *offset* must be a multiple of :const:`ALLOCATIONGRANULARITY`
which is equal to :const:`PAGESIZE` on Unix systems.

If *trackfd* is ``False``, the file descriptor specified by *fileno* will

@zooba zooba Jan 11, 2024

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd love to have some idea of why I might want to use this parameter. Right now it only describes the downsides.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

erlend-aasland and serhiy-storchaka reacted with thumbs up emoji
All reactions
  • 👍 2 reactions

@eryksun eryksun Jan 12, 2024

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On Windows, the internally duplicated handle probably references an open that lacks delete access. It thus prevents deleting the file, even if the mapped section otherwise allows it (e.g. the section is mapped readonly). For example:

>>> f = open('spam.txt')
>>> m = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ)
>>> f.close()
>>> os.remove('spam.txt')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'spam.txt'

>>> # I manually closed the internal handle via Process Explorer.
>>> os.remove('spam.txt')
>>> m[:]
b'spam'

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@zooba zooba Jan 12, 2024

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That sounds like a reason to at least add the argument for all platforms, which I'm generally in favour of anyway. It can have more appropriate semantics on Windows if needed (i.e. "doesn't hold an extra HANDLE" rather than "FD").

It's probably actually pretty useful to be able to immediately delete the file but keep the mapping open (which will keep the file on disk on Windows at least, so you can't reuse the name while it's in use). And it looks like the mapping doesn't lock out deletes, so I guess it'll work as intended.

I'm not going to hold up this PR for it though. All I'll say is that if we ever do add that option, it should be trackfd=False to "activate" it, for consistency between platforms.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@eryksun eryksun Jan 12, 2024

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That sounds like a reason to at least add the argument for all platforms, which I'm generally in favour of anyway. It can have more appropriate semantics on Windows if needed (i.e. "doesn't hold an extra HANDLE" rather than "FD").

I think trackfd would be fine on Windows. The fileno parameter is a C file descriptor, not a native OS handle.

It's probably actually pretty useful to be able to immediately delete the file but keep the mapping open (which will keep the file on disk on Windows at least, so you can't reuse the name while it's in use). And it looks like the mapping doesn't lock out deletes, so I guess it'll work as intended.

NTFS supports POSIX delete, in which a deleted file gets renamed to a reserved system directory until all references to the file object have been closed. That includes the internal pointer reference to a file object that's held by the memory manager for the mapped section. The internal file reference doesn't count toward the file's share mode, i.e. a memory-mapped file can be deleted even if the source open didn't share delete access. Actually, I just checked that the delete is allowed nowadays even if the mapped section has write access to the file, so my assumption was wrong that it would only work for a readonly mapping.

You can observe this in Process Explorer. Switch the lower-pane view to DLLs (file- and pagefile-backed memory mappings), and add the name and path columns to the view. You'll see that the backing file gets moved to the "\$Extend\$Deleted" system directory on the volume after the file is 'deleted'.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@encukou
Note why you might want to use this
e33a7fd
@encukou

encukou commented Jan 15, 2024

Copy link
Copy Markdown
Member

I'd prefer if the Windows functionality was added in a separate PR. I don't have a Windows box set up (yet), so I won't send one myself.
Other than that, if the added note is OK this looks good to merge.

zooba reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

serhiy-storchaka
serhiy-storchaka reviewed Jan 15, 2024
View reviewed changes

@serhiy-storchaka serhiy-storchaka left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
serhiy-storchaka
serhiy-storchaka approved these changes Jan 15, 2024
View reviewed changes
@bedevere-app bedevere-app Bot added awaiting merge and removed awaiting review labels Jan 15, 2024
@encukou
encukou merged commit 8fd287b into python:main Jan 16, 2024
@bedevere-app bedevere-app Bot removed the awaiting merge label Jan 16, 2024
kulikjak pushed a commit to kulikjak/cpython that referenced this pull request Jan 22, 2024
@ZackerySpytz @erlend-aasland
@encukou @serhiy-storchaka @kulikjak
pythongh-78502: Add a trackfd parameter to mmap.mmap() (pythonGH-25425)
fcd786b
If *trackfd* is False, the file descriptor specified by *fileno*
will not be duplicated.

Co-authored-by: Erlend E. Aasland <[email protected]>
Co-authored-by: Petr Viktorin <[email protected]>
Co-authored-by: Serhiy Storchaka <[email protected]>
aisk pushed a commit to aisk/cpython that referenced this pull request Feb 11, 2024
@ZackerySpytz @erlend-aasland
@encukou @serhiy-storchaka @aisk
pythongh-78502: Add a trackfd parameter to mmap.mmap() (pythonGH-25425)
7024b8d
If *trackfd* is False, the file descriptor specified by *fileno*
will not be duplicated.

Co-authored-by: Erlend E. Aasland <[email protected]>
Co-authored-by: Petr Viktorin <[email protected]>
Co-authored-by: Serhiy Storchaka <[email protected]>
Glyphack pushed a commit to Glyphack/cpython that referenced this pull request Sep 2, 2024
@ZackerySpytz @erlend-aasland
@encukou @serhiy-storchaka @Glyphack
pythongh-78502: Add a trackfd parameter to mmap.mmap() (pythonGH-25425)
94a648d
If *trackfd* is False, the file descriptor specified by *fileno*
will not be duplicated.

Co-authored-by: Erlend E. Aasland <[email protected]>
Co-authored-by: Petr Viktorin <[email protected]>
Co-authored-by: Serhiy Storchaka <[email protected]>
@potiuk potiuk mentioned this pull request Jul 17, 2025
Add Python 3.13 support for Airflow. apache/airflow#46891
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@encukou encukou encukou left review comments
@eryksun eryksun eryksun left review comments
@zooba zooba zooba left review comments
@serhiy-storchaka serhiy-storchaka serhiy-storchaka approved these changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

@encukou encukou

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

10 participants

@ZackerySpytz @MojoVampire @erlend-aasland @encukou @serhiy-storchaka @eryksun @zooba @the-knights-who-say-ni @ezio-melotti @bedevere-bot
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.