GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-103987: fix crash in mmap module by Agent-Hellboy · Pull Request #103990 · python/cpython

Conversation

@Agent-Hellboy

@Agent-Hellboy Agent-Hellboy commented Apr 29, 2023 •
edited by bedevere-bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor
  • Issue: segfault in mmap object when using __index__ method that closes the mmap #103987

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

arhadthedev reacted with thumbs up emoji
All reactions
  • 👍 1 reaction
@Agent-Hellboy
pythongh-103987: fix crash in mmap module
d5cf0dd
@bedevere-bot bedevere-bot mentioned this pull request Apr 29, 2023
segfault in mmap object when using __index__ method that closes the mmap #103987
Closed
@bedevere-bot bedevere-bot added the awaiting review label Apr 29, 2023
@arhadthedev arhadthedev added the extension-modules C modules in the Modules dir label Apr 29, 2023
@Agent-Hellboy
fix test
af99baa
@arhadthedev arhadthedev added the needs backport to 3.11 only security fixes label Apr 29, 2023
@Agent-Hellboy

Agent-Hellboy commented Apr 29, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor Author

I am closing the mmap object, still the test is failing

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

Eclips4
Eclips4 reviewed Apr 29, 2023
View reviewed changes
Comment thread Lib/test/test_mmap.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Eclips4
Eclips4 reviewed Apr 29, 2023
View reviewed changes
Comment thread Lib/test/test_mmap.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Agent-Hellboy added 2 commits April 29, 2023 23:42
@Agent-Hellboy
fix test
6b7af10
@Agent-Hellboy
fix test
6ab6a70
@Eclips4

Eclips4 commented Apr 29, 2023

Copy link
Copy Markdown
Member

Let's wait CI/CD results. Also, can you add a few lines about this in NEWS entry?

Agent-Hellboy reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

blurb-it Bot and others added 2 commits April 29, 2023 18:23
@blurb-it
📜🤖 Added by blurb_it.
50cc20b
@Agent-Hellboy
fix doc
4d26d53
@Eclips4

Eclips4 commented Apr 29, 2023

Copy link
Copy Markdown
Member

Great work @Agent-Hellboy !
Let's see what core devs will say.

Agent-Hellboy reacted with heart emoji
All reactions
  • ❤️ 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@cfbolz

cfbolz commented Apr 29, 2023

Copy link
Copy Markdown
Contributor

still some cases not covered properly. Now that you moved the CHECK_VALID(NULL); into the if-branch, you don't check whether the mmap is closed any more if the index is a slice. try adding a test like:

m = mmap(...)
m.close()
m[0:5] # should complain about a closed file

also you should add a test that uses the weird X class as part of a slice

m[X():5]

and you need to do the same thing again for assignment, both with X as an index and a slice:

m[X()] = 1
...
m[X():5] = b'abcde'

or something like that.

Also there needs to be a test for the special case I mentioned at the end of the issue:

m = mmap(...)
m.close()
with self.assertRaises(ValueError):
    m["abc"] # not TypeError!
Eclips4 reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@Agent-Hellboy
fix test
d8a1365
@Agent-Hellboy

Agent-Hellboy commented Apr 29, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor Author

hi @cfbolz, Now I am checking at the start as well which denies throwing TypeError for closed mmap

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@cfbolz

cfbolz commented Apr 29, 2023

Copy link
Copy Markdown
Contributor

now you need to do the same things for item assignments still.

sunmy2019 reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

sunmy2019
sunmy2019 requested changes Apr 29, 2023
View reviewed changes
Comment thread Modules/mmapmodule.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot bedevere-bot added awaiting core review and removed awaiting review labels Apr 29, 2023
@sunmy2019

sunmy2019 commented Apr 29, 2023

Copy link
Copy Markdown
Member

We should CHECK_VALID(NULL); before each self->data access, unless we have strong proof that our handle is not closed.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@Agent-Hellboy
add new test
428338f
@sunmy2019

sunmy2019 commented Apr 30, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

@Agent-Hellboy

I am saying you should add these

    else if (PySlice_Check(item)) {
        Py_ssize_t start, stop, step, slicelen;
        if (PySlice_Unpack(item, &start, &stop, &step) < 0) {
            return NULL;
        }
        CHECK_VALID(NULL); /////////////////////////////////////////////////////////////////
        slicelen = PySlice_AdjustIndices(self->size, &start, &stop, step);
        if (slicelen <= 0)
            return PyBytes_FromStringAndSize("", 0);
        else if (step == 1)
            return PyBytes_FromStringAndSize(self->data + start,
                                              slicelen);
        else {
            char *result_buf = (char *)PyMem_Malloc(slicelen);
            size_t cur;
            Py_ssize_t i;
            PyObject *result;
            if (result_buf == NULL)
                return PyErr_NoMemory();
            CHECK_VALID(NULL); /////////////////////////////////////////////////////////////////

            for (cur = start, i = 0; i < slicelen;
                 cur += step, i++) {
                result_buf[i] = self->data[cur];
            }
            result = PyBytes_FromStringAndSize(result_buf,
                                                slicelen);
            PyMem_Free(result_buf);
            return result;
        }
    }
    else {
        PyErr_SetString(PyExc_TypeError,
                        "mmap indices must be integers");
        return NULL;
    }
}
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@Agent-Hellboy

Agent-Hellboy commented Apr 30, 2023

Copy link
Copy Markdown
Contributor Author

Hi @sunmy2019, please share with me a scenario where it's bypassing the first CHECK_VALID(NULL); at the start for a slice

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@sunmy2019

sunmy2019 commented Apr 30, 2023

Copy link
Copy Markdown
Member

Hi @sunmy2019, please share with me a scenario where it's bypassing the first CHECK_VALID(NULL); at the start for a slice

m[X():5]

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@Agent-Hellboy

Agent-Hellboy commented Apr 30, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor Author

Currently, it's throwing expected Value error, i have added a test for the same

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

sunmy2019
sunmy2019 requested changes Apr 30, 2023
View reviewed changes
Comment thread Lib/test/test_mmap.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

@sunmy2019

sunmy2019 commented Apr 30, 2023

Copy link
Copy Markdown
Member

Currently, it's throwing expected Value error

This is wrong. And you are not creating a new m, which covered your mistake.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

sunmy2019 and others added 3 commits May 12, 2023 18:25
@sunmy2019 @Agent-Hellboy
Update Misc/NEWS.d/next/Library/2023-04-29-18-23-16.gh-issue-103987.s…
49d1094
…RgALL.rst
@Agent-Hellboy
Add new test
61cdd6e
@Agent-Hellboy
Merge branch 'fix-issue-103987' of https://github.com/Agent-Hellboy/c…
5c8d709
…python into fix-issue-103987
@Agent-Hellboy
Agent-Hellboy requested a review from markshannon as a code owner May 12, 2023 13:31
sunmy2019
sunmy2019 reviewed May 12, 2023
View reviewed changes
Comment thread Misc/NEWS.d/next/Library/2023-04-29-18-23-16.gh-issue-103987.sRgALL.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

@sunmy2019
Update Misc/NEWS.d/next/Library/2023-04-29-18-23-16.gh-issue-103987.s…
341b90a
…RgALL.rst
sunmy2019
sunmy2019 reviewed May 12, 2023
View reviewed changes
Comment thread Misc/NEWS.d/next/Library/2023-04-29-18-23-16.gh-issue-103987.sRgALL.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

sunmy2019 and others added 5 commits May 12, 2023 21:50
@sunmy2019
Update Misc/NEWS.d/next/Library/2023-04-29-18-23-16.gh-issue-103987.s…
6dece9f
…RgALL.rst
@Agent-Hellboy
fix test
5cf8f90
@Agent-Hellboy
Merge branch 'fix-issue-103988' of https://github.com/Agent-Hellboy/c…
93163d9
…python into fix-issue-103987
@sunmy2019
Merge branch 'main' into fix-issue-103987
3b61c74
@sunmy2019
Merge branch 'main' into fix-issue-103987
f2d2d00
JelleZijlstra
JelleZijlstra reviewed May 19, 2023
View reviewed changes
Comment thread Misc/NEWS.d/next/Library/2023-04-29-18-23-16.gh-issue-103987.sRgALL.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

@sunmy2019

sunmy2019 commented May 19, 2023

Copy link
Copy Markdown
Member

then we access self->fd

I think checking self->fd != -1 is enough here.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

sunmy2019 and others added 3 commits May 20, 2023 04:35
@sunmy2019
add more checks, fix test cases
843c2eb
@sunmy2019 @JelleZijlstra
Update Misc/NEWS.d/next/Library/2023-04-29-18-23-16.gh-issue-103987.s…
168f5b1
…RgALL.rst

Co-authored-by: Jelle Zijlstra <[email protected]>
@sunmy2019
remove trailing spaces
435ed41
@sunmy2019

sunmy2019 commented May 19, 2023

Copy link
Copy Markdown
Member

I think it's now ready for review. @JelleZijlstra @cfbolz

Sorry for the delay!

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@JelleZijlstra
JelleZijlstra self-requested a review May 19, 2023 20:48
JelleZijlstra
JelleZijlstra reviewed May 19, 2023
View reviewed changes
Comment thread Lib/test/test_mmap.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Modules/mmapmodule.c

if (result_buf == NULL)
return PyErr_NoMemory();

@JelleZijlstra JelleZijlstra May 19, 2023

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could the buffer have been invalidated here as a side effect of the PyMem_Malloc call above triggering GC? I seem to recall that's possible, but not 100% sure.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@sunmy2019 sunmy2019 May 20, 2023

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I cannot find any GC-related code in Object/obmalloc.c

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@sunmy2019 @JelleZijlstra
Update Lib/test/test_mmap.py
320feac
Co-authored-by: Jelle Zijlstra <[email protected]>
@JelleZijlstra
JelleZijlstra merged commit ceaa4c3 into python:main May 20, 2023
@miss-islington

miss-islington commented May 20, 2023

Copy link
Copy Markdown
Contributor

Thanks @Agent-Hellboy for the PR, and @JelleZijlstra for merging it 🌮🎉.. I'm working now to backport this PR to: 3.11.
🐍🍒⛏🤖 I'm not a witch! I'm not a witch!

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot bedevere-bot removed the awaiting core review label May 20, 2023
@bedevere-bot

bedevere-bot commented May 20, 2023

Copy link
Copy Markdown

GH-104677 is a backport of this pull request to the 3.11 branch.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot bedevere-bot removed the needs backport to 3.11 only security fixes label May 20, 2023
miss-islington pushed a commit to miss-islington/cpython that referenced this pull request May 20, 2023
@Agent-Hellboy @sunmy2019
@JelleZijlstra @miss-islington
pythongh-103987: fix several crashes in mmap module (pythonGH-103990)
7f358a0
(cherry picked from commit ceaa4c3)

Co-authored-by: Prince Roshan <[email protected]>
Co-authored-by: sunmy2019 <[email protected]>
Co-authored-by: Jelle Zijlstra <[email protected]>
JelleZijlstra added a commit that referenced this pull request May 20, 2023
@miss-islington @Agent-Hellboy
@sunmy2019 @JelleZijlstra
[3.11] gh-103987: fix several crashes in mmap module (GH-103990) (#10…
dbb011a
…4677)

gh-103987: fix several crashes in mmap module (GH-103990)
(cherry picked from commit ceaa4c3)

Co-authored-by: Prince Roshan <[email protected]>
Co-authored-by: sunmy2019 <[email protected]>
Co-authored-by: Jelle Zijlstra <[email protected]>
carljm added a commit to gsallam/cpython_with_perfmap_apii that referenced this pull request May 20, 2023
@carljm
Merge branch 'main' into gsallam/main
ea5965a
* main: (30 commits)
  pythongh-103987: fix several crashes in mmap module (python#103990)
  docs: fix wrong indentation causing rendering error in dis page (python#104661)
  pythongh-94906: Support multiple steps in math.nextafter (python#103881)
  pythongh-104472: Skip `test_subprocess.ProcessTestCase.test_empty_env` if ASAN is enabled (python#104667)
  pythongh-103839: Allow building Tkinter against Tcl 8.7 without external libtommath (pythonGH-103842)
  pythongh-85984: New additions and improvements to the tty library. (python#101832)
  pythongh-104659: Consolidate python examples in enum documentation (python#104665)
  pythongh-92248: Deprecate `type`, `choices`, `metavar` parameters of `argparse.BooleanOptionalAction` (python#103678)
  pythongh-104645: fix error handling in marshal tests (python#104646)
  pythongh-104600: Make type.__type_params__ writable (python#104634)
  pythongh-104602: Add additional test for listcomp with lambda (python#104639)
  pythongh-104640: Disallow walrus in comprehension within type scopes (python#104641)
  pythongh-103921: Rename "type" header in argparse docs (python#104654)
  Improve readability of `typing._ProtocolMeta.__instancecheck__` (python#104649)
  pythongh-96522: Fix deadlock in pty.spawn (python#96639)
  pythonGH-102818: Do not call `PyTraceBack_Here` in sys.settrace trampoline.  (pythonGH-104579)
  pythonGH-103545: Add macOS specific constants for ``os.setpriority`` to ``os`` (python#104606)
  pythongh-104623: Update macOS installer to SQLite 3.42.0 (pythonGH-104624)
  pythongh-104619: never leak comprehension locals to outer locals() (python#104637)
  pythongh-104602: ensure all cellvars are known up front (python#104603)
  ...
@Eclips4 Eclips4 mentioned this pull request May 20, 2023
test_mmap are leaked #104698
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@JelleZijlstra JelleZijlstra JelleZijlstra left review comments
@sunmy2019 sunmy2019 sunmy2019 requested changes
@Eclips4 Eclips4 Eclips4 left review comments
@markshannon markshannon Awaiting requested review from markshannon

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

extension-modules C modules in the Modules dir

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

8 participants

@Agent-Hellboy @Eclips4 @cfbolz @sunmy2019 @JelleZijlstra @miss-islington @bedevere-bot @arhadthedev
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.