GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — bpo-30555: Fix WindowsConsoleIO errors in the presence of fd redirection by segevfiner · Pull Request #1927 · python/cpython

Conversation

@segevfiner

@segevfiner segevfiner commented Jun 3, 2017 •
edited by erlend-aasland
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

This works by not caching the handle and instead getting the handle from the file descriptor each time, so that if the actual handle changes by fd redirection closing/opening the console handle beneath our feet, we will keep working correctly.

I think I also fixed some resource handling issues along the way since ucrt/msvcrt fds own the HANDLEs they use and this module tried to close them itself.

See pytest-dev/py#103 for the issue it causes in Pytest. Also see: pytest-dev/pytest#2462. I was able to run the Pytest test suite, and my own little test described in the PR, to completion after this change.

Having your console handle closed by fd redirection is likely a change in the ucrt/msvcrt or a change in newer Windows versions (Console handles were rewritten in Windows 8).

This likely also affects any other module caching GetStdHandle or _get_osfhandle. For example colorama (pytest-dev/pytest#2465 - tartley/colorama#131), and I can think of "pyreadline", "win-unicode-console" and "click" probably suffering from the same issue but I haven't tested them.

I'm also not sure of the performance implications of this change.

This affects Python 3.6-3.7

P.S. python -m test -R 3:2 test.test_winconsoleio reports ref leaks on master for me 😛

CC @zooba @eryksun

  • bpo-30555

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

nicoddemus reacted with thumbs up emoji
All reactions
  • 👍 1 reaction
@mention-bot

mention-bot commented Jun 3, 2017

Copy link
Copy Markdown

@segevfiner, thanks for your PR! By analyzing the history of the files in this pull request, we identified @zooba, @benjaminp and @serhiy-storchaka to be potential reviewers.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@the-knights-who-say-ni the-knights-who-say-ni added the CLA signed label Jun 3, 2017
@nicoddemus nicoddemus mentioned this pull request Jun 3, 2017
[WIP] A workaround for Python 3.6 WindowsConsoleIO breaking with FDCapture pytest-dev/pytest#2462
Merged
3 of 4 tasks
@eryksun

eryksun commented Jun 4, 2017

Copy link
Copy Markdown
Contributor

Most of the reference leaks are due to initializing decodedname = Py_None followed by Py_INCREF(decodedname). It should be initialized as NULL.

Some of the reported ref leaks are from calling getwindowsversion in test_conout_path. This call can be moved out of the test by using a global variable, or by splitting the test up like this:

def conout_path_test(self, dos_device):
    temp_path = tempfile.mkdtemp()
    self.addCleanup(support.rmtree, temp_path)
    conout_path = os.path.join(temp_path, 'CONOUT$')

    with open(conout_path, 'wb', buffering=0) as f:
        if dos_device:
            self.assertIsInstance(f, ConIO)
        else:
            self.assertNotIsInstance(f, ConIO)

@unittest.skipIf(sys.getwindowsversion()[:2] <= (6, 1),
    "Windows 7 and earlier resolve CONOUT$ in a directory as a disk file")
def test_conout_path_device(self):
    self.conout_path_test(dos_device=True)

@unittest.skipIf(sys.getwindowsversion()[:2] > (6, 1),
    "Windows 8 and later resolve CONOUT$ in a directory as a DOS device")
def test_conout_path_file(self):
    self.conout_path_test(dos_device=False)
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

eryksun
eryksun reviewed Jun 4, 2017
View reviewed changes
Comment thread Modules/_io/winconsoleio.c Outdated

@eryksun eryksun Jun 4, 2017

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_Py_get_osfhandle, _Py_get_osfhandle_noraise, _Py_open_osfhandle, and _Py_open_osfhandle_noraise could be added in Python/fileutils.c. These CRT functions are called in fileutils.c, msvcrtmodule.c, posixmodule.c, mmapmodule.c, myreadline.c, _testconsole.c, and here.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@segevfiner segevfiner Jun 4, 2017 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea.

Do you wish me to:

  1. Do this entire refactor right here in this PR.
  2. Add the functions and only use them in winconsoleio.c and _testconsole.c which I touched in this PR and leave the rest of the modules to a separate PR.
  3. Leave this to a separate PR entirely.

?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@segevfiner segevfiner Jun 4, 2017

Copy link
Copy Markdown
Contributor Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I implemented this, with the full replacement in a separate commit just in case it's frowned upon by a core developer during review.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@segevfiner

segevfiner commented Jun 4, 2017 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor Author

@eryksun So the leaks are real than, nice find! 😄 I can submit a PR/issue with your suggested fix if it helps you or just leave it to you. Why would calling sys.getwindowsversion in a test function leak though? Does that function leak? Or is there something else at play?

EDIT: I can also apply your fix right here if you wish me to.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@eryksun

eryksun commented Jun 4, 2017

Copy link
Copy Markdown
Contributor

It should be ok to fix the _WindowsConsoleIO leak and rewrite the test in this issue. The ref leak in getwindowsversion needs a new issue. If you want to work on it, the problem is that it calls PyTuple_Pack directly on the int objects created from realMajor, etc. PyTuple_Pack increments the ref count of each object instead of stealing it.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@segevfiner segevfiner mentioned this pull request Jun 4, 2017
bpo-30567: Fix refleak in sys.getwindowsversion #1940
Merged
segevfiner added 5 commits June 4, 2017 21:32
@segevfiner
bpo-30555: Fix WindowsConsoleIO failure in the presence of fd redirec…
b02408d
…tion

This works by not caching the handle and instead getting the handle from
the file descriptor each time, so that if the actual handle changes by
fd redirection closing/opening the console handle beneath our feet, we
will keep working correctly.

I think I also fixed some resource handling issues along the way since
ucrt/msvcrt fds *own* the HANDLEs they use and this module tried to
close them itself.
@segevfiner
bpo-30555: Fixed the kludgey PC/_testconsole.c, grumble grumble...
ac242eb
@segevfiner
bpo-30555: Fix refleaks
645dfa3
The change in test_winconsoleio.py is only to avoid the ref leak in
`sys.getwindowsversion`.
@segevfiner
bpo-30555: Add _Py_{get,open}_osfhandle{,_noraise} and use them instead
013d8ea
@segevfiner
bpo-30555: Swapped the rest of the uses of _{get,open}_osfhandle
aaf6020
@segevfiner
segevfiner force-pushed the bpo-30555-windowsconsoleio-fd-redirection branch from ac3af9e to aaf6020 Compare June 4, 2017 18:35
eryksun
eryksun reviewed Jun 4, 2017
View reviewed changes
Comment thread Include/fileutils.h Outdated
PyAPI_FUNC(int) _Py_set_blocking(int fd, int blocking);
#endif /* !MS_WINDOWS */
#else /* MS_WINDOWS */
PyAPI_FUNC(intptr_t) _Py_get_osfhandle_noraise(int fd);

@eryksun eryksun Jun 4, 2017 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought one of benefits would be using HANDLE in the signature instead of intptr_t, to avoid all of the casts. I guess we can't. How about void * like _PyOS_SigintEvent?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@segevfiner segevfiner Jun 5, 2017

Copy link
Copy Markdown
Contributor Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think CPython explicity doesn't include Windows.h anywhere in it's public headers. fileutils.h is included by Python.h so I don't think I can really include Windows.h there. Ae such I don't have HANDLE defined and assuming that someone included Windows.h by himself before this header breaks horribly of course.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@eryksun eryksun Jun 5, 2017

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HANDLE is a typedef for void *, so we can use void * instead, like _PyOS_SigintEvent does.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@segevfiner
bpo-30555: Swap intptr_t to void* in _Py_{get,open}_osfhandle{,_noraise}
7727bc5
@segevfiner
segevfiner force-pushed the bpo-30555-windowsconsoleio-fd-redirection branch from 50ff023 to 7727bc5 Compare June 5, 2017 17:04
eryksun
eryksun reviewed Jun 5, 2017
View reviewed changes
Comment thread PC/msvcrtmodule.c Outdated
@@ -167,15 +167,7 @@ static long
msvcrt_open_osfhandle_impl(PyObject *module, intptr_t handle, int flags)

@eryksun eryksun Jun 5, 2017

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The clinic definition could be updated here, e.g. to use a HANDLE type based on void *:

class HANDLE_converter(CConverter):
    type = 'void *'
    format_unit = '"_Py_PARSE_INTPTR"'

class HANDLE_return_converter(CReturnConverter):
    type = 'void *'

    def render(self, function, data):
        self.declare(data)
        self.err_occurred_if("_return_value == INVALID_HANDLE_VALUE", data)
        data.return_conversion.append(
            'return_value = PyLong_FromVoidPtr(_return_value);\n')

Then change the open_osfhandle parameter to handle: HANDLE and the get_osfhandle return type to -> HANDLE.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
segevfiner added 2 commits June 6, 2017 01:47
@segevfiner
bpo-30555: msvcrtmodule.c: Use an argument clinic convertor instead o…
0644634
…f a cast
@segevfiner
Merge remote-tracking branch 'upstream/master' into bpo-30555-windows…
82a18bb
…consoleio-fd-redirection
@segevfiner

segevfiner commented Jun 8, 2017

Copy link
Copy Markdown
Contributor Author

Merged from master since @Haypo fixed the ref leak separately in #2003.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@segevfiner
Merge remote-tracking branch 'upstream/master' into bpo-30555-windows…
ad77cd3
…consoleio-fd-redirection
@segevfiner

segevfiner commented Jun 10, 2017

Copy link
Copy Markdown
Contributor Author

Merged from master due to argument clinic changes.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

segevfiner added 4 commits June 30, 2017 17:38
@segevfiner
bpo-30555: Also consider NULL as a possible error in HANDLE_return_co…
9dd4ea9
…nverter
@segevfiner
Merge remote-tracking branch 'upstream/master' into bpo-30555-windows…
07a979f
…consoleio-fd-redirection
@segevfiner
Merge remote-tracking branch 'upstream/master' into bpo-30555-windows…
69b13f7
…consoleio-fd-redirection
@segevfiner
Merge remote-tracking branch 'upstream/master' into bpo-30555-windows…
e54f9c0
…consoleio-fd-redirection
@segevfiner
segevfiner requested a review from a team as a code owner December 16, 2017 10:17
@segevfiner
Add NEWS.d entry for bpo-30555
2d252ab
@segevfiner
segevfiner force-pushed the bpo-30555-windowsconsoleio-fd-redirection branch from c3953d5 to 2d252ab Compare December 16, 2017 10:26
@segevfiner
Merge remote-tracking branch 'upstream/master' into bpo-30555-windows…
7990e20
…consoleio-fd-redirection
@segevfiner
segevfiner requested review from lysnikolaou and pablogsal as code owners March 16, 2021 14:23
segevfiner and others added 2 commits March 16, 2021 16:24
@segevfiner @eryksun
fileno should error when closed
bd8233e
Co-authored-by: Eryk Sun <[email protected]>
@segevfiner
Fix faulty call to IOBase.close introduced by automatic merge
b76da3e
@zooba
zooba merged commit 5e437fb into python:master Apr 23, 2021
@bedevere-bot bedevere-bot removed the awaiting review label Apr 23, 2021
@zooba

zooba commented Apr 23, 2021

Copy link
Copy Markdown
Member

Thanks for persisting with this, and sorry for taking so long! Let's put it in and deal with any fallout in 3.10 before backporting.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@zooba zooba added needs backport to 3.8 OS-windows labels Apr 23, 2021
@miss-islington

miss-islington commented Apr 23, 2021

Copy link
Copy Markdown
Contributor

Thanks @segevfiner for the PR, and @zooba for merging it 🌮🎉.. I'm working now to backport this PR to: 3.9.
🐍🍒⛏🤖

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@miss-islington

miss-islington commented Apr 23, 2021

Copy link
Copy Markdown
Contributor

Thanks @segevfiner for the PR, and @zooba for merging it 🌮🎉.. I'm working now to backport this PR to: 3.8.
🐍🍒⛏🤖

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@miss-islington

miss-islington commented Apr 23, 2021

Copy link
Copy Markdown
Contributor

Sorry, @segevfiner and @zooba, I could not cleanly backport this to 3.9 due to a conflict.
Please backport using cherry_picker on command line.
cherry_picker 5e437fb872279960992c9a07f1a4c051b4948c53 3.9

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@miss-islington miss-islington assigned zooba Apr 23, 2021
@miss-islington

miss-islington commented Apr 23, 2021

Copy link
Copy Markdown
Contributor

Sorry @segevfiner and @zooba, I had trouble checking out the 3.8 backport branch.
Please backport using cherry_picker on command line.
cherry_picker 5e437fb872279960992c9a07f1a4c051b4948c53 3.8

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@zooba zooba added needs backport to 3.8 and removed needs backport to 3.8 labels Apr 23, 2021
@miss-islington

miss-islington commented Apr 23, 2021

Copy link
Copy Markdown
Contributor

Thanks @segevfiner for the PR, and @zooba for merging it 🌮🎉.. I'm working now to backport this PR to: 3.8.
🐍🍒⛏🤖

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@miss-islington

miss-islington commented Apr 23, 2021

Copy link
Copy Markdown
Contributor

Thanks @segevfiner for the PR, and @zooba for merging it 🌮🎉.. I'm working now to backport this PR to: 3.9.
🐍🍒⛏🤖 I'm not a witch! I'm not a witch!

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@miss-islington

miss-islington commented Apr 23, 2021

Copy link
Copy Markdown
Contributor

Sorry, @segevfiner and @zooba, I could not cleanly backport this to 3.9 due to a conflict.
Please backport using cherry_picker on command line.
cherry_picker 5e437fb872279960992c9a07f1a4c051b4948c53 3.9

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@miss-islington

miss-islington commented Apr 23, 2021

Copy link
Copy Markdown
Contributor

Sorry @segevfiner and @zooba, I had trouble checking out the 3.8 backport branch.
Please backport using cherry_picker on command line.
cherry_picker 5e437fb872279960992c9a07f1a4c051b4948c53 3.8

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@zooba zooba removed needs backport to 3.8 labels Apr 23, 2021
@segevfiner
segevfiner deleted the bpo-30555-windowsconsoleio-fd-redirection branch February 21, 2022 09:38
@segevfiner segevfiner mannequin mentioned this pull request Apr 10, 2022
_io._WindowsConsoleIO breaks in the face of fd redirection #74740
Closed
This was referenced Jan 5, 2024
gh-77046: Pass the _O_NOINHERIT flag to _open_osfhandle() calls #13739
Closed
[os] pipe() creates inheritable FDs with a bad internal state in Windows #77046
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@eryksun eryksun eryksun left review comments
@zooba zooba Awaiting requested review from zooba
@lysnikolaou lysnikolaou Awaiting requested review from lysnikolaou
@pablogsal pablogsal Awaiting requested review from pablogsal

Uh oh!

There was an error while loading. Please reload this page.

Assignees

@zooba zooba

Labels

OS-windows

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

9 participants

@segevfiner @mention-bot @eryksun @csabella @zooba @miss-islington @brettcannon @the-knights-who-say-ni @bedevere-bot
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.