GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-109534: switch from sock_call to sock_call_ex in sock_send by geraldog · Pull Request #114311 · python/cpython

Conversation

@geraldog

@geraldog geraldog commented Jan 19, 2024 •
edited by bedevere-app Bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown

While writing webcrawlers with https://github.com/sonic182/aiosonic I came across the issues best described in #109534

I had the great idea of explicitly deleting the event loop my crawler was running on, and hey - I got a traceback of the leaks, repeated ad nauseam:

OSError: [Errno 9] Bad file descriptor

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.10/asyncio/sslproto.py", line 692, in _process_write_backlog
    self._transport.write(chunk)
  File "/usr/lib/python3.10/asyncio/selector_events.py", line 935, in write
    self._fatal_error(exc, 'Fatal write error on socket transport')
  File "/usr/lib/python3.10/asyncio/selector_events.py", line 729, in _fatal_error
    self._force_close(exc)
  File "/usr/lib/python3.10/asyncio/selector_events.py", line 741, in _force_close
    self._loop.call_soon(self._call_connection_lost, exc)
  File "/usr/lib/python3.10/asyncio/base_events.py", line 753, in call_soon
    self._check_closed()
  File "/usr/lib/python3.10/asyncio/base_events.py", line 515, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
Fatal error on SSL transport
protocol: <asyncio.sslproto.SSLProtocol object at 0x7fb576dadab0>
transport: <_SelectorSocketTransport closing fd=329>
Traceback (most recent call last):
  File "/usr/lib/python3.10/asyncio/selector_events.py", line 929, in write
    n = self._sock.send(data)

Notice this traceback happens with Python <= 3.10 only. Python >= 3.11 leaks forever regardless of whether the loop is deleted or not.

That got me thinking, there must be something special about the socket.send() function that isn't raising the exception... so I went looking in socketmodules.c, and lo and behold, there are multiple instances in that file where we use the sock_call() thin wrapper around sock_call_ex().

The problem is that wrapper does not set an err pointer to pass to sock_call_ex(). As a result, the exception may never be raised.

As I said, multiple instances here, this one-liner won't solve #109534 by itself but I believe it's a step forward. I have a crawler running much more stabler now with this fix, particularly when I have to abort() the underlying SSL transport.

  • Issue: TLS/SSL asyncio leaks memory  #109534

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@geraldog
pythongh-109534: switch from sock_call to sock_call_ex in sock_send
c566d01
@ghost

ghost commented Jan 19, 2024

Copy link
Copy Markdown

The following commit authors need to sign the Contributor License Agreement:

  • [email protected]

Click the button to sign:
CLA not signed
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-app

bedevere-app Bot commented Jan 19, 2024

Copy link
Copy Markdown

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-app bedevere-app Bot added the awaiting review label Jan 19, 2024
@bedevere-app bedevere-app Bot mentioned this pull request Jan 19, 2024
TLS/SSL asyncio leaks memory #109534
Closed
@geraldog

geraldog commented Jan 21, 2024

Copy link
Copy Markdown
Author

Closed in favor of #114367
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@geraldog geraldog closed this Jan 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

awaiting review

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

1 participant

@geraldog
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.