GeezerVison is showing github.com — TLS/SSL asyncio leaks memory · Issue #109534 · python/cpython
TLS/SSL asyncio leaks memory #109534DescriptionBug report
Bug description:
python3.9 without uvloop doesn't leaks memory (or noticeably smaller).
python3.11+ (and others?) leaks memory A LOT under load (with or without uvloop) - up to +2gb per every test!
test commands:
ab -n50000 -c15000 -r https://127.0.0.1/
(apt install apache2-utils)
import asyncio, ssl, uvloop
class HTTP(asyncio.Protocol):
def __init__(self):
self.transport = None
def connection_made(self, transport):
self.transport = transport
def data_received(self, data):
self.transport.write(
b'HTTP/1.1 200 OK\r\nContent-Length: 0\r\nConnection: keep-alive\r\n\r\n'
)
self.transport.close()
def make_tls_context():
ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
ctx.load_cert_chain('cert.crt', 'cert.key')
return ctx
tls_context = make_tls_context()
loop = uvloop.new_event_loop()
async def start_server(loop):
return await loop.create_server(
HTTP, '127.0.0.1', 443, backlog=65535,
ssl=tls_context)
loop.run_until_complete(start_server(loop))
loop.run_forever()CPython versions tested on:
3.9, 3.11, 3.12
Operating systems tested on:
Debian Linux
Linked PRs
Reacted by dursk, Yann, Suraj, Tobiasz Kędzierski, Nico Neumann and Daniel Loy Reacted by Felippe Medeiros, Suraj, Nico Neumann and Oleg A. Reacted by Sergey Levitin, rojamit, Felippe Medeiros, Suraj, Tobiasz Kędzierski, Nico Neumann and Oleg A. Activityselevit commented on Nov 17, 2023 Can confirm that the issue exists - Python 3.11, and uvloop 0.17.0, when trying to set up many client SSL connections that sometimes reconnects - even directly with asyncio.create_connection, and it's cleaned up not properly in some place.
Sometimes it leaked several megabytes per second for me.
I was able to track this by attaching to a running python process via gdb, and when it leaked, I put a catchpoint for brk syscalls, and examined 40 sequential breakpoint events, when all were related to /usr/lib/python3.10/ssl.py.
(gdb) py-bt
Traceback (most recent call first):
File "/usr/lib/python3.10/ssl.py", line 917, in read
v = self._sslobj.read(len)
<built-in method buffer_updated of uvloop.loop.SSLProtocol object at remote 0xffff758f90c0>
<built-in method run of _contextvars.Context object at remote 0xffff758c7340>
File "/home/ubuntu/py-core/env/lib/python3.10/site-packages/aiohttp/web.py", line 544, in run_app
loop.run_until_complete(main_task)
File "/home/ubuntu/py-core/server_app.py", line 39, in <module>
web.run_app(app_factory(), loop=LOOP, host=args.host, port=args.port)
(gdb)
Also, a gc.get_objects() count was not actually growing, that's why I suppose that this is related a lower-level native implementation. arhadthedev commented on Nov 18, 2023 gvanrossum commented on Nov 18, 2023 Unfortunately I am really bad at this type of low level stuff. :-( If you suspect a leak in the C accelerator, you can disable it and see if the leak goes away. (Or at least becomes less severe.) gvanrossum commented on Nov 18, 2023 OTOH if the problem is in the SSL code, we need a different kind of expert. gvanrossum commented on Nov 18, 2023 Finally please figure out if this is in uvloop or not. If it’s tied to uvloop this is the wrong tracker. selevit commented on Nov 19, 2023 Unfortunately, this bug was reproduced only in our production, and we weren't able to cause it in other environments.
It's already mitigated, as the root cause was in poor stability of connections.
I don't think I would be able to reproduce it again in production without uvloop, because it caused a lot of harm for our system.
But ready to assist and provide more details, if anyone wants to ask me some clarifying questions about it.
Also, there are several related tickets, and I'm not sure if they are all about uvloop:
- asyncio: potential leak of TLS connections #106684 - very similar cpython issue, already closed though.
- Memory leak with Python 3.11.2 aio-libs/aiohttp#7252 - also, an already closed, but recent
aiohttp issue, they were able to mitigate it somehow, but not sure if it was the same bug.
Reacted by Yury Zhuravlev selevit commented on Nov 19, 2023 I actually was able to reproduce this leak isolated several days ago, but it leaked for only ~5 megabytes in several hours. I adopted a repro script from this comment to the aiohttp bug mentioned above to be ready for run, and ran it for about 4 hours.
The initial memory consumption (RSS) had initially stabilised on 258120 bytes, but after several hours increased to ~264000. Hope it's the same bug.
#!/usr/bin/env python3
import aiohttp
import tracemalloc
import ssl
import asyncio
def init():
timeout = aiohttp.ClientTimeout(connect=5, sock_read=5)
ssl_ctx = ssl._create_unverified_context()
conn = aiohttp.TCPConnector(ssl=ssl_ctx, enable_cleanup_closed=True)
session = aiohttp.ClientSession(connector=conn, timeout=timeout, cookie_jar=aiohttp.CookieJar(unsafe=True))
return session
async def fetch(client):
try:
async with client.request('GET', url=f'https://api.pro.coinbase.com/products/BTC-USD/ticker') as r:
msg = await r.text()
print(msg)
except asyncio.CancelledError:
raise
except Exception as err:
print("error", err)
pass
async def main():
requests = 600
clients = [init() for _ in range(requests)]
tracemalloc.start()
try:
while True:
await asyncio.gather(*[fetch(client) for client in clients])
await asyncio.sleep(5)
except asyncio.CancelledError:
pass # end and clean things up
finally:
memory_used = tracemalloc.get_traced_memory()
snapshot = tracemalloc.take_snapshot()
stats = snapshot.statistics('lineno')
for stat in stats[:10]:
print(stat)
try:
await asyncio.gather(*[client.close() for client in clients.values()])
except:
pass
asyncio.run(main())gvanrossum commented on Nov 19, 2023 Could you find a reproducer without aiohttp? Reacted by Sergey Levitin Faolain commented on Nov 27, 2023 Unsure if this is helpful at all or related but I came across this issue Kludex/uvicorn#2078 in which in the thread it is discussed/concluded that the issue of memory not being released is not isolated to uvicorn but seen in granian/gunicorn/hypercorn and as a result could be interpreter level (apologies for butchering the summary). Thread has some great charts/analysis though it is on the server level, however the different implementations w.r.t granian and uvicorn can help approximate where the issue might be surfacing if it's related. Example repo: https://github.com/Besedo/memory_issue/tree/main by @EBazarov Apologies in advance if it is unrelated and will remove the comment/create one in the right place.
gi0baro:
Given everything we stated in the upper replies, I suspect there's no much to do about this. Given the number of frameworks and servers tested, it just seems the Python interpreter is the culprit here, as even if the heap memory gets released, the RSS remains high.
gyula-lakatos commented on Jul 2, 2024
This should be highlighted as I (and many others) could've avoided a lot of headache had I not upgraded my applications to the latest Python (or used a different language altogether).
I can absolutely relate to this because we ended up rewriting the whole app in Rust. One of the main reasons was this bug (and async libs being quite buggy in general). It took me days to debug what's going on and finally landed in here. fifdee commented on Jul 3, 2024 In my case MALLOC_TRIM_THRESHOLD_=131072 seems to work. Those vertical lines are project updates. I'm using Python 3.11 and Django 5.0.
I've not seen memory consumption higher than 450 MB, but I got rid of media passing through Django server, just JSON requests and responses.
 Reacted by Gaurav Kumbhat, Faolain and Oleg A. nikumar1206 commented on Sep 29, 2024 Are there any updates on this issue? Did anyone open up a bug report with the glibc maintainers?
This issue has made trying to run a long-running Python server nearly impossible. stalkerg commented on Oct 7, 2024 geraldog commented on Oct 7, 2024 This is helpful @stalkerg assuming the glibc maintainers actually get to reading the thread they will eventually come up with something, hopefully not a won't-fix 1st1 commented on Oct 8, 2024 Reacted by Marcelo Trylesinski and Suraj CodeToFreedom commented on Feb 21, 2025 Hello,
we have the same problem. Did anyone find a solution or workaround in the meantime? Many thanks for your efforts. geraldog commented on Feb 22, 2025
Hello, we have the same problem. Did anyone find a solution or workaround in the meantime? Many thanks for your efforts.
Ditching glibc entirely seems to be a solution after all... CodeToFreedom commented on Feb 22, 2025 @geraldog Many thanks for your reply.
It looks like a great workaround but to be fully honest I am not familiar enough with the low level package dependencies to completely understand the consequences.
I assume most python environments (docker containers like f.e. python-3.11-slim etc) and many python packages depend on c support.
How exactly do you recommend to replace glibc then? is there a compatible package as drop-in replacement that you can recommend? Or do you use other base images to create the runtime environments?
Again many thanks for your help. I really appreciate it a lot. geraldog commented on Feb 22, 2025
How exactly do you recommend to replace glibc then? is there a compatible package as drop-in replacement that you can recommend? Or do you use other base images to create the runtime environments?
Try an Alpine Linux image. People in this thread have had success in working around this bug with it. It's built around musl libc instead of GNU libc. Eclips4 commented on Mar 11, 2025 Hi!
@geraldog could you please summarize what's happening there and if we should close this issue if the problem is not on CPython's end? Thank you in advance. geraldog commented on Mar 11, 2025 Hi @Eclips4
This is definitely not a CPython issue and I believe the issue should therefore be closed.
The observed "memory leak" is actually a glibc quirk or optimization as discovered by @stalkerg where memory allocated is never reclaimed with a free(), keeping the previously allocated memory in a pool for later re-use.
Some users like me have reported success with setting the env var MALLOC_TRIM_THRESHOLD_ to a low value as a workaround, while other users proved the issue is not reproducible with musl libc based system like Alpine. Reacted by Oleg A., Yury Zhuravlev and Jamie Phan Eclips4 commented on Mar 12, 2025 Thank you for the answer, @geraldog!
I'm closing this as not planned, since the problem is not at CPython's end.
Please ping me and I'll reopen this issue(or maybe it would be better to create a new one, because this one has had quite big discussion..) if you find something on CPython's side related to this issue.
Thank you all for spending time finding the root cause!
MetadataMetadataStatusShow more project fields
DevelopmentNo branches or pull requestsIssue actions
|
