GeezerVison is showing github.com — Observed memory leak in ssl library: Python 3.14 GC issue · Issue #142516 · python/cpython
Observed memory leak in ssl library: Python 3.14 GC issue #142516DescriptionBug report
Bug description:
I already opened a bug on urllib3 for this issue, it looks that may application using MSAL for Python library (which relies on requests , which relies on urllib3, which relies on ssl) is leaking memory when being upgraded from Python 3.13 to Python 3.14 (I tested 3.14.{0, 1, 2})
The details are explained in the urllib3 ticket including outputs from memray and it points to ssl package (precisely ssl.SSLContext.load_verify_locations), see: urllib3/urllib3#3738
CPython versions tested on:
3.14
Operating systems tested on:
Linux
Linked PRs
Reacted by Sebastian, HenriBlacksmith, Simon Gurcke, Bartlomiej Wiater, curekoshimizu, Sergei K., Masaya, Rikuto, Yuki Furukawa, Reona Sato and 8 more Reacted by Alex Poone, Jalin Wang and will-wc-chen-appier Activityvstinner commented on Dec 10, 2025 zweizeichen commented on Dec 11, 2025 No reproducer but if it helps: very similar situation here when connecting to valkey with a custom CA cert.
 zweizeichen commented on Dec 12, 2025 I did some more testing to come up with a minimal repro: seems like rss just keeps growing while the heap stays somewhat constant. I was unable to reproduce it using a simple sync case locally and on the cluster and this started to smell an awful lot like #109534 which might be async-related. Like @HenriBlacksmith's case we're also seeing this in an ASGI app on k8s using the debian trixie base image since the 3.14 release.
I'll attempt to redeploy this glibc-less and report back.
Update: the alleged leak is still there even in an alpine container zweizeichen commented on Dec 15, 2025 HenriBlacksmith commented on Dec 15, 2025 I applied a fix that improved the memory situation on my other FastAPI app by reducing the calls generated by MSAL.
I applied a few dependency upgrades on top including a bump of the Python image from 3.14.0-slim to 3.14.2-slim and it caused the memory leak to be worse.
Though I also need to collect detailed figures about this one as urllib3 version is different in this app because of some version constraints on a library.
I will try to get more info from memray on this one. HenriBlacksmith commented on Dec 16, 2025 I was able to export new memray profiles from my other application.
This time the leak is coming from httpx which also relies on ssl package under the hood:
Sorry for the cuts on the capture, I am trying to "anonymize" things zweizeichen commented on Dec 16, 2025 Thank you! I'm not 100% sure how to interpret these memray outputs to be honest, because if I understand this correctly this shows us where allocations do happen but not necessarily where the leak is (it may be correlated though). This is useful for finding out 'which objects eat up my memory?'. Technically speaking memray will only show us the allocations/leaks the python allocator knows about, but the leak kind of escapes the realm of the allocator by definition. Another layer which might be adding to this is that this might actually be fragmentation instead of leaks. I hope I ruled this out more or less by checking it's not glibc-related by also observing this behaviour on an alpine container, feel free to double-check.
I'm going to try to find some time to give the minimal (hopefully stdlib only) repro another go and explore analyzing a full heap dump. HenriBlacksmith commented on Dec 16, 2025 I am trying to use valgrind instead of memray to get more details (but I am a bit out of my comfort zone to be honest), it does not report much until k8s restarts the container.
The command looks like:
valgrind --tool=memcheck --log-file=/tmp/valgrind.log --trace-children=yes --leak-check=yes --verbose
python -m uvicorn app.main:shell --host 0.0.0.0 --log-config logging.yamlBut nothing really comes out of it HenriBlacksmith commented on Dec 16, 2025
Another layer which might be adding to this is that this might actually be fragmentation instead of leaks. I hope I ruled this out more or less by checking it's not glibc-related by also observing this behaviour on an alpine container, feel free to double-check.
I just built an Alpine container and ran it and the behavior is similar (based on python:3.14.2-alpine3.23)
I am running out of ideas, if anyone has guidance to help with investigating this, I would be really happy to test those, if I have extra time, I will try to build a MRO but I am not sure how it can be done HenriBlacksmith commented on Dec 16, 2025 Good news, I might have a MRE, it is using httpx but that's the best I was able to do, I will share the whole code in a repo, including commands to export the graphs.
Here is the code:
import httpx
import asyncio
async def _main() -> None:
while(1):
async with httpx.AsyncClient() as client:
await client.request("GET", "https://example.com/")
await asyncio.sleep(1)
if __name__ == "__main__":
asyncio.run(_main())FROM astral/uv:0.9.17-python3.14-bookworm AS builder
ENV UV_COMPILE_BYTECODE=1 UV_LINK_MODE=copy UV_PYTHON_DOWNLOADS=0
# Create and set the working directory
WORKDIR /mro-app
# Install runtime dependencies
RUN --mount=type=cache,target=/root/.cache/uv \
--mount=type=bind,source=uv.lock,target=uv.lock \
--mount=type=bind,source=pyproject.toml,target=pyproject.toml \
uv sync --frozen --no-install-project --no-dev
# Copy the application code (excluding files in .dockerignore)
COPY . .
RUN --mount=type=cache,target=/root/.cache/uv \
uv sync --frozen --no-dev
# Use an official lightweight Python image with slim variant
FROM python:3.14.2-slim
# Set environment variables
# See: https://docs.python.org/3/using/cmdline.html#envvar-PYTHONUNBUFFERED
ENV PYTHONUNBUFFERED=1
# Create and set the working directory
WORKDIR /mro-app
# Expose the port the app runs on
EXPOSE 8000
# Copy the application from the builder
COPY --from=builder /mro-app /mro-app
# Place executables in the environment at the front of the path
ENV PATH="/mro-app/.venv/bin:$PATH"
# Define the default command to run when the container starts
CMD ["memray", "run", "--native", "--output", "/tmp/capture.bin", "main.py"][project]
name = "mro"
version = "0.1.0"
description = "Add your description here"
readme = "README.md"
requires-python = ">=3.14"
dependencies = [
"httpx>=0.28.1",
"memray>=1.19.1",
"urllib3>=2.6.2",
]etc. will build a repo asap with all the other files
 Reacted by Sebastian and Max Chernoff HenriBlacksmith commented on Dec 16, 2025 Reacted by Sebastian, Faolain and Max Chernoff HenriBlacksmith commented on Dec 16, 2025 And I was finally able to get the native symbols:
I will try to compare with older Python versions to see if the sensitivity to Python version holds with the MRE.
I was able to reproduce on my Mac in a non-Docker environment. bedevere-app commented on Apr 17, 2026 vstinner commented on Apr 30, 2026 Reacted by Neil Schemenauer, HenriBlacksmith, Sergey Miryanov, curekoshimizu, Jalin Wang and Yuki Furukawa Reacted by Alex Poone hugovk commented on May 5, 2026 Reacted by Alex Poone, Gastón Avila, Nathan Burns, Haowei Wang, Jalin Wang, Yuki Furukawa and Vladyslav Burylov HenriBlacksmith commented on May 19, 2026 Reacted by Hugo van Kemenade, Victor Stinner, Sergey Miryanov and Alessio Fachechi HenriBlacksmith commented on Jun 2, 2026
This has been reverted and released in Python 3.14.5 release candidate, please test.
https://discuss.python.org/t/python-3-14-5-release-candidate/107185/1
We plan a 3.14.5 final for Friday.
I have been testing this on two APIs in our dev environment and it looks memory is stable. Will provide more feedback once it will be in prod.
It has been running in prod for a week with similar memory profiles as for 3.13.x.
The issue is resolved from my point of view Reacted by Hugo van Kemenade, Sergey Miryanov, Cristian M, Gregory P. Smith and 93578237 miketheman commented on Jun 2, 2026 I can also confirm that the memory footprint is now "stable" since 3.14.5 - and a side benefit has been that the "floor" is lower from 3.13.x
 Reacted by Hugo van Kemenade, Sergey Miryanov, Cody Maloney, HenriBlacksmith, Cristian M, Gregory P. Smith, Felix Fanghaenel, 93578237, Victor Stinner and Kuthair Habboush
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|
