GeezerVison is showing github.com — perf: Source context re-reads the whole file via loader.get_source() for every frame · Issue #8089 · getsentry/sentry-python
perf: Source context re-reads the whole file via loader.get_source() for every frame #8089DescriptionProblem Statement
When include_source_context=True (the default), get_lines_from_file() calls loader.get_source(module) and then .splitlines() for every frame of every event, whenever the frame's module has a __loader__ with get_source (i.e. any normally imported module). SourceFileLoader.get_source() re-reads and decodes the whole file each time; linecache, which is the fallback a few lines below, keeps it in memory.
All of this runs synchronously in the thread that raises/logs the error, before the event reaches the transport worker. In asyncio apps that log many handled exceptions in a burst, this cost is paid on the event loop (related: #2824). sample_rate / before_send don't avoid it, since the frames are built before they run (#4365).
Reproducer (only sentry_sdk + stdlib, events go to a no-op transport):
repro_source_context.py
"""Source context re-reads the whole source file through loader.get_source() for every frame of every event."""
import importlib.machinery, linecache, logging, os, sys, tempfile, time
import sentry_sdk
import sentry_sdk.utils
from sentry_sdk.transport import Transport
SENT = []
class NullTransport(Transport):
def capture_envelope(self, envelope):
SENT.append(envelope.get_event())
sentry_sdk.init(dsn="http://[email protected]/1", transport=NullTransport())
# A ~1,500-line module; the exception passes through 10 frames in it.
code = "def f0():\n raise TimeoutError('boom')\n"
code += "".join(f"def f{i}():\n return f{i - 1}()\n" for i in range(1, 10))
code += "".join(f"# filler line {j}\n" for j in range(1500))
tmp = tempfile.mkdtemp()
with open(os.path.join(tmp, "bigmod.py"), "w") as f:
f.write(code)
sys.path.insert(0, tmp)
import bigmod # noqa: E402
calls = {"get_source": 0}
_get_source = importlib.machinery.SourceFileLoader.get_source
def counting_get_source(self, name):
calls["get_source"] += 1
return _get_source(self, name)
importlib.machinery.SourceFileLoader.get_source = counting_get_source
log = logging.getLogger("repro")
log.addHandler(logging.NullHandler())
log.propagate = False
def emit():
try:
bigmod.f9()
except TimeoutError:
log.exception("something failed")
def bench(n=1000):
SENT.clear()
calls["get_source"] = 0
t = time.perf_counter()
for _ in range(n):
emit()
return 1000 * (time.perf_counter() - t) / n, calls["get_source"] / n, SENT[-1]
_orig = sentry_sdk.utils.get_lines_from_file
def linecache_first(filename, lineno, max_length=None, loader=None, module=None):
# Prefer the cached linecache copy; only fall back to the loader (zipimport,
# frozen apps, ...) when linecache has nothing for this file.
if linecache.getlines(filename):
loader = None
return _orig(filename, lineno, max_length, loader, module)
emit() # warm up
ms_a, gs_a, ev_a = bench()
sentry_sdk.utils.get_lines_from_file = linecache_first
ms_b, gs_b, ev_b = bench()
print(f"sentry-sdk {sentry_sdk.VERSION}, Python {sys.version.split()[0]}")
print(f"stock: {ms_a:.3f} ms/event, loader.get_source() calls/event: {gs_a:.0f}")
print(f"linecache first: {ms_b:.3f} ms/event, loader.get_source() calls/event: {gs_b:.0f}")
print("identical exception payload:", ev_a["exception"] == ev_b["exception"])
Results with sentry-sdk==2.72.0 (same code on master), Linux x86_64:
Python 3.14.8
stock: 3.693 ms/event, loader.get_source() calls/event: 11
linecache first: 2.166 ms/event, loader.get_source() calls/event: 0
identical exception payload: True
Python 3.13.16
stock: 2.937 ms/event, loader.get_source() calls/event: 11
linecache first: 2.466 ms/event, loader.get_source() calls/event: 0
identical exception payload: True
Solution Brainstorm
Read source from linecache first and only fall back to the loader when linecache has nothing for that file (zipimport, PyInstaller/frozen apps, see #812). Alternatively linecache.getlines(filename, frame.f_globals) gives the loader fallback and caching in one call. The output payload is unchanged (checked in the script). Activitylinear-code commented on Oct 20, 2026 Shellghostt commented on Nov 12, 2026 Looked at get_lines_from_file a bit. It tries loader.get_source(module) first and only falls back to linecache when there is no loader, so for a normal file-backed module every frame re-reads and decodes the whole file, and the linecache path is never reached.
I would flip that order: read linecache.getlines(filename) first and keep loader.get_source(module) as the fallback for modules whose source is not a file, such as zipimport or a custom import hook. After the first frame of a given file, every later frame becomes a dict lookup. The strip("\r\n") and max_length handling stays as it is.
For a test I would check that a multi-frame exception from one module consults the loader once and that the captured frame context is unchanged. The timing number from your reproducer belongs in the PR description rather than as an assertion, since it would be flaky in CI.
Does that sound right? If so I will open a draft PR.
MetadataMetadataStatusWaiting for: Product Owner Show more project fields
DevelopmentNo branches or pull requestsIssue actions
|