GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — perf: Source context re-reads the whole file via loader.get_source() for every frame · Issue #8089 · getsentry/sentry-python

perf: Source context re-reads the whole file via loader.get_source() for every frame #8089

New issue
Copy link
New issue
Copy link
Open
Open
perf: Source context re-reads the whole file via loader.get_source() for every frame#8089
Copy link
Labels
ImprovementPythonSDKSDKWaiting for: Product Owner

Description

@devin-ai-integration
devin-ai-integration
bot
opened on Oct 20, 2026
Issue body actions

Problem Statement

When include_source_context=True (the default), get_lines_from_file() calls loader.get_source(module) and then .splitlines() for every frame of every event, whenever the frame's module has a __loader__ with get_source (i.e. any normally imported module). SourceFileLoader.get_source() re-reads and decodes the whole file each time; linecache, which is the fallback a few lines below, keeps it in memory.

All of this runs synchronously in the thread that raises/logs the error, before the event reaches the transport worker. In asyncio apps that log many handled exceptions in a burst, this cost is paid on the event loop (related: #2824). sample_rate / before_send don't avoid it, since the frames are built before they run (#4365).

Reproducer (only sentry_sdk + stdlib, events go to a no-op transport):

repro_source_context.py
"""Source context re-reads the whole source file through loader.get_source() for every frame of every event."""
import importlib.machinery, linecache, logging, os, sys, tempfile, time

import sentry_sdk
import sentry_sdk.utils
from sentry_sdk.transport import Transport

SENT = []


class NullTransport(Transport):
    def capture_envelope(self, envelope):
        SENT.append(envelope.get_event())


sentry_sdk.init(dsn="http://[email protected]/1", transport=NullTransport())

# A ~1,500-line module; the exception passes through 10 frames in it.
code = "def f0():\n    raise TimeoutError('boom')\n"
code += "".join(f"def f{i}():\n    return f{i - 1}()\n" for i in range(1, 10))
code += "".join(f"# filler line {j}\n" for j in range(1500))
tmp = tempfile.mkdtemp()
with open(os.path.join(tmp, "bigmod.py"), "w") as f:
    f.write(code)
sys.path.insert(0, tmp)
import bigmod  # noqa: E402

calls = {"get_source": 0}
_get_source = importlib.machinery.SourceFileLoader.get_source


def counting_get_source(self, name):
    calls["get_source"] += 1
    return _get_source(self, name)


importlib.machinery.SourceFileLoader.get_source = counting_get_source

log = logging.getLogger("repro")
log.addHandler(logging.NullHandler())
log.propagate = False


def emit():
    try:
        bigmod.f9()
    except TimeoutError:
        log.exception("something failed")


def bench(n=1000):
    SENT.clear()
    calls["get_source"] = 0
    t = time.perf_counter()
    for _ in range(n):
        emit()
    return 1000 * (time.perf_counter() - t) / n, calls["get_source"] / n, SENT[-1]


_orig = sentry_sdk.utils.get_lines_from_file


def linecache_first(filename, lineno, max_length=None, loader=None, module=None):
    # Prefer the cached linecache copy; only fall back to the loader (zipimport,
    # frozen apps, ...) when linecache has nothing for this file.
    if linecache.getlines(filename):
        loader = None
    return _orig(filename, lineno, max_length, loader, module)


emit()  # warm up
ms_a, gs_a, ev_a = bench()
sentry_sdk.utils.get_lines_from_file = linecache_first
ms_b, gs_b, ev_b = bench()

print(f"sentry-sdk {sentry_sdk.VERSION}, Python {sys.version.split()[0]}")
print(f"stock:            {ms_a:.3f} ms/event, loader.get_source() calls/event: {gs_a:.0f}")
print(f"linecache first:  {ms_b:.3f} ms/event, loader.get_source() calls/event: {gs_b:.0f}")
print("identical exception payload:", ev_a["exception"] == ev_b["exception"])

Results with sentry-sdk==2.72.0 (same code on master), Linux x86_64:

Python 3.14.8
stock:            3.693 ms/event, loader.get_source() calls/event: 11
linecache first:  2.166 ms/event, loader.get_source() calls/event: 0
identical exception payload: True

Python 3.13.16
stock:            2.937 ms/event, loader.get_source() calls/event: 11
linecache first:  2.466 ms/event, loader.get_source() calls/event: 0
identical exception payload: True

Solution Brainstorm

Read source from linecache first and only fall back to the loader when linecache has nothing for that file (zipimport, PyInstaller/frozen apps, see #812). Alternatively linecache.getlines(filename, frame.f_globals) gives the loader fallback and caching in one call. The output payload is unchanged (checked in the script).

Activity

  1. linear-code commented on Oct 20, 2026

    @linear-code
    linear-codeBot
    on Oct 20, 2026 – with Linear Code
    More actions

    PY-3026

  2. linear-code
    added
    Improvement
    PythonSDKSDK
    on Oct 20, 2026
  3. Shellghostt commented on Nov 12, 2026

    @Shellghostt
    Shellghostt
    on Nov 12, 2026
    Contributor
    More actions

    Looked at get_lines_from_file a bit. It tries loader.get_source(module) first and only falls back to linecache when there is no loader, so for a normal file-backed module every frame re-reads and decodes the whole file, and the linecache path is never reached.

    I would flip that order: read linecache.getlines(filename) first and keep loader.get_source(module) as the fallback for modules whose source is not a file, such as zipimport or a custom import hook. After the first frame of a given file, every later frame becomes a dict lookup. The strip("\r\n") and max_length handling stays as it is.

    For a test I would check that a multi-frame exception from one module consults the loader once and that the captured frame context is unchanged. The timing number from your reproducer belongs in the PR description rather than as an assertion, since it would be flaky in CI.

    Does that sound right? If so I will open a draft PR.

  4. getsantry
    added
    Waiting for: Product Owner
    on Nov 12, 2026
  5. getsantry
    added this to GitHub Issues with 👀 3on Nov 12, 2026
  6. getsantry
    moved this to Waiting for: Product Owner in GitHub Issues with 👀 3on Nov 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    ImprovementPythonSDKSDKWaiting for: Product Owner

    Projects

    • GitHub Issues with 👀 3
      Status
      Waiting for: Product Owner
      Show more project fields

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions

    You are visitor 01402503 — GeezerVison does not store the pages you look up.
    Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.