GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-138122: Add differential flame graph by ivonastojanovic · Pull Request #145785 · python/cpython

Conversation

@ivonastojanovic

@ivonastojanovic ivonastojanovic commented Mar 10, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

Differential flame graphs compare two profiling runs and highlight where performance has changed. This makes it easier to detect regressions introduced by code changes and to verify that optimizations have the intended effect.

The visualization renders the current profile with frame widths representing current time consumption. Color is then applied to show the difference relative to the baseline profile: red gradients indicate regressions, while blue gradients indicate improvements.

Some call paths may disappear entirely between profiles. These are referred to as elided stacks and occur when optimizations remove code paths or when certain branches stop executing. When elided stacks are present, an "Elided" toggle is displayed, allowing the user to switch between the main differential view and a view showing only the removed paths.

In the diff view, slow_function improved (blue) and fast_function regressed (red) compared to the baseline. medium_function was removed (shown in the elided stacks toggle) and new_function was added (purple).

Differential view

image

Elided view

image

CC: @pablogsal @lkollar

  • Issue: Implement PEP 799 – A dedicated profiling package for organizing Python profiling tools #138122

📚 Documentation preview 📚: https://cpython-previews--145785.org.readthedocs.build/

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@ivonastojanovic
Add differential flame graph
65528db
Differential flame graphs compare two profiling runs and highlight where
performance has changed. This makes it easier to detect regressions
introduced by code changes and to verify that optimizations have the
intended effect.

The visualization renders the current profile with frame widths
representing current time consumption. Color is then applied to show the
difference relative to the baseline profile: red gradients indicate
regressions, while blue gradients indicate improvements.

Some call paths may disappear entirely between profiles. These are
referred to as elided stacks and occur when optimizations remove code
paths or when certain branches stop executing. When elided stacks are
present, an "Elided" toggle is displayed, allowing the user to switch
between the main differential view and a view showing only the removed
paths.
@ivonastojanovic
ivonastojanovic requested a review from pablogsal as a code owner March 10, 2026 19:46
@bedevere-app bedevere-app Bot added the awaiting review label Mar 10, 2026
@ivonastojanovic ivonastojanovic changed the title Add differential flame graph gh-138122: Add differential flame graph Mar 10, 2026
@bedevere-app bedevere-app Bot mentioned this pull request Mar 10, 2026
Implement PEP 799 – A dedicated profiling package for organizing Python profiling tools #138122
Closed
@ivonastojanovic

ivonastojanovic commented Mar 10, 2026

Copy link
Copy Markdown
Contributor Author

I’m a bit stuck on what colors we should use for new vs deleted functions.

Right now:

  • New functions (not in baseline, present in current) are purple.
  • Deleted functions (shown in elided view, were in baseline, gone in current) are deep red, mostly so they stand out and because red = “removed”.

But I’m not sure if we should even treat them differently from other functions visually.

From a perf perspective it’s kind of confusing:

  • New functions are technically a regression (the code path now exists and adds cost).
  • Deleted ones are technically an improvement (code path removed).

So I’m not sure what the right visual semantics are here. Curious what you think 🙂

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@ivonastojanovic
Add news
2d97700
@ivonastojanovic ivonastojanovic mentioned this pull request Mar 16, 2026
Improve Tachyon UX #142927
Open
11 tasks done
pablogsal
pablogsal reviewed Mar 25, 2026
View reviewed changes
Comment thread Lib/profiling/sampling/_flamegraph_assets/flamegraph.js Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Lib/profiling/sampling/_flamegraph_assets/flamegraph.js Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Lib/test/test_profiling/test_sampling_profiler/test_collectors.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Lib/test/test_profiling/test_sampling_profiler/mocks.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Lib/profiling/sampling/stack_collector.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Lib/profiling/sampling/_flamegraph_assets/flamegraph.css Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Lib/profiling/sampling/_flamegraph_assets/flamegraph.js Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Lib/profiling/sampling/_flamegraph_assets/flamegraph.css Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Lib/test/test_profiling/test_sampling_profiler/test_collectors.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Lib/test/test_profiling/test_sampling_profiler/test_collectors.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

@pablogsal

pablogsal commented Mar 25, 2026

Copy link
Copy Markdown
Member

I’m a bit stuck on what colors we should use for new vs deleted functions.

Right now:

* New functions (not in baseline, present in current) are purple.

* Deleted functions (shown in elided view, were in baseline, gone in current) are deep red, mostly so they stand out and because red = “removed”.

But I’m not sure if we should even treat them differently from other functions visually.

From a perf perspective it’s kind of confusing:

* New functions are technically a regression (the code path now exists and adds cost).

* Deleted ones are technically an improvement (code path removed).

So I’m not sure what the right visual semantics are here. Curious what you think

Perhaps use purple for both new and elided functions, with different shades to distinguish them:

Bright/saturated purple for new functions (present in current, not in baseline): the current color, kept as-is and then muted/desaturated purple for elided functions?

This unifies the two "out-of-band" categories under a single visual language that reads as "no direct comparison is available", keeping them distinct from the red/blue performance axis entirely. The legend becomes simpler too: instead of explaining four special cases, you just say "purple = this function has no counterpart in the other profile."

ivonastojanovic reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

ivonastojanovic added 7 commits March 26, 2026 19:52
@ivonastojanovic
Fix colors
6e642e4
Use four gradients consistently for both regressions and
improvements. Also avoid calling
getComputedStyle(document.documentElement) on every frame, cache
the values like we do for the heatmap, since doing this per frame
can be expensive for large profiles.
@ivonastojanovic
Show diff in ms
37289cb
Since baseline and current self time are shown in ms, the diff
should also be displayed in ms instead of samples.
@ivonastojanovic
Refactor active flamegraph data retrieval
edaac9f
We were determining the selected data and applying
filtering/processing inside each toggle handler, which led to
inconsistencies (e.g. thread filtering) and missed cases. This
approach also won’t scale well as we add more toggles. Instead,
introduce a single centralized function that returns the active
flamegraph data and use it consistently for all updates and
processing.
@ivonastojanovic
Improve tests
c2203d6
Clean up tests by extracting repeated logic (resolving function
names and finding child nodes by name) into helper functions. Also
add a test that doesn’t mock BinaryCollector to cover the full
round trip.
@ivonastojanovic
Change removed functions color
afe2f4e
Use purple (with gradients) for both removed and new functions to
unify these “out-of-band” cases under a single visual language,
meaning no direct comparison is available. This keeps them clearly
separate from the red/blue performance axis and simplifies the
legend: “purple = this function has no counterpart in the other
profile.”
@ivonastojanovic
Fix opcode merging in inverted flamegraph view
c0d937f
                                                                                                                                                           Opcodes from multiple call paths were silently dropped, only the
first path's opcodes were kept. Now they're summed correctly when
nodes merge.
@ivonastojanovic
fixup! Improve tests
080d2c8
@ivonastojanovic
ivonastojanovic requested a review from pablogsal March 29, 2026 15:03
@ivonastojanovic
Merge branch 'main' into differential_flamegraph
440d333
picnixz
picnixz reviewed Mar 29, 2026
View reviewed changes
Comment thread Lib/profiling/sampling/_flamegraph_assets/flamegraph.js
}

// Neutral zone: small percentage change
if (Math.abs(diff_pct) < 15) {

@picnixz picnixz Mar 29, 2026

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This may be a stupid question but do you plan to have the sections configurable? I can see some worth where people are only interested in improvements above a certain threshold and the notion of what is a deep / medium / light / negligible improvement is quite subjective.

For instance, if I really needed to estimate the cost of something, vs readability for instance, I would be happy to only catch 3x improvements for instance if the change is complex.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@pablogsal pablogsal Mar 29, 2026

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good question! Let's keep it as-is for this PR and explore making them configurable separately. An interactive slider in the HTML toolbar would probably be the most useful approach since you'd want to experiment without re-running the profiler.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

picnixz reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

@ivonastojanovic ivonastojanovic Mar 29, 2026

Copy link
Copy Markdown
Contributor Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the idea as well, I can experiment with it in a separate PR

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@picnixz picnixz Mar 29, 2026

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes that's what I had in mind. I would suggest a slide bar with multiple markers that you can move left/right to define the regions of what improvements are. I found https://brechtdr.github.io/enhanced-range-slider-poc/ where it shows a rangegroup in Vite but since it's not a standard element nor something that is implemented natively, we'll need to create the pure HTML/CSS alternative. It would look like:

image

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Lib/profiling/sampling/cli.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Lib/profiling/sampling/stack_collector.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

@pablogsal

pablogsal commented Mar 29, 2026

Copy link
Copy Markdown
Member

@ivonastojanovic Great job. I have pushed a small commit fixing some small stuff:

  • Elided node value overwrite that dropped self-samples
  • Baseline scaling now accounts for sample interval differences
  • Empty current profiles produce differential metadata instead of silently dropping baseline
  • Source line collection suppressed for elided nodes (files may not exist)
  • Dark theme CSS overrides for differential colors
  • Docs corrected: elided stacks are purple, not "deep red"
  • Removed redundant baseline_strings key
  • Early path validation in DiffFlamegraphCollector.init
  • CLI cleanup: proper set_defaults, direct attribute access
  • diff_pct no longer misleadingly shows -100% on elided internal nodes with zero self-time
ivonastojanovic reacted with heart emoji
All reactions
  • ❤️ 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@pablogsal @picnixz
Update Lib/profiling/sampling/stack_collector.py
f4928b9
Co-authored-by: Bénédikt Tran <[email protected]>
picnixz
picnixz reviewed Mar 29, 2026
View reviewed changes
Comment thread Lib/profiling/sampling/stack_collector.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

picnixz and others added 3 commits March 29, 2026 23:21
@picnixz
Update Lib/profiling/sampling/stack_collector.py
5556396
@pablogsal
Random fixes
d515f73
@pablogsal
Small fix to cli
f753071
@pablogsal
pablogsal merged commit f4d3c61 into python:main Mar 30, 2026
48 checks passed

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-app bedevere-app Bot removed the awaiting review label Mar 30, 2026
@pablogsal

pablogsal commented Mar 30, 2026

Copy link
Copy Markdown
Member

Amazing job @ivonastojanovic 🖤

hugovk reacted with hooray emoji hugovk, ivonastojanovic, and savannahostrowski reacted with rocket emoji
All reactions
  • 🎉 1 reaction
  • 🚀 3 reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

maurycy added a commit to maurycy/cpython that referenced this pull request Mar 30, 2026
@maurycy
Merge remote-tracking branch 'upstream/main' into tachyon-ndjson-kole…
6a0ea81
…ktor

* upstream/main: (119 commits)
  pythongh-144270: Make SubElement parent and tag positional-only (pythonGH-144845)
  pythongh-146558: JIT optimize dict access for objects with known hash (python#146559)
  pythongh-139922: always run MSVC 64-bit tail-calling CI (pythonGH-146570)
  pythongh-126835: Fix _PY_IS_SMALL_INT() macro (python#146631)
  pythongh-146587: fix type slot assignment incase of multiple slots for same name (python#146593)
  pythongh-138122: Add differential flame graph (python#145785)
  pythongh-146416: Emscripten: Improve standard stream handling in node_entry.mjs (python#146417)
  pythongh-146444: Don't package as part of iOS 'build hosts' target (python#146628)
  pythongh-138850: Add --disable-epoll to configure (pythonGH-145768)
  pythongh-146444: Make Platforms/Apple/ compatible with Python 3.9 (python#146624)
  pythongh-138577: Fix keyboard shortcuts in getpass with echo_char (python#141597)
  pythongh-146556: Fix infinite loop in annotationlib.get_annotations() on circular __wrapped__ (python#146557)
  pythongh-146579: _zstd: Fix decompression options dict error message (python#146577)
  pythongh-146083: Upgrade bundled Expat to 2.7.5 (python#146085)
  pythongh-146080: fix a crash in SNI callbacks when the SSL object is gone (python#146573)
  pythongh-146090: fix memory management of internal `sqlite3` callback contexts (python#146569)
  pythongh-145876: Do not mask KeyErrors raised during dictionary unpacking in call (pythonGH-146472)
  pythongh-146004: fix test_args_from_interpreter_flags on windows (python#146580)
  pythongh-139003: Use frozenset for module level attributes in _pyrepl.utils (python#139004)
  pythonGH-146527: Add more data to GC statistics and add it to PyDebugOffsets (python#146532)
  ...
@pablogsal pablogsal mentioned this pull request Apr 4, 2026
gh-130273: Fix traceback color output with unicode characters #142529
Merged
clin1234 pushed a commit to clin1234/cpython that referenced this pull request Apr 16, 2026
@ivonastojanovic @pablogsal @clin1234
pythongh-138122: Add differential flame graph (python#145785)
d33faa0
Differential flame graphs compare two profiling runs and highlight where
performance has changed. This makes it easier to detect regressions
introduced by code changes and to verify that optimizations have the
intended effect.

The visualization renders the current profile with frame widths
representing current time consumption. Color is then applied to show the
difference relative to the baseline profile: red gradients indicate
regressions, while blue gradients indicate improvements.

Some call paths may disappear entirely between profiles. These are
referred to as elided stacks and occur when optimizations remove code
paths or when certain branches stop executing. When elided stacks are
present, an "Elided" toggle is displayed, allowing the user to switch
between the main differential view and a view showing only the removed
paths.

Co-authored-by: Pablo Galindo Salgado <[email protected]>
ljfp pushed a commit to ljfp/cpython that referenced this pull request Apr 25, 2026
@ivonastojanovic @pablogsal @ljfp
pythongh-138122: Add differential flame graph (python#145785)
5e9205b
Differential flame graphs compare two profiling runs and highlight where
performance has changed. This makes it easier to detect regressions
introduced by code changes and to verify that optimizations have the
intended effect.

The visualization renders the current profile with frame widths
representing current time consumption. Color is then applied to show the
difference relative to the baseline profile: red gradients indicate
regressions, while blue gradients indicate improvements.

Some call paths may disappear entirely between profiles. These are
referred to as elided stacks and occur when optimizations remove code
paths or when certain branches stop executing. When elided stacks are
present, an "Elided" toggle is displayed, allowing the user to switch
between the main differential view and a view showing only the removed
paths.

Co-authored-by: Pablo Galindo Salgado <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@picnixz picnixz picnixz left review comments
@pablogsal pablogsal Awaiting requested review from pablogsal pablogsal is a code owner

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

3 participants

@ivonastojanovic @pablogsal @picnixz
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.