GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-152721: Fix quadratic RLE replay time in the profiling binary reader by tonghuaroot · Pull Request #152722 · python/cpython

Conversation

@tonghuaroot

@tonghuaroot tonghuaroot commented Jul 1, 2026 •
edited by pablogsal
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

Repeated samples with alternating thread status caused quadratic allocation and trimming in the binary reader. Build timestamp lists incrementally and split long runs into batches of at most 8,192 samples.

Tests cover alternating statuses and a single 8,193-sample record, including exact batch sizes and timestamp order. The batch-limit regression fails without the cap. The profiling test suite passes.

Closes #152721.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@tonghuaroot
pythongh-152721: Fix quadratic RLE replay time in the profiling binar…
1dac106
…y reader
@tonghuaroot
tonghuaroot requested a review from pablogsal as a code owner July 1, 2026 02:56
@bedevere-app bedevere-app Bot mentioned this pull request Jul 1, 2026
_remote_debugging: quadratic replay time for RLE records with alternating status #152721
Closed
@bedevere-app bedevere-app Bot added the awaiting review label Jul 1, 2026
pablogsal
pablogsal reviewed Jul 1, 2026
View reviewed changes

@pablogsal pablogsal left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Modules/_remote_debugging/binary_io_reader.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Lib/test/test_profiling/test_sampling_profiler/test_binary_format.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

@tonghuaroot
pythongh-152721: Clarify the batch-list comment
beb5980
Reword per review: the list is built with append per element, not
pre-sized; note the old alloc(count - i) + trim per batch was O(count^2).
maurycy
maurycy reviewed Jul 2, 2026
View reviewed changes
Comment thread Modules/_remote_debugging/binary_io_reader.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

tonghuaroot added 2 commits July 2, 2026 22:23
@tonghuaroot
Cap RLE batch size to bound the per-batch timestamp list
3e71061
@tonghuaroot
Move MAX_RLE_BATCH_SAMPLES comment to its own line (keep under 79 cols)
6594318
pablogsal
pablogsal reviewed Aug 16, 2026
View reviewed changes

@pablogsal pablogsal left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some minor nits

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Modules/_remote_debugging/binary_io_reader.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Modules/_remote_debugging/binary_io_reader.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Modules/_remote_debugging/binary_io_reader.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Modules/_remote_debugging/binary_io_reader.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

@tonghuaroot
Drop the now-redundant emit_batch trim and fix the review nits
70a2e41
The batch list is built to its exact size, so the PyList_SetSlice trim was a no-op; call emit_sample directly. Also point the cap comment at the issue (pythongh-151378) and drop the stale over-long comment.
@tonghuaroot

tonghuaroot commented Aug 18, 2026

Copy link
Copy Markdown
Contributor Author

Done in 70a2e41: call emit_sample directly (the PyList_SetSlice trim was a no-op), dropped the stale comment, and pointed the cap comment at gh-151378. I'll follow up with a test for the >8192 MAX_RLE_BATCH_SAMPLES split.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

pablogsal added 2 commits October 5, 2026 14:57
@pablogsal
Merge remote-tracking branch 'upstream/main' into tachyon-land-152722
8836caa
@pablogsal
pythongh-152721: Test replay batches beyond the RLE limit
e3327f5
@pablogsal pablogsal added the needs backport to 3.15 bugs and security fixes label Oct 5, 2026
@pablogsal
pablogsal enabled auto-merge (squash) October 5, 2026 14:06
pablogsal
pablogsal approved these changes Oct 5, 2026
View reviewed changes
@bedevere-app bedevere-app Bot added awaiting merge and removed awaiting review labels Oct 5, 2026
@pablogsal pablogsal removed the needs backport to 3.15 bugs and security fixes label Oct 5, 2026
@pablogsal pablogsal mentioned this pull request Oct 5, 2026
[3.15] gh-152721: Fix quadratic RLE replay time in the profiling binary reader (GH-152722) #158850
Merged
@bedevere-app

bedevere-app Bot commented Oct 5, 2026

Copy link
Copy Markdown

GH-158850 is a backport of this pull request to the 3.15 branch.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@pablogsal
pablogsal merged commit e4a2473 into python:main Oct 5, 2026
63 checks passed

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-app bedevere-app Bot removed the awaiting merge label Oct 5, 2026
pablogsal added a commit that referenced this pull request Oct 5, 2026
@pablogsal @tonghuaroot
[3.15] gh-152721: Fix quadratic RLE replay time in the profiling bina…
ebaca2a
…ry reader (GH-152722) (#158850)

Backport of GH-152722.

Co-authored-by: tonghuaroot <[email protected]>
pablogsal added a commit to pablogsal/cpython that referenced this pull request Oct 5, 2026
@pablogsal
Revert "[3.15] pythongh-152721: Fix quadratic RLE replay time in the …
80eb57e
…profiling binary reader (pythonGH-152722) (python#158850)"

This reverts commit ebaca2a.
pablogsal added a commit that referenced this pull request Oct 11, 2026
@pablogsal @miss-islington
@maurycy @claude @vstinner @StanFromIreland @tonghuaroot @hetaozdh @edvilme @deadlovelll
[3.15] Restore deferred backports for 3.15.1, including GH-158874 (#1…
4ceb3fa
…58882)

* [3.15] gh-153364: Make frame, coroutine, and task-waiter chain walks iterative and bounded (GH-153365) (#158813)

gh-153364: Make frame, coroutine, and task-waiter chain walks iterative and bounded (GH-153365)

* let me declare single limit

* use our new limit in process_frame_chain()

* add it in parse_async_frame_chain()

* parse_coro_chain()

* NEWS

* async in the message?

* test

* no race

* process_task_awaited_by

* process_task_awaited_by limit test

* NEWS

* MAX_TASK_WAITER_CHAIN_DEPTH

* TASK_WAITER_CHAIN_DEPTH in test

* TASK_WAITER_CHAIN_DEPTH 256

* prevent the drift with the comment

* better naming, better style

* MAX_TASK_WAITER_CHAIN_DEPTH comment

* task-waiter iterative bfs walk

* iterative coro-walk

* nicer news

* 1 << 14

* comment

* unused read_Py_ssize_t

* fix tombstones

* simplify

* correct msg

* better test

* news for tombstones

* left-over from when testing buggy version

* redundant new line
(cherry picked from commit e0861c6)

Co-authored-by: Maurycy Pawłowski-Wieroński <[email protected]>
(cherry picked from commit 9e401cf)

* [3.15] gh-155811: Add a seqcount to `gc_stats` to prevent torn reads (GH-155828) (#158829)

* update_seq

* no need for XCHGL, MOVL is enough?

* gh-155811: Retry an inconsistent GC snapshot once

---------
(cherry picked from commit 5fecd44)

Co-authored-by: Pablo Galindo Salgado <[email protected]>
Co-authored-by: Claude Fable 5.1 <[email protected]>
(cherry picked from commit 0281240)

* [3.15] gh-151292: `_remote_debugging`: Do not corrupt the binary file when hitting `OverflowError` (GH-152892) (#158830)

gh-151292: `_remote_debugging`: Do not corrupt the binary file when hitting `OverflowError` (GH-152892)

* the kolektor

* test

* better test

* news

* =Q, move const to the base, not self.running

* gh-151292: Track binary writer finalization state

---------
(cherry picked from commit f839c06)

Co-authored-by: Maurycy Pawłowski-Wieroński <[email protected]>
Co-authored-by: Pablo Galindo Salgado <[email protected]>
(cherry picked from commit 8a7c23f)

* [3.15] gh-158583: Fix uninitialized memory read in bytes.fromhex() (GH-158584) (#158691)

gh-158583: Fix uninitialized memory read in bytes.fromhex() (GH-158584)
(cherry picked from commit 9d22a53)

Co-authored-by: Victor Stinner <[email protected]>
(cherry picked from commit 4f7af46)

* [3.15] gh-154194: Degrade frames in Tachyon instead of failing the sample (GH-154195) (#158831)

* gh-154194: Degrade frames in Tachyon instead of failing the sample (#154195)

* degrade gracefully

* news

* better NEWS wording

* do not raise on MAX_REMOTE_STR_READ

* bye MAX_REMOTE_STR_READ

* fix -m asyncio ps|pstree

* test truncation and linetable sentinel

* simpler

* simpler

* redundant now

* respect #157790 in the news

---------

Co-authored-by: Pablo Galindo Salgado <[email protected]>
(cherry picked from commit 7d25916)

* Preserve the stable ABI when creating fallback frame names

---------

Co-authored-by: Maurycy Pawłowski-Wieroński <[email protected]>
(cherry picked from commit c27f494)

* [3.15] Add MSan to CI (GH-158625) (#158832)

Run a job that the test suite with MSan to the CI (#158625)

* Run the test suite with MSan in CI

* Additional fixes

* Add `_Py_MSAN_UNPOISON_STRING`

* Apply Victor's suggestions

* Apply Victor's suggestions

---------

(cherry picked from commit b93fb19)

Co-authored-by: Stan Ulbrych <[email protected]>
Co-authored-by: Victor Stinner <[email protected]>
(cherry picked from commit 1bc78de)

* [3.15] gh-156810: Write the profiler's collapsed-stack export as UTF-8 (GH-156811) (#156814)

gh-156810: Write the profiler's collapsed-stack export as UTF-8 (GH-156811)
(cherry picked from commit c3706f4)

Co-authored-by: tonghuaroot (童话) <[email protected]>
(cherry picked from commit fce28da)

* [3.15] gh-158552: Wait for Windows threads to suspend before blocking sampling (GH-158802) (#158845)

gh-158552: Wait for Windows threads to suspend before blocking sampling (GH-158802)

* gh-158552: Wait for Windows threads to suspend before blocking sampling

* Use a named Windows thread enumeration status constant
(cherry picked from commit 1643525)

Co-authored-by: Pablo Galindo Salgado <[email protected]>
(cherry picked from commit 31288da)

* [3.15] gh-152721: Fix quadratic RLE replay time in the profiling binary reader (GH-152722) (#158850)

Backport of GH-152722.

Co-authored-by: tonghuaroot <[email protected]>
(cherry picked from commit ebaca2a)

* [3.15] gh-156545: Fix flamegraph export RecursionError on deeply recursive programs (GH-156546) (#158851)

Backport of GH-156546.

Co-authored-by: tonghuaroot (童话) <[email protected]>
(cherry picked from commit 0ac7217)

* [3.15] gh-158540: Add the profiled script's directory to sys.path (GH-158548) (#158844)

gh-158540: Add the profiled script's directory to sys.path (GH-158548)

* gh-158540: Add the profiled script's directory to sys.path

When a script is profiled with ``python -m profiling.sampling run
script.py`` from another directory, the script cannot import modules
placed next to it, because ``_sync_coordinator._execute_script()``
executes it with the working directory (added by
``_setup_environment()`` for the module case) as ``sys.path[0]``
instead of the script's own directory.

Make the script's directory importable in ``_execute_script()``,
matching the behavior of ``python script.py``.

Add a regression test that runs the coordinator on a script importing
a sibling module.

* Update _sync_coordinator.py Comment simplified

* gh-158540: Resolve symlinks when adding the script directory to sys.path

``python script.py`` resolves symlinks when computing ``sys.path[0]``, so
a script reached through a symlink (``link.py -> sub/where.py``) imports
modules from the real script's directory.  Apply ``os.path.realpath()``
before taking the directory name, and make sure the result is placed at
the front of ``sys.path`` even if it was already listed.

Add a regression test for a symlinked script.

---------
(cherry picked from commit 3f02aab)

Co-authored-by: he_tao <[email protected]>
Co-authored-by: Eduardo Villalpando Mello <[email protected]>
(cherry picked from commit f52d831)

* [3.15] gh-153838: Skip non-regular source files in the heatmap exporter (GH-153839) (#158853)

Backport of GH-153839.

Co-authored-by: tonghuaroot <[email protected]>
(cherry picked from commit 48998df)

* [3.15] gh-158539: Fix exception mode missing handlers in generators/coroutines (GH-158581) (#158852)

* [3.15] gh-158539: Fix exception mode missing handlers in generators/coroutines (GH-158581)

Backport of GH-158581.

Co-authored-by: LucasZhou <[email protected]>

* [3.15] gh-158539: Use portable static assertion messages

* [3.15] gh-158539: Keep layout assertions with debug-offset validation

* [3.15] gh-158539: Use the platform guard for in-process inspection tests

---------

Co-authored-by: LucasZhou <[email protected]>
(cherry picked from commit d625ecb)

* [3.15] gh-158522: Fix truncated stack for a task whose coroutine recurses (GH-158526) (#158870)

Co-authored-by: Timofei Ivankov <[email protected]>
(cherry picked from commit a904b39)

* gh-156545, gh-158539: Fix deep flamegraph export on small C stacks and macOS runtime lookup (#158874)

(cherry picked from commit 114de19)

Include the C-stack test helper from main, introduced by
ce5ae29, which the regression test
requires but 3.15 does not yet provide.

---------

Co-authored-by: Miss Islington (bot) <[email protected]>
Co-authored-by: Maurycy Pawłowski-Wieroński <[email protected]>
Co-authored-by: Claude Fable 5.1 <[email protected]>
Co-authored-by: Victor Stinner <[email protected]>
Co-authored-by: Stan Ulbrych <[email protected]>
Co-authored-by: Victor Stinner <[email protected]>
Co-authored-by: tonghuaroot (童话) <[email protected]>
Co-authored-by: he_tao <[email protected]>
Co-authored-by: Eduardo Villalpando Mello <[email protected]>
Co-authored-by: LucasZhou <[email protected]>
Co-authored-by: Timofei Ivankov <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pablogsal pablogsal pablogsal approved these changes
+1 more reviewer
@maurycy maurycy maurycy left review comments
Reviewers whose approvals may not affect merge requirements

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

_remote_debugging: quadratic replay time for RLE records with alternating status

Uh oh!

There was an error while loading. Please reload this page.

3 participants

@tonghuaroot @maurycy @pablogsal
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.