GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — Fix reference-chain nightly flakes and drop their quarantine by jbachorik · Pull Request #899 · DataDog/java-profiler

Conversation

@jbachorik

@jbachorik jbachorik commented Oct 25, 2026 •
edited by atlassian Bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Collaborator

What does this PR do?:
Fixes the intermittently-red reference-chain slow e2e family (PROF-16305) with one product fix plus test-side hardening, and removes the PROF-16305 quarantine added in #888. The branch carries four commits:

  1. 2277cb539 — PROF-16316: keep resolved leak chains across improve-path invalidations (EVICT/REBUILD invalidation kinds, in-band sign-bit encoding, generation-guarded BFS rebuild from the intact frontier; replace-only, no behavior change elsewhere).
  2. 4b0879ed4 — fix reference-chain revisits and leak-candidate admission.
  3. 2908999d6 — make the reference-chain slow e2e tests robust to walk-pacing latency (schedule reshape, front-loaded reference pressure, richer failure diagnostics).
  4. 37fc26255 — remove all four PROF-16305 quarantine lines from ddprof-test/quarantine.txt.

Motivation:
The nightly sanitized run 39893765385 had the reference-chain slow family failing in ~20 of ~32 eligible HotSpot 11+ debug/asan jobs, which is what #888 quarantined. Root causes are now fixed rather than quarantined:

  • Late thread-local variant (PROF-16316): improveChain's deferred invalidation erased cached leak chains at dump time, and the terminal pass zeroed candidate/discovered state so nothing could rebuild them — defeating the restartSearch() keep-across-restart contract. Fixed by 2277cb539.
  • Cache-leak variant: the walk-starvation mode (tags assigned, but the BFS never intercepts a leak-tagged payload in the 131,072-slot table) is addressed by the 4b0879ed4 revisit/admission regime plus the 2908999d6 test-side pacing fixes.
  • gc-root variant on musl-aarch64 JDK 11: validated clean on the full stack (see below), so its quarantine line is dropped.

Additional Notes:
Dependencies — this PR is stacked and should not merge before its dependencies:

  • Depends on Reference-chain cache: rebuild stale chains (improved path, correlation) instead of erasing them #898 (PROF-16316 improveChain invalidation fix): commit 2277cb539 in this branch is that change, carried here so the whole stack validates together. If Reference-chain cache: rebuild stale chains (improved path, correlation) instead of erasing them #898 merges first, this PR's effective diff shrinks to commits 2–4.
  • Depends on Fix reference-chain discovery and leak test prerequisites #889 (revisit/admission fix): commit 4b0879ed4 is its replay on top of Reference-chain cache: rebuild stale chains (improved path, correlation) instead of erasing them #898, for the same reason. Fix reference-chain discovery and leak test prerequisites #889 is currently an open draft (Paul Fournillon).
  • Quarantine flaky reference-chain slow e2e tests (PROF-16305) #888 (quarantine) is already merged into main; commit 4 removes the lines it added, so this PR supersedes the quarantine.

How to test the change?:

  • Native: gtest for the PROF-16316 fix fails before 2277cb539 and passes after (full :ddprof-lib:gtestDebug 189/189).
  • e2e: full reference-chain slow family validated on the complete stack — Temurin 21: 20/20 (+ prebuild pass) incl. a dedicated 10/10 musl-aarch64 JDK 11 container run; jdk11 9/10 with the only residual being Fix reference-chain discovery and leak test prerequisites #889's known cache-leak residual (tracked there); late-TL variant 0 failures post-fix.
  • Negative control: sabotaging the drain path makes the fixed tests fail loudly with the new diagnostics suffix.
  • Quarantine validator: ddprof-test/quarantine.txt validate script passes; regular (non-slow) suite 61/61 unaffected.

For Datadog employees:

  • If this PR touches code that signs or publishes builds or packages, or handles
    credentials of any kind, I've requested a security review (run the dd:platform-security-review
    skill, or file a request via the PSEC review form).
    bewaire also runs automatically on every PR.
  • This PR doesn't touch any of that.
  • JIRA: PROF-16305 (and PROF-16316)

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

datadog-prod-us1-4[bot] reacted with thumbs up emoji
All reactions
  • 👍 1 reaction
@jbachorik jbachorik added the AI label Oct 25, 2026
@dd-octo-sts

dd-octo-sts Bot commented Oct 25, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

CI Test Results

Run: #43036053668 | Commit: 438f3c9 | Duration: 20m 5s (longest job)

✅ All 32 test jobs passed

Status Overview

JDK glibc-aarch64/debug glibc-amd64/debug musl-aarch64/debug musl-amd64/debug
8 - ✅ - -
8-ibm - ✅ - -
8-j9 ✅ ✅ - -
8-librca - - ✅ ✅
8-orcl - ✅ - -
11 - ✅ - -
11-j9 ✅ ✅ - -
11-librca - - ✅ ✅
17 ✅ ✅ - -
17-graal ✅ ✅ - -
17-j9 ✅ ✅ - -
17-librca - - ✅ ✅
21 ✅ ✅ - -
21-graal ✅ ✅ - -
21-librca - - ✅ ✅
25 ✅ ✅ - -
25-graal ✅ ✅ - -
25-librca - - ✅ ✅

Legend: ✅ passed | ❌ failed | ⚪ skipped | 🚫 cancelled

Summary: Total: 32 | Passed: 32 | Failed: 0


Updated: 2026-11-11 17:24:37 UTC

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@dd-octo-sts

dd-octo-sts Bot commented Oct 25, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

✅ All 40 integration tests passed

📊 Dashboard · 👷 Pipeline · 📦 23f8861f

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@jbachorik
jbachorik marked this pull request as ready for review October 28, 2026 14:56
@jbachorik
jbachorik requested a review from a team as a code owner October 28, 2026 14:56
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 28, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-28T14:59:42.438407Z 0ce265c Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

chatgpt-codex-connector[bot]
chatgpt-codex-connector Bot reviewed Oct 28, 2026
View reviewed changes

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0ce265c9ba

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread ddprof-lib/src/main/cpp/referenceChainTraversal.cpp Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread ddprof-lib/src/main/cpp/referenceChainTraversal.cpp Resolved

Uh oh!

There was an error while loading. Please reload this page.

@jbachorik jbachorik changed the title Fix reference-chain nightly flakes (PROF-16305) and drop their quarantine Fix reference-chain nightly flakes and drop their quarantine Oct 29, 2026
jbachorik added a commit that referenced this pull request Nov 2, 2026
@jbachorik
sphinx: address review feedback on PR #899
e237729
@dd-octo-sts

dd-octo-sts Bot commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

Reliability & Chaos Results

✅ All reliability & chaos checks passed Pipeline: https://gitlab.ddbuild.io/DataDog/java-profiler/-/pipelines/149491470

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

jbachorik added a commit that referenced this pull request Nov 3, 2026
@jbachorik
sphinx: address review feedback on PR #899
19eeb3f
@jbachorik

jbachorik commented Nov 3, 2026

Copy link
Copy Markdown
Collaborator Author

Fixed in 19eeb3f (addressing general review comment) — verified against the full claim: ddprof-lib/src/main/cpp/referenceChainAnchors.cpp:437-458 — compactLocalAnchors() now iterates only _local_anchor_tags (now a tag->own-class-tag map, referenceChains.h), so the locked scan is O(cap) and the per-slot count() double hash is gone; referenceChainAnchors.cpp:413-423 — a compaction that removes nothing arms LOCAL_ANCHOR_COMPACT_BACKOFF=8 skipped attempts, so a saturated-but-live share no longer pays the scan every pass (pinned by LocalRootAnchorCompactBackoffSkipsNoProgressCompactions). The O(index) vector compaction remains but runs only when entries are actually removed.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@jbachorik

jbachorik commented Nov 3, 2026

Copy link
Copy Markdown
Collaborator Author

Fixed in 19eeb3f (addressing general review comment) — verified against the full claim: ddprof-lib/src/main/cpp/referenceChainAnchors.cpp:362-392 — at a full pending buffer a confirmed container (met at the local cap) evicts the oldest not-yet-confirmed entry via the forward-only _pending_local_evict_cursor, dedup set kept exact; it is dropped only when every parked entry is already a confirmed container (pinned by LocalRootAnchorPendingBufferEvictsForConfirmedContainer).

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@jbachorik

jbachorik commented Nov 3, 2026

Copy link
Copy Markdown
Collaborator Author

Fixed in 19eeb3f (addressing general review comment) — verified against the full claim: ddprof-lib/src/main/cpp/referenceChainAnchors.cpp:641-667 — collection step extracted verbatim into const collectUnclassifiedAnchorClassTags(); ddprof-lib/src/test/cpp/referenceChainsAnchorTests.inc:1167 AnchorShapeReconcileBudgetCapsRequestedClasses parks budget+10 distinct classes and asserts the request is exactly ANCHOR_SHAPE_RECONCILE_BUDGET in order (fails for <= or a removed stop) and that the unrequested remainder is dropped by resolvePendingLocalAnchorsForTest().

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@jbachorik

jbachorik commented Nov 3, 2026

Copy link
Copy Markdown
Collaborator Author

Fixed in 19eeb3f (addressing general review comment) — verified against the full claim: ddprof-lib/src/test/cpp/referenceChainsAnchorTests.inc:1209 AnchorShapeReconcilePrefersPendingHolders — phase 1 asserts a full budget of pending classes is requested with no durable class (fails if loop order is swapped, or if room && is dropped from the index loop since the size would exceed the budget); phase 2 asserts pending classes first, then durable classes in index order, size == budget.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@dd-octo-sts

dd-octo-sts Bot commented Nov 7, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

Scan-Build Report

User:runner@runnervmwuk1x
Working Directory:/home/runner/work/java-profiler/java-profiler/ddprof-lib/src/test/make
Command Line:make -j4 all
Clang Version:Ubuntu clang version 21.1.8 (6ubuntu1)
Date:Wed Nov 11 17:02:56 2026

Bug Summary

Bug TypeQuantityDisplay?
All Bugs2
Logic error
Dereference of null pointer1
Unused code
Dead initialization1

Filter Results by File

Reports

Bug Group Bug Type ▾ File Function/Method Line Path Length
Unused codeDead initializationreferenceChains.cppapplyOomRamp4971
Logic errorDereference of null pointerperfEvents_linux.cppunregisterThread72026
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

jbachorik and others added 8 commits November 11, 2026 10:50
@jbachorik
Keep resolved leak chains across improve-path invalidations
7e32148
improveChain/reparent/root-kind upgrades queued a deferred invalidation that
drainPendingChainEvents applied as an erase at dump time. If the search then
terminated via the canary, nothing could rebuild the erased chains (candidate
and discovered state are zeroed, search pain-budget blocked ~11-15s), losing
the correlation for the rest of the window - against the restartSearch()
contract that resolved chains stay cached across a restart.

Split deferred invalidations into EVICT (noise eviction, filter: erased at
dump time as before) and REBUILD (path changed, source still live: applied by
the BFS-thread poll, which rebuilds the chain from the intact frontier and
keeps the old chain if the rebuild fails). Kind is encoded in-band in the
existing bounded pending buffers; no new allocation or locks.

Reproducer: ReferenceChainsBfsTest.ImprovedPathInvalidationKeepsLeakChainAcrossTerminalReset
(fails before, passes after). PROF-16316.
@kaahos @jbachorik
Fix reference-chain revisits and leak candidate admission
5e00ea7
@jbachorik
Make reference-chain slow e2e tests robust to walk-pacing latency
eead10d
The reference-chain walks are paced by the CPU pain budget; on slow cells the
second pass can land late in the tests' dump windows, leaving only the first
pass's uncorrelated chains (TL/LeakTag loose variants report out-of-pool or
not-found in CI nightlies, PROF-16305).

Test-side schedule changes within the existing bounds (no assertion weakened,
no retries/skips, static entry caps kept):
- cache-leak: growth completes by round 10, rounds 11-16 poll with a small
  top-up so the BFS drains its frontier backlog and intercepts a leak-tagged
  payload with slack (<= 36,601 entries, down from 69,001)
- failure-only referenceChainDiagnostics() helper appended to the
  assertNotNull messages of the cache-leak and gc-root tests (debug-
  instrumented builds only; release returns empty string)
- ThreadLocalLeakScenario: keeps full per-round chunk growth (the late-variant
  correlation needs the whole interception surface; verified on JDK 11) but
  keeps the filler at a quarter size so pass 0's pain-budget debt is repaid
  well inside the window
- LeakTagCorrelationScenario: front-loaded growth with sparse top-ups, tail
  rounds quiet; keep-best ladder unchanged
@jbachorik
Remove PROF-16305 quarantine from reference-chain slow tests
23a31e0
All four reference-chain slow e2e tests leave quarantine: the pr-889
regime plus the W5 test-side pacing fixes and the PROF-16316
improveChain invalidation fix (PR #898) cleared the failures. The
gc-root musl-aarch64 line is dropped after a 10/10 clean
musl-aarch64 JDK11 container validation of the full stack.
@jbachorik
sphinx: address review feedback on PR #899
db93e4b
@jbachorik
sphinx: address review feedback on PR #899
3f6a31d
@jbachorik
style(java): apply configured Java formatting
1d543fc
@jbachorik
Cap the durable anchor-index share with FIFO eviction
42f5c0b
@jbachorik
jbachorik force-pushed the fix-nightlies branch from 38c388c to 42f5c0b Compare November 11, 2026 10:33
datadog-prod-us1-4[bot]
datadog-prod-us1-4 Bot reviewed Nov 11, 2026
View reviewed changes

@datadog-prod-us1-4 datadog-prod-us1-4 Bot left a comment

Copy link
Copy Markdown

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: FAIL

Durable-anchor eviction lacks a working traversal-based readmission path. Once the index exceeds its cap, live collections can lose targeted walks, causing later backing-storage replacements and leak instances to be missed.

Open Bits AI session

🤖 Bits Code Review · Commit 42f5c0b

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread ddprof-lib/src/main/cpp/referenceChainAnchors.cpp Resolved

Uh oh!

There was an error while loading. Please reload this page.

@datadog-prod-us1-4

datadog-prod-us1-4 Bot commented Nov 11, 2026 •
edited by datadog-datadog-prod-us1-2 Bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown

Pipelines

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 23f8861 | Docs | View more details | Give us feedback!
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@jbachorik
Re-enlist evicted durable anchors from traversal reports
6444508
datadog-prod-us1-4[bot]
datadog-prod-us1-4 Bot reviewed Nov 11, 2026
View reviewed changes

@datadog-prod-us1-4 datadog-prod-us1-4 Bot left a comment

Copy link
Copy Markdown

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: FAIL

The PR preserves leak chains, improves traversal/admission, hardens slow tests, and removes their quarantine; the latest commit restores callback-based anchor readmission. The earlier finding remains partially open: sweep completion can discard resweep requests, and eviction-driven cooldowns can stall, leaving static-only collections unindexed and missing later leak instances.

Open Bits AI session

🤖 Bits Code Review · Commit 6444508

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread ddprof-lib/src/main/cpp/referenceChainAnchors.cpp Resolved

Uh oh!

There was an error while loading. Please reload this page.

@jbachorik
Test the re-enlist wiring in both traversal callbacks
ac7570d
@jbachorik
Keep eviction resweep requests separate from sweep completion
957ad12
datadog-prod-us1-4[bot]
datadog-prod-us1-4 Bot reviewed Nov 11, 2026
View reviewed changes

@datadog-prod-us1-4 datadog-prod-us1-4 Bot left a comment

Copy link
Copy Markdown

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: FAIL

The PR preserves leak chains, improves traversal, hardens slow tests, and removes quarantine; callback-based anchor readmission is fixed. The earlier scheduling finding remains open despite the pending flag: sweep completion still loses requests and cooldown suppresses later requests, so static-only holders can miss targeted leak discovery.

Open Bits AI session

🤖 Bits Code Review · Commit 957ad12

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread ddprof-lib/src/main/cpp/referenceChainTraversal.cpp Resolved

Uh oh!

There was an error while loading. Please reload this page.

@jbachorik
Never drop eviction resweep requests during the restart cooldown
85eab40
datadog-prod-us1-4[bot]
datadog-prod-us1-4 Bot reviewed Nov 11, 2026
View reviewed changes

@datadog-prod-us1-4 datadog-prod-us1-4 Bot left a comment

Copy link
Copy Markdown

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: FAIL

The PR preserves leak chains, improves traversal, hardens slow tests, and removes quarantine; earlier request-loss and cooldown mechanisms are fixed. Forced resweeps can still stop after one chunk, excluding later static-only holders from targeted leak discovery, and the new cooldown regression test never triggers an eviction.

Open Bits AI session

🤖 Bits Code Review · Commit 85eab40

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread ddprof-lib/src/main/cpp/referenceChainTraversal.cpp Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread ddprof-lib/src/test/cpp/referenceChainsAnchorTests.inc Resolved

Uh oh!

There was an error while loading. Please reload this page.

@jbachorik
Serve resweep generations only by a completing lap from class 0
23f8861
datadog-prod-us1-4[bot]
datadog-prod-us1-4 Bot reviewed Nov 11, 2026
View reviewed changes

@datadog-prod-us1-4 datadog-prod-us1-4 Bot left a comment

Copy link
Copy Markdown

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: PASS

More details

The PR preserves resolved leak chains, improves traversal/admission, hardens slow tests, and removes quarantine. Durable-anchor resweeps now remain enabled across chunks until completion, and the cooldown regression case refills to capacity and checks an eviction-generation increment; both reported findings are resolved, with no open findings remaining.

Was this helpful? React 👍 or 👎

Open Bits AI session

🤖 Bits Code Review · Commit 23f8861

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@rkennke rkennke mentioned this pull request Nov 12, 2026
Add PR-description staleness guardrails (PROF-16592) #980
Merged
2 of 3 tasks

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@chatgpt-codex-connector chatgpt-codex-connector[bot] chatgpt-codex-connector[bot] left review comments
@datadog-prod-us1-4 datadog-prod-us1-4[bot] datadog-prod-us1-4[bot] left review comments

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

AI

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

2 participants

@jbachorik @kaahos
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.