GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — docs: randomized_contraction always needs a checkpoint directory by yuchaoran2011 · Pull Request #909 · graphframes/graphframes

Conversation

@yuchaoran2011

@yuchaoran2011 yuchaoran2011 commented Sep 29, 2026

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

Corrects the docs on useLocalCheckpoints for randomized_contraction. They currently say local checkpoints remove the need for a checkpoint directory for both connected components algorithms, but that holds for two_phase only.

Documented Actual
useLocalCheckpoints removes the need for a checkpoint directory, for two_phase and randomized_contraction two_phase skips the directory when it is set (TwoPhase.scala:271). randomized_contraction always requires one (RandomizedContraction.scala:63-75), because it writes each iteration's representatives there and reads them back by path
an IOException is thrown only if no directory is set "and local checkpoints are not enabled" randomized_contraction throws whenever no directory is set

Changed: docs/src/04-user-guide/13-configurations.md (four sentences) and docs/src/04-user-guide/05-traversals.md (one sentence added to use_local_checkpoints). No code changes.

Why are the changes needed?

Following the current docs, a user who sets use_local_checkpoints=True for randomized_contraction still gets java.io.IOException: Checkpoint directory is not set:

g = GraphFrame(
    spark.createDataFrame([(1,), (2,), (3,)], ["id"]),
    spark.createDataFrame([(1, 2)], ["src", "dst"]),
)

# works
g.connectedComponents(algorithm="two_phase", use_local_checkpoints=True).collect()

# throws
g.connectedComponents(algorithm="randomized_contraction", use_local_checkpoints=True).collect()

On Spark Connect this leaves no way forward from the client, since spark.checkpoint.dir is a static conf there (CANNOT_MODIFY_CONFIG).

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

SemyonSinchenko reacted with thumbs up emoji
All reactions
  • 👍 1 reaction
@yuchaoran2011
docs: randomized_contraction always needs a checkpoint directory
cf1c877
useLocalCheckpoints removes the need for a checkpoint directory for two_phase (TwoPhase.scala:271), but randomized_contraction always requires one (RandomizedContraction.scala:63-75): it writes each iteration's representatives under that directory and reads them back by path.
Copilot AI balanced review requested due to automatic review settings September 29, 2026 08:52
Copilot started reviewing on behalf of yuchaoran2011 September 29, 2026 08:53
View session
Copilot AI reviewed Sep 29, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

The documentation now accurately matches both connected-components implementations.

Review effort: Balanced
Findings: None

What changed in this PR

Corrects connected-components documentation to reflect that randomized_contraction always requires a checkpoint directory.

Changes:

  • Clarifies checkpoint requirements by algorithm.
  • Documents randomized_contraction’s intermediate-table storage requirement.
File Description
docs/​src/​04-user-guide/​13-configurations.md Corrects configuration and checkpoint guidance.
docs/​src/​04-user-guide/​05-traversals.md Clarifies use_local_checkpoints behavior.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
SemyonSinchenko
SemyonSinchenko approved these changes Sep 29, 2026
View reviewed changes
@SemyonSinchenko

SemyonSinchenko commented Sep 29, 2026

Copy link
Copy Markdown
Collaborator

@yuchaoran2011 Thanks for the contribution!

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@SemyonSinchenko
SemyonSinchenko merged commit 407773a into graphframes:main Sep 29, 2026
12 checks passed

Uh oh!

There was an error while loading. Please reload this page.

@yuchaoran2011
yuchaoran2011 deleted the docs/randomized-contraction-checkpoint-dir branch October 4, 2026 22:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@SemyonSinchenko SemyonSinchenko SemyonSinchenko approved these changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

3 participants

@yuchaoran2011 @SemyonSinchenko
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.