GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — feat: native pagerank by SemyonSinchenko · Pull Request #904 · graphframes/graphframes

Conversation

@SemyonSinchenko

@SemyonSinchenko SemyonSinchenko commented Sep 16, 2026

Copy link
Copy Markdown
Collaborator

What changes were proposed in this pull request?

  1. new PageRank contract (v2)
  2. deprecation of v1
  3. native implementation (GF Pregel)

Why are the changes needed?

Close #569
Part of #714

No user facing changes right now: docs, pyspark api, etc. in the follow-up PR

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
SemyonSinchenko added 2 commits September 16, 2026 15:27
@SemyonSinchenko
feat: native PageRank implementation
fa3d60f
- no graphx
- returns only one DataFrame
@SemyonSinchenko
chore: use mixin
2e8f409
@SemyonSinchenko
SemyonSinchenko requested a review from james-willis September 16, 2026 13:57
@SemyonSinchenko SemyonSinchenko self-assigned this Sep 16, 2026
@SemyonSinchenko SemyonSinchenko added the scala label Sep 16, 2026
@SemyonSinchenko SemyonSinchenko mentioned this pull request Sep 16, 2026
[EPIC] Alternative implementations of GraphX based algorithms #556
Open
2 of 6 tasks
@SemyonSinchenko
fix: ambigous link
39718cb
raviston
raviston reviewed Nov 10, 2026
View reviewed changes

@raviston raviston left a comment

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, nice to see PageRank on the Pregel engine. I went through it against the vendored GraphX implementation (from reading the code; I couldn't run the suites on this branch locally). The core looks right to me:

  • seeding the rank and the delta with resetProb reproduces the bootstrap of the GraphX dynamic PageRank;
  • the out-degree join keeps sinks and isolated vertices;
  • messages only use source columns, so the second join is skipped and inactive sources are pruned;
  • vertices without messages keep their rank through the left_outer join.

One thing that matters for fixed-iteration mode, and a few edge cases (inline):

  1. Truncation in maxIter mode for personalized PageRank (inline at DELTA_EPS).
  2. maxIter semantics differ from v1, and no test compares them. v1 (GraphX static) starts every rank at 1.0 and applies maxIter updates, while v2 seeds resetProb and runs maxIter message rounds. Unnormalized, after k iterations v1 has resetProb * sum_{j<k} (dM)^j 1 + (dM)^k 1 and v2 has resetProb * sum_{j<=k} (dM)^j 1, so they differ by d * (dM)^k 1, which is about 0.85^(k+1) (about 17% of the tail at k = 10). Normalization does not cancel this on irregular graphs, so the two only agree once converged. The comparison with v1 in PageRankSuite covers tol mode only, and the maxIter tests check counts and one zero rank, not values. The LDBC test does check maxIter values, but against the LDBC reference, not v1. Since the API mirrors v1, could you add a note on this to the docs (or the deprecation message), and a value test for maxIter mode, for example on a small hand-computed graph?
  3. NaN when every rank is 0 (inline).
  4. Possible non-terminating run in tol mode (inline).
  5. Deprecation warning before there is a way to migrate (inline).

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread core/src/main/scala/org/graphframes/lib/PageRankV2.scala
* to the untruncated run, while still shrinking the active set of vertices when the algorithm
* approaches the fixpoint.
*/
private val DELTA_EPS = 1e-8

@raviston raviston Nov 10, 2026

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In fixed-iteration mode the participation filter is an absolute threshold on unnormalized deltas. For the uniform case that is harmless: each vertex starts at resetProb, so deltas are around 0.15 regardless of the graph size. In the personalized case, though, the total mass is 1.0 and spreads over the reachable set, so per-vertex deltas scale like 1 / reach.

For example, with d = 0.85 and about 1e8 vertices reachable at hop 4, a vertex receives about 0.85^4 / 1e8, roughly 5e-9, which is below DELTA_EPS. About half of the total mass then stops propagating after hop 4, and the ranks of farther vertices are underestimated by a lot. So "numerically identical to the untruncated run" holds for the uniform case but not for personalized PageRank on large graphs.

Two options: disable the participation filter in maxIter mode (use it only when tol is set), or make the threshold relative, for example scaled by 1 / numVertices in the personalized case.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread core/src/main/scala/org/graphframes/lib/PageRankV2.scala

// Normalize the ranks so they sum up to 1.0 and project out all the internal columns
// (out-degrees, deltas and the active flag).
val rankSum = rawVertices.agg(coalesce(sum(col(PAGERANKS)), lit(1.0))).head().getDouble(0)

@raviston raviston Nov 10, 2026

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

coalesce(sum, 1.0) covers the empty graph, but not a zero sum. If sourceId is not a vertex of the graph, every rank stays 0.0, and so does every rank with resetProbability(0.0) in the uniform case. Then this divides by zero and every rank becomes NaN. GraphX's normalizeRankSum has the same behaviour, but a require that the source exists, or a guard for rankSum == 0, would give a clear error instead of silent NaNs.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread core/src/main/scala/org/graphframes/lib/PageRankV2.scala
srcId,
resetProb,
Some(t),
Int.MaxValue,

@raviston raviston Nov 10, 2026

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With Int.MaxValue the run relies entirely on the deltas decaying. In the personalized case with resetProbability(0.0), damping = 1, so the mass circulates on any cycle without shrinking, and the loop never stops. (The GraphX runUntilConvergence has the same property.) A require(resetProb > 0) in tol mode, or a large but finite iteration cap, would avoid it.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread core/src/main/scala/org/graphframes/lib/PageRank.scala
extends Arguments
with Logging {

logWarn(

@raviston raviston Nov 10, 2026

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This warning fires on every g.pageRank call. The Python API (classic and Connect) and the Connect plugin (GraphFramesConnectUtils) still call pageRank, and pageRankV2 isn't exposed there yet. So Python and Connect users get a message they can't act on, although the description says there are no user-facing changes yet. Would it make sense to add the warning in the follow-up that exposes v2 in Python and Connect?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@james-willis james-willis Awaiting requested review from james-willis
1 more reviewer
@raviston raviston raviston left review comments
Reviewers whose approvals may not affect merge requirements

At least 1 approving review is required to merge this pull request.

Uh oh!

There was an error while loading. Please reload this page.

Assignees

@SemyonSinchenko SemyonSinchenko

Labels

scala

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

feat: PageRank without GraphX

Uh oh!

There was an error while loading. Please reload this page.

2 participants

@SemyonSinchenko @raviston
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.