Summary

A query combining many UNION ALL branches with ROW_NUMBER() causes excessive memory consumption during SQL compilation.

Using an unmodified upstream compiler, the attached anonymized SQL took approximately 369 seconds and reached 8.38 GiB peak RSS. A local prototype that avoids expanding expression lineage for window expressions compiled the same SQL in approximately 4.7–4.8 seconds, using around 522–531 MiB peak RSS.

All three runs generated byte-identical Rust source.

This happens during SQL compilation, without starting a Pipeline or ingesting any data.

Environment

• Feldera revision: 0f3c32534cbc1ba00467010566b288cd7b0b6b4e
• Calcite revision: a2e574ec271866eb352f98d7101a386a30dc8bc8
• macOS, Apple M1 Max, 32 GiB RAM
• Zulu Open JDK 21.0.2
• JVM options: -Xmx8g -XX:ActiveProcessorCount=6
• Default compiler optimization level
• Runs executed serially on the same machine

oom-anonymized.sql.txt

Reproduction

Attach oom-anonymized.sql.txt and rename it to oom-anonymized.sql.

The attachment is 97,062 bytes and contains:

• 25 tables
• 91 views
• 89 UNION ALL operators
• A final ROW_NUMBER() window over the combined results, followed by filtering on the row number

Unused UDF declarations and connector configurations have been removed. Business identifiers and string literals have been anonymized.

Build the compiler at the revision above, then run:

env -u JAVA_TOOL_OPTIONS -u JDK_JAVA_OPTIONS -u _JAVA_OPTIONS \
  java -Xmx8g -XX:ActiveProcessorCount=6 \
  -Xlog:gc:file=gc.log:time,uptime,level,tags \
  -jar sql2dbsp-jar-with-dependencies.jar \
  oom-anonymized.sql \
  -o generated.rs

Input SHA-256:

884c6b0b17379844276ade8b3ec2118cf46650fd9acb73f2b8adfa27eed60f62

Results

The upstream run spent approximately 87% of its elapsed time in Full GC.

Importantly, this particular upstream run eventually succeeded; it did not terminate with a fatal OOM. An exit code of zero does not necessarily rule out an exception caught inside an optional optimizer step.

The generated Rust files were byte-identical across all three runs:

93ad810eea8e6097fe1bfb42cc4c96cf6c65068b88e281886127481d78a9c748

These measurements cover SQL-to-Rust source generation only, not native Rust compilation or runtime correctness.

Root-cause investigation

The investigation points to expression-lineage expansion, rather than input data volume or runtime state.

The relevant call chain is:

ExpandDisjunctionForTableRule.apply
  -> RelMetadataQuery.getExpressionLineage
  -> RelMdExpressionLineage.getExpressionLineage(Union)
  -> createAllPossibleExpressions
  -> createExpressions
  -> copying RexOver / RexWindow expressions and constructing their digests

The UNION lineage handler gathers possible origins for each referenced column. The expression-expansion code then enumerates combinations of those origins.

For a window expression referencing several columns, this can produce a combinatorial number of copied expressions, rather than work proportional to the number of UNION branches.

Calcite already returns unknown lineage for references to results produced by a relational Window node. However, a window represented as RexOver inside a Project or Calc expression can still enter this expansion path.

Additional heap evidence

In a separate diagnostic run of the original, pre-anonymization query with an earlier local compiler build and a 12 GiB heap, compilation encountered Java heap space.

Heap analysis showed:

• One HashSet retained approximately 11.78 GiB, or 99.36% of the analyzed heap.
• Its retained object graph contained approximately 24 million RexOver objects and 24 million RexWindow objects.
• Associated strings, byte arrays, collections and hash-map entries accounted for much of the remaining memory.

That heap dump is supporting evidence from a separate run, not a measurement from the clean-upstream benchmark above.

Prototype mitigation

I tested a local RelMetadataQuery override that returns unknown lineage before dispatching queries whose expression contains a window:

@Override
public @Nullable Set<RexNode> getExpressionLineage(
        RelNode rel, RexNode expression) {
    if (RexOver.containsOver(expression)) {
        return null;
    }
    return super.getExpressionLineage(rel, expression);
}

The custom query is installed through the cluster's metadata-query supplier, so recursive lineage queries and query invalidation use the same guard.

This does not remove the window operation or globally disable optimizer rules. It withholds optional lineage metadata for window expressions while retaining ordinary column/scalar lineage behavior.

For the attached query, it eliminated the excessive allocation and Full GC while preserving the generated Rust output exactly. This is a result for this reproducer, not a general semantic-equivalence proof.

Would guarding window-expression lineage in Calcite be the preferred fix, or should Feldera apply this restriction at its metadata-query boundary?