Write your data pipeline once. Run it anywhere.

You write your pipeline against a single API, then decide how it runs. Point it at one engine and it runs there. Or hand Wayang's cost-based optimizer the choice and let it pick the best platform for each step across your laptop, Apache Spark, Apache Flink, or a database, even splitting a single job across several. Either way, when your data outgrows one machine you don't rewrite anything, you just make another engine available.

• How it works
• Quickstart
• Install
• Documentation
• Research
• Contributing
• Community
• Authors
• License
• Acknowledgements

Most data processing systems are designed around a single execution engine. That keeps things simple, but your pipeline ends up tied to that engine's API. So combining engines, or moving to another, typically means rewriting and gluing together which is costly and time-consuming.

Wayang sits one level up. You write a pipeline against Wayang's API and register the engines you have. Then it's your call. Want control? Register one engine and it runs there. Want it handled? Register several and let the cost-based optimizer pick the best one for each step, even splitting a single job across engines.

Supported platforms today

• Java Streams
• Apache Spark
• Apache Flink
• Apache Giraph
• PostgreSQL
• SQLite
• Apache Kafka
• TensorFlow

Wayang's APIs

• Java (Scala-like fluent builder)
• Scala
• SQL
• Java native (low-level, we recommend the fluent scala-like)

The plugin architecture makes adding new operators and platforms straightforward without touching internals — see Adding operators.

We'll run a word count locally first — no cluster, nothing to install on a server — then make Spark available with a one-line change. The pipeline itself never changes; only the set of engines you register does.

import org.apache.wayang.core.api.Configuration;
import org.apache.wayang.core.api.WayangContext;
import org.apache.wayang.api.JavaPlanBuilder;
import org.apache.wayang.basic.data.Tuple2;
import org.apache.wayang.java.Java;
import java.util.Arrays;

public class WordCount {
    public static void main(String[] args) {
        // Register ONLY the local Java engine → runs on your machine, no cluster needed.
        WayangContext wayang = new WayangContext(new Configuration())
                .withPlugin(Java.basicPlugin());

        new JavaPlanBuilder(wayang)
                .withJobName("WordCount")
                .withUdfJarOf(WordCount.class)
                .readTextFile("file:///path/to/input.txt")
                .flatMap(line -> Arrays.asList(line.split("\\W+")))
                .filter(word -> !word.isEmpty())
                .map(word -> new Tuple2<>(word.toLowerCase(), 1))
                .reduceByKey(Tuple2::getField0,
                             (t1, t2) -> new Tuple2<>(t1.getField0(), t1.getField1() + t2.getField1()))
                .writeTextFile("file:///path/to/output.txt", t -> t.getField0() + ": " + t.getField1());
    }
}

It executes locally. Good for development, tests, and small data.

Now run the exact same pipeline on Spark instead of locally. You don't touch the pipeline — you change which platform you register: comment out Java and register Spark.

import org.apache.wayang.spark.Spark;               // swap the import

// Same pipeline as before — only the registered platform changed.
WayangContext wayang = new WayangContext(new Configuration())
        // .withPlugin(Java.basicPlugin())           // comment out the local engine
        .withPlugin(Spark.basicPlugin());            // register Spark instead

Run it again. The same pipeline now executes on Spark. You changed where it runs without changing what it does. Switch to Flink or any other supported platform the same way: swap the import and the registered plugin.

Why register only Spark here? Wayang's real power is registering several platforms and letting the optimizer pick. But on small test data the optimizer will almost always pick the local engine (Spark's startup overhead isn't worth it for a tiny file) so you'd never actually see Spark run. Registering Spark alone forces the issue so you can confirm it works. Step 3 shows the production pattern.

This is the point of Wayang. In practice you don't pick a platform at all: you register every engine you have and let the optimizer choose the best one for each step.

// Register BOTH platforms — Wayang's optimizer decides which to use per step.
WayangContext wayang = new WayangContext(new Configuration())
        .withPlugin(Java.basicPlugin())
        .withPlugin(Spark.basicPlugin());

Now Wayang owns the placement decision. For each operator it estimates the cost on every registered platform and picks the cheapest, keeping a small job entirely local, pushing a large one onto Spark, or mixing both within the same job as the data and query demands. On a tiny input you'll see it keep everything local (that's the optimizer working correctly, not ignoring Spark); cross-platform splits show up once the data is big enough to justify them.

Replace WAYANG_VERSION with the latest Maven Central release.

<dependency>
  <groupId>org.apache.wayang</groupId>
  <artifactId>wayang-core</artifactId>
  <version>WAYANG_VERSION</version>
</dependency>
<dependency>
  <groupId>org.apache.wayang</groupId>
  <artifactId>wayang-basic</artifactId>
  <version>WAYANG_VERSION</version>
</dependency>
<dependency>
  <groupId>org.apache.wayang</groupId>
  <artifactId>wayang-api-scala-java</artifactId>
  <version>WAYANG_VERSION</version>
</dependency>
<!-- add one artifact per engine you want available -->
<dependency>
  <groupId>org.apache.wayang</groupId>
  <artifactId>wayang-java</artifactId>
  <version>WAYANG_VERSION</version>
</dependency>
<dependency>
  <groupId>org.apache.wayang</groupId>
  <artifactId>wayang-spark</artifactId>
  <version>WAYANG_VERSION</version>
</dependency>

The available modules:

• wayang-core — core data structures and the optimizer (required)
• wayang-basic — common operators and data types (recommended)
• wayang-api-scala-java — fluent Scala/Java API for building plans (recommended)
• wayang-java, wayang-spark, wayang-flink, wayang-postgres, wayang-sqlite3, wayang-graphchi, wayang-tensorflow, wayang-kafka — per-platform adapters; include one per engine you want available
• wayang-profiler — learns operator and UDF cost functions from historical executions

For snapshot builds, add Apache's snapshot repository:

<repositories>
  <repository>
    <id>apache-snapshots</id>
    <name>Apache Foundation Snapshot Repository</name>
    <url>https://repository.apache.org/content/repositories/snapshots</url>
  </repository>
</repositories>

git clone https://github.com/apache/wayang.git
cd wayang
./mvnw clean install -DskipTests

The current snapshot version lives in pom.xml.

• Java 17 — set JAVA_HOME to your Java 17 installation.
• Apache Spark 3.4.4 with Scala 2.12 — set SPARK_HOME.
• Apache Hadoop 3+ — set HADOOP_HOME.
• Maven for building from source.

--add-exports=java.base/sun.nio.ch=ALL-UNNAMED
--add-opens=java.base/java.nio=ALL-UNNAMED
--add-opens=java.base/java.lang=ALL-UNNAMED
--add-opens=java.base/java.util=ALL-UNNAMED
--add-opens=java.base/java.io=ALL-UNNAMED
--add-opens=java.base/java.lang.reflect=ALL-UNNAMED
--add-opens=java.base/java.util.concurrent=ALL-UNNAMED
--add-opens=java.base/java.net=ALL-UNNAMED
--add-opens=java.base/java.lang.invoke=ALL-UNNAMED

After building, unpack the assembly and put Wayang on your PATH:

tar -xvf wayang-WAYANG_VERSION.tar.gz
cd wayang-WAYANG_VERSION

# Linux
echo "export WAYANG_HOME=$(pwd)" >> ~/.bashrc
echo "export PATH=${PATH}:${WAYANG_HOME}/bin" >> ~/.bashrc
source ~/.bashrc

# macOS
echo "export WAYANG_HOME=$(pwd)" >> ~/.zshrc
echo "export PATH=${PATH}:${WAYANG_HOME}/bin" >> ~/.zshrc
source ~/.zshrc

Then run the bundled WordCount on your local Java engine:

bin/wayang-submit org.apache.wayang.apps.wordcount.Main java file://$(pwd)/README.md

./mvnw test

• Getting started — the full tabbed walkthrough in Java, Scala, and Python.
• How Wayang chooses a platform — what drives the optimizer's decisions.
• Adding operators — extend Wayang with new operators or platforms.
• Example applications — runnable apps in this repo.
• Developing with Wayang — using Wayang in your own Java/Scala project.

Wayang (formerly called Rheem) is the product of many years of top quality research. Below you can find the main publications:

• Apache Wayang: A Unified Data Analytics Framework. SIGMOD Rec. 52(3): 30-35 (2023) pdf
• Apache Wayang in Action: Enabling Data Systems Integration via a Unified Data Analytics Framework. SIGMOD Conference Companion 2025: 35-38 pdf
• ML-based Cross-Platform Query Optimization. ICDE 2020: 1489-1500 pdf
• RHEEMix in the data jungle: a cost-based optimizer for cross-platform systems. VLDB J. 29(6): 1287-1310 (2020) pdf
• RHEEM: Enabling Cross-Platform Data Processing - May The Big Data Be With You! -. Proc. VLDB Endow. 11(11): 1414-1427 (2018) pdf

If you use Apache Wayang in your research, please cite the SIGMOD Record paper:

@article{beedkar2023wayang,
  author  = {Kaustubh Beedkar and Bertty Contreras-Rojas and Haralampos Gavriilidis and Zoi Kaoudi and Volker Markl and Rodrigo Pardo-Meza and Jorge-Arnulfo Quian{\'{e}}-Ruiz},
  title   = "{Apache Wayang: A Unified Data Analytics Framework}",
  journal = {{SIGMOD} Rec.},
  volume  = {52},
  number  = {3},
  pages   = {30--35},
  year    = {2023},
  doi     = {10.1145/3631504.3631510}
}

Contributions are welcome — bug reports, doc fixes, new platform adapters, new operators, optimizer improvements, anything. Start with CONTRIBUTING.md and the building guide, open an issue if you're not sure where to start, and introduce yourself on the dev mailing list — that's where active work gets discussed.

If you're looking for somewhere to begin, doc improvements, new operators, and additional examples are areas where a focused PR can land quickly.

• Mailing lists — https://wayang.apache.org/docs/community/mailinglist (user and dev)
• LinkedIn — Apache Wayang

See the full list of contributors.

All files in this repository are licensed under the Apache License 2.0.

Copyright 2020 - 2026 The Apache Software Foundation.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

The logo was donated by Brian Vera.