GeezerVison

…e-common

Move the standalone `native` crate into a root Cargo workspace and extract
shared JNI plumbing (error->exception mapping, Tokio runtime singleton,
StreamingReader) into a new `datafusion-jni-common` crate under `native-common/`.
`native/src/errors.rs` moves to `native-common/src/errors.rs`; the nine native
modules now import error/runtime helpers from `datafusion_jni_common`.

Build glue follows: single root `Cargo.lock`, `.cargo/config.toml` redirects
output to `rust-target/`, Makefile/CI/poms updated to build `--workspace` and
target `-p datafusion-jni`. Core javadoc build commands updated to match.

Pure refactor; no behavior change. First of a 6-PR stack splitting the Spark
DataSource V2 connector work.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

New `spark/bridge` workspace crate providing the `export_bridge!` macro that
generates the six JNI entry points a Spark connector bridge exposes
(providerSchemaIpc, createScan, partitionCount, executeStreamPartition,
executeStream, closeScan). Includes the options decoder, scan planning/execution
glue, and the Arrow type-widening layer (wraps any TableProvider for Spark type
compatibility).

Self-contained SDK with no Java/Scala coupling. Depends only on
datafusion-jni-common. Second of the 6-PR connector stack.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

Introduce the `spark` Maven module and the pure-Java contracts a bridge
implements: BridgeProviderFactory (no-arg factory + scanBackend()), ScanBackend
(delegates to the bridge's JNI methods), NativeLibraryLoader (cdylib
extraction/loading), OptionsCodec (cross-language options encoder), PartitionInfo
(one entry per Spark task), and ReportedPartitioning (optional shuffle-elision
declaration).

Compiles standalone with no Scala main yet. Includes the two SPI-only tests
(OptionsCodecTest, BridgeProviderFactoryDefaultsTest). Third of the 6-PR stack.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

The connector implementation on top of the Java SPI and the bridge SDK:
DatafusionSource/Table/Scan/ScanBuilder DSv2 wiring, per-partition columnar
read path (FfiStream + Arrow->Spark batch conversion), V2 predicate pushdown
(SparkPredicateTranslator), shared-scan mode with a per-executor refcounted
cache (SharedScanCache, SharedScanPartitionReader, NativeSharedScanResources,
PinnedSessionConfig), and SupportsReportPartitioning for shuffle elision.

These pieces share the DatafusionScanMode sealed trait and the scan builder, so
they land together. Includes the connector test suite and the module README.
DataSourceRegister SPI file registers DatafusionSource. Fourth of the 6-PR stack.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

Add `spark/scaffold/new_bridge.py` plus the `bridge-template/` it stamps out: a
standalone Maven+Cargo bridge project wired to the datafusion-spark-bridge SDK —
a Rust cdylib with `export_bridge!` + a demo in-memory provider, the four Java
classes, the DataSourceRegister service file, a shaded-jar pom that bundles the
cdylib, and a pyspark smoke test. Stdlib-only generator.

Standalone tooling. Fifth of the 6-PR stack.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

Add a worked example exercising the full stack: examples/native cdylib (a demo
provider built on datafusion-spark-bridge, added as a workspace member), the
ExampleBridge Java classes implementing the SPI, a pyspark bridge_demo.py
end-to-end smoke test, and READMEs. Validates connector + bridge + SDK together.

Last of the 6-PR stack.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>