Tested against the Rust reference implementation v0.93.0. For the rest of the API surface (reader, writer, scan, CLI), see reference.md.

The reader and inspector modules carry no transitive dependency on the writer module. A consumer that only needs to read Vortex files can depend on a strict subset:

<dependency>
  <groupId>io.github.dfa1.vortex</groupId>
  <artifactId>vortex-reader</artifactId>
  <version>0.20.0</version>
</dependency>

<!-- optional: inspector for layout-tree introspection -->
<dependency>
  <groupId>io.github.dfa1.vortex</groupId>
  <artifactId>vortex-inspector</artifactId>
  <version>0.20.0</version>
</dependency>

The zstd binding is optional on both sides: files using vortex.zstd need io.github.dfa1.zstd:zstd plus io.github.dfa1.zstd:zstd-platform on the classpath (versions pinned by the vortex BOM); without them, touching a vortex.zstd segment fails with a VortexException that names the two artifacts. All other encodings are pure Java.

./mvnw -pl core,reader,inspector verify builds the read-only artifact set without the writer module on the classpath. ReadRegistry.loadAll() registers only the built-in decoders in reader; no encoder class is loaded.

Bulk loops (min/max, sums, run counting, widening, bit-packing, ...) live behind SimdOperations. Two implementations exist: the default, plain Java loops shaped for C2's auto-vectorizer, and one written with the incubating Vector API, selected only when the JVM is launched with --add-modules jdk.incubator.vector (omitting the flag is the opt-out; the JVM prints WARNING: Using incubator modules once). Both give Rust's results, pinned by a differential test: floating-point sums are sequential (lane-wise accumulation would change the last bits), and the I64 sum sums in vector lanes only when no prefix can overflow (every element in [-2^k, 2^k) with n * 2^k <= 2^63), otherwise it runs Rust's checked add per element, so [MAX, 1, -1] is dropped in both. F16 sum and min/max are scalar in both.

Separate from the Vector API, found while comparing against the Rust source:

• Float min/max statistics order -0.0 before 0.0 and skip NaN, as Rust's total_compare with skip_nans does.
• Float filters order values totally, as vortex-jni does: -0.0 < 0.0, and NaN equals itself and is the maximum (so NaN >= 0.0 matches, where IEEE says it does not). Zone-map pruning follows: a float zone without a nan_count (the legacy vortex.stats table) may hold a NaN, so it is never pruned on its maximum. Pinned by JavaWritesRustReadsIntegrationTest#javaWriter_jniReader_filteredFloats_matchJni (#518).
• A float or decimal filter operand on an integer column is refused (VortexException), as Rust's compare refuses operands of different dtypes. Integer operands of any width are still accepted, a deliberate leniency: Rust would refuse an i32 literal against an i64 column too. Pinned by FractionalOperandInteropIntegrationTest (#701).
• U64 min/max and the dense-counter span in ArrayStats are computed in unsigned order, as Rust's typed_int_stats does.
• The zoned float Sum skips NaN, as Rust's skip_nans() binding does; an all-NaN zone sums to 0.0 (#523).

Cross-implementation conformance against Raincloud (tracked in #205): 430 curated public datasets whose .vortex files are written by the Vortex Python bindings (vortex-data 0.86.1, pinned via Raincloud v0.3.0) with a Parquet sibling built from the same Arrow table. RaincloudConformanceIntegrationTest reads each hydrated .vortex with vortex-java, exports CSV, and diffs it against the Parquet sibling read by hardwood (the oracle) — a zero-diff proves every value survived the Python-write / Java-read boundary.

scripts/hydrate-raincloud-corpus.sh --max-mb 200      # cache → mirror → local build
./mvnw verify -pl integration -am -Dvortex.it.excludedGroups= -Dit.test=RaincloudConformanceIntegrationTest

The 96 tpcgen-rs-tpcds-* datasets are generated by tpcgen-cli, which Raincloud does not install: build it once, with a Rust toolchain, from a tpcgen-rs checkout at the commit the error message names (<version>+git.<commit>). The checkout's rust-toolchain.toml fetches the pinned compiler; the installer drops the binary and the receipt Raincloud checks into the venv.

brew install rustup && export PATH="$(brew --prefix rustup)/bin:$PATH"
git clone https://github.com/clflushopt/tpchgen-rs tpcgen-rs && git -C tpcgen-rs checkout <commit>
~/.cache/raincloud/conformance-venv-v0.3.0/bin/python -m raincloud.pipeline.generators.install_tpcgen --source tpcgen-rs

The conformance test proves values match; scripts/RaincloudCompare.java reports what they cost. Per hydrated slug it prints the Parquet size, two Rust sizes (Raincloud's file, from the vortex-data its release pins, and that file rewritten by vortex-jni at the Rust version vortex-java mirrors), the Java size with default and compact write options, the import time, and the full-scan time of the Rust file and of Java's, flagging a Java file over 2% larger than vortex-jni's. It reports and never fails: size is information, not a reason to diverge from Rust. vortex-jni rewrites the batches its scan yields, not the Parquet row groups Java imports, so its chunk boundaries can differ slightly. It needs vortex-jni and Arrow, so it runs with the integration module's test class path and Arrow's JVM flags:

./mvnw -q -pl integration -am -DskipTests package dependency:build-classpath \
    -Dmdep.includeScope=test -Dmdep.outputFile=target/test-classpath.txt
java --enable-native-access=ALL-UNNAMED --add-opens java.base/java.nio=ALL-UNNAMED \
    --sun-misc-unsafe-memory-access=allow -cp "$(cat integration/target/test-classpath.txt)" \
    scripts/RaincloudCompare.java [slug ...]   # no slug: the last manifest

Per-slug status lives in integration/src/test/resources/raincloud/expected-status.csv (ok must pass; gap:<issue> must still fail, so a fix flips the entry in the same change; untriaged runs and reports without failing the build; missing_auth marks a slug that can't even be hydrated without credentials this environment lacks — Kaggle, gated HuggingFace, etc. — distinguishing "blocked on a credential" from plain untriaged). No CI job runs it: the corpus is hydrated and checked locally. Current triage — 398 ok, 0 known gaps, 32 missing_auth (all Kaggle), 0 untriaged: on the v0.3.0 corpus, all 398 datasets that hydrate match their Parquet sibling exactly. Every gap found so far is fixed (#206–#211, #215–#217, #221, #225 RunEnd run-values validity, #226 Sparse null fill / nullable patches, #257 CsvExporter FixedSizeList/List support, #259 conformance-test pipe deadlock, #261 oracle repeated/list column support, plus two bugs the widened oracle then caught: CsvExporter list offsets assumed I32/I64 only (any integer width is wire-legal, mirroring VarBinArray), and ScanIterator could not slice a shared ListArray/FixedSizeListArray spanning several split windows). The v0.3.0 bump found three more, all fixed: vortex.bool bit offsets (#429), CSV export of decimals (#430) and nulls in nullable decimal_byte_parts columns (#431).

A flat segment's ArrayNode tree carries array stats, and Rust's flat layout does not compute them: it keeps the ones the compressor had cached on the array, and each encoding passes on only some (#583). The writer mirrors that for every node of the tree the compressor encodes, the root and the children it cascades into alike:

So a Java-written string column has no bound in its flat segments, as a Rust-written one has none: VortexReader.columnStats() reports no string min/max for either, and a string bound lives only in the zoned layout's stats table. A decimal column's min and max are decoded from the zone table too: columnStats() reports them and zone-map pruning uses them, compared by value at the column's scale.

The nodes below the root follow the same table, plus three shapes of their own: a vortex.sequence carries is_sorted and is_strict_sorted (false for a falling step), the primitive values of ALP, ALP-RD and bit-packed patches carry is_constant = false, and the vortex.bool validity bitmap of a nullable primitive carries is_constant when every row is valid (or none is) and otherwise min, max, sum (the valid-row count) and null_count. NodeStatsInteropIntegrationTest compares every flat root, and the whole tree of the numeric columns, against vortex-jni. The string columns' children (FSST lengths and codes, whose values depend on the trained symbol table) are not covered, nor is the zone table of a column whose few bounds Rust's trained table compresses and ours does not. Text is vortex.varbinview unless a scheme beats its size (vortex.varbin is a binary-only scheme, as in Rust), and that node carries is_constant, null_count and the size of its views (#583).

Per ADR 0010 and ADR 0012, each decoder falls into one of three shapes:

• Zero-copy — output is a view over the memory-mapped file (or a wrapper over child arrays). No arena allocation, no per-element copy.
• Lazy — output is a LazyXxxArray / ChunkedXxxArray record that holds the encoded child plus the transform parameters. Per-row getXxx(i) applies the transform on demand. No output buffer is allocated unless a caller explicitly materializes via Array.materialize(arena).
• Materialized — output is a buffer allocated from ctx.arena() populated during decode(). Required for decompression-style encodings (Bitpacked, Pco, Zstd, etc.) where reading element i would require decoding a window.

Decompression-style encodings (Bitpacked / Pco / Zstd / Delta) stay Materialized by design — element-at-i requires decoding a window, so they must allocate output (ADR 0010). Their output can itself be wrapped in a 1:1 lazy transform (e.g. ALP over Bitpacked produces LazyAlp(MaterializedXxx)). Fsst is the one exception: its per-row code range is independent of every other row, so it stays Lazy instead (ADR 0026).

A scan filtered on a vortex.dict layout column skips the chunks of a dictionary none of whose values satisfies the filter, when a null row cannot satisfy it either, deciding from the values without reading a code. Rust evaluates a dictionary filter over the values too, but always reads the codes: results are the same, but a corrupt codes buffer under such a filter goes unreported where Rust fails the scan. This is a deliberate divergence, for WHERE col = 'x' with x absent from the dictionary. The row test Compute.matcher(ScanFilter) builds for a string dictionary follows Rust: the filter is evaluated once per dictionary value per scan, and u8/u16 codes are mapped to the selected rows in one pass, Rust's values.take(codes).

Files containing unrecognized encoding IDs throw VortexException by default. The message names which edition the id belongs to, or says the id is unknown to every edition if it belongs to none. Opt in to passthrough mode to read such files without failing:

ReadRegistry registry = ReadRegistry.builder()
        .registerDefaults()
        .allowUnknown()
        .build();
try (VortexReader vf = VortexReader.open(path, registry)) {
    // columns with unknown encodings are returned as UnknownArray
}

Frozen, named, additive sets of encoding IDs, each carrying a forever read-compatibility guarantee once frozen (ADR 0023) — a write-time/read-time policy, not part of the wire format. WriteOptions.defaults() targets the latest frozen core edition; see the reference doc for the writer/reader integration.

Mirrors Rust's vortex-edition declarations at the pinned vortex-jni release (0.93.0); EditionCatalogParityIntegrationTest fails the build on drift. core editions are frozen with a forever read-compatibility guarantee; preview holds opt-in components not yet adopted by core; zstd is declared by Rust's vortex-zstd plugin crate rather than the core declarations, and is never enabled by default. Zone maps follow the edition too. A write targeting core2026.08.0 or later (the default) emits Rust's vortex.zoned with fixed 8192-row zones and the aggregate set vortex-jni writes; earlier targets emit the legacy vortex.stats, one zone per writeChunk batch. vortex-java readers from 0.12.1 read vortex.zoned; before this release they placed a dictionary column's vortex.zoned zones by its code chunks rather than the stride, which can mis-attribute zone stats when the chunk count happens to equal the zone count, so read new files with this release or later.

Per-column encoder choice (WriteOptions.withColumnEncoding) mirrors Rust's WriteStrategyBuilder::with_field_writer with a compressor restricted to chosen schemes, but is narrower: a Rust field writer replaces the column's whole layout strategy, while vortex-java keeps the column's zone map and chunking and only restricts its encodings.

fastlanes.delta and vortex.patched are in no edition: as in Rust, only a write with the guard off (WriteOptions.withoutEditions()) may emit them. Rust editions also gate layouts, extension dtypes and zone-map aggregates; vortex-java's catalog models array encodings only. (Upstream additionally records each edition's minimum Rust Vortex release; that number belongs to a different implementation's release train and isn't surfaced in any vortex-java API.)

Extension dtypes wrap a primitive storage array with a logical-id tag plus optional metadata. The Rust catalog lives in vortex-array/src/extension/; each subdir below names a canonical extension id and its on-disk shape.

Read-side extensions live in io.github.dfa1.vortex.reader.extension (write-side encoders implement ExtensionEncoder in the writer module). Both are registered on their registry, the built-ins by registerDefaults(), as Rust registers an ExtVTable on the session. Each spec extension is a singleton with typed decode methods on the concrete impl — grab the singleton directly:

DType.Extension dtype = (DType.Extension) schema.field(ColumnName.of("birthdays"));
List<LocalDate> values = DateExtensionDecoder.INSTANCE.decodeAll(dtype, chunk.column(ColumnName.of("birthdays")));

End-to-end round-trip — write a List<LocalDate>, read it back:

var schema = DType.structBuilder()
        .field(ColumnName.of("birthdays"), DateExtensionDecoder.INSTANCE.dtype(false))
        .build();
writer.writeChunk(c -> c.put(ColumnName.of("birthdays"), dates));              // Collection auto-routed

try (var iter = reader.scan(ScanOptions.all());
     Chunk chunk = iter.next()) {
    List<LocalDate> back = chunk.as(ColumnName.of("birthdays"), LocalDate.class);
}

Chunk.as(name, Class) decodes through the ExtensionDecoder registered for the column's id (LocalDate ↔ vortex.date, LocalTime ↔ vortex.time, Instant ↔ vortex.timestamp, UUID ↔ vortex.uuid, plus any registered third-party type). Both sides run the registered validate(DType.Extension) — Rust's validate_dtype — on every extension dtype, at VortexWriter.create and at open, and on the dtypes embedded in Variant metadata. The four built-ins apply Rust's deserialize_metadata + validate_dtype rules (ExtensionId.WellKnown#validate): vortex.date Days → I32, Milliseconds → I64; vortex.time s/ms → I32, μs/ns → I64; vortex.timestamp metadata ≥ 3 bytes with a UTF-8 zone and I64 storage; vortex.uuid 0–1 metadata bytes (a known version) over FixedSizeList(non-null U8, 16). An extension dtype must carry its metadata field, empty when there is none, as Rust requires; the writer always emits it. As in Rust, the metadata is parsed when the file opens (a malformed one refuses the file) and kept typed; a timestamp's zone is kept as the string the file stores, so a zone this JDK's database does not know still opens and round-trips, and only TimestampMetadata#zone() parses it; an id nothing parses keeps its bytes (OpaqueMetadata, Rust's ForeignExtMetadata).

ExtensionId is a sealed WellKnown/Custom type like EncodingId: the spec ids (VORTEX_DATE, VORTEX_TIME, VORTEX_TIMESTAMP, VORTEX_UUID) plus any third-party id. The ids the vortex-jni session registers by default are ExtensionId.WellKnown: besides the four above, vortex.json and the vortex.st.* geometries (wkb, point, linestring, polygon, multipoint, multilinestring, multipolygon, box). A file with one of them opens without a decoder and the column reads as its storage — JSON text as Utf8, WKB as Binary, geometries as GeoArrow coordinate structs and lists — validated with Rust's storage rules (the spatial CRS metadata is not decoded). Java has no JSON or geometry domain type, so Chunk.as refuses them.

As in Rust, opening a file with any other extension id that has no registered decoder fails (VortexException; Rust: "No such DType extension ID"), unless ReadRegistry.Builder#allowUnknown() is set — then the id round-trips verbatim through DType.Extension's raw String field and the column reads as its storage array, Rust's ForeignExtDType. vortex-jni 0.93.0 exposes no allow_unknown, so it cannot read a file with a custom extension id at all. A blank id, or one with a control character, fails the read with VortexException (Rust accepts any string).

TimeUnit (see extension/datetime/unit.rs) encodes precision in the first metadata byte:

For unsupported extension ids the inspector falls back to a placeholder cell (<GenericArray ext<vortex.X>>); the underlying storage array still decodes correctly via the primitive accessors, callers just have to format the value themselves.

Note: the oracle round-trip suite is pinned to v0.88.0/ (current Rust release). The bucket reuses identical fixture file names across versions but rewrites the bytes, so the local cache is version-keyed; bump the RustFixtures.VERSION constant in the integration tests and refresh this section when a newer set is published.

Cross-language round-trips tested against Rust-written fixture files hosted at s3://vortex-compat-fixtures/v0.88.0/arrays/.