GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — Iterator/stream: fix reverse-prefix bug + Spliterator SORTED contract, add coverage by bernardladenthin · Pull Request #292 · lmdbjava/lmdbjava

Conversation

@bernardladenthin

@bernardladenthin bernardladenthin commented Jul 24, 2026

Copy link
Copy Markdown

Follow-up work stacked on the iterator-performance branch (targets that branch, not master). It fixes two correctness/contract issues in the new iterator/stream code, adds the missing unit/integration coverage for the gh-269 helpers, and tidies a few related items. All changes are additive or self-contained; the design decisions on the branch (see "Deliberately left open" below) are untouched.

Fixes

1. Reverse prefix iteration silently drops rows when the prefix ends in 0xFF

BufferProxy.incrementLeastSignificantByte computed the "prefix successor" by incrementing the least-significant non-0xFF byte but keeping the trailing 0xFF bytes — e.g. {0x01,0xFF} → {0x02,0xFF} instead of the tight {0x02}.

In LmdbPrefixReversedIterator/LmdbPrefixReversedSpliterator the successor is used to MDB_SET_RANGE past the prefix range and step back. When it over-shoots, an unrelated higher key that sorts between the last prefix match and the over-shot successor is landed on, the containsPrefix check fails, and iteration wrongly yields nothing.

  • Reproduced by LmdbPrefixReversedSuccessorTest (keys {0x01,0xFF}, {0x01,0xFF,0x05}, {0x02,0x00}; reverse-prefix {0x01,0xFF} returned [] before the fix).
  • Fixed by truncating trailing 0xFF (tight successor) across all four proxies' big-endian byte-string branch (ByteArray, ByteBuffer, DirectBuffer, Netty). The little-endian/integer-key branch is left as-is — prefix scans operate on big-endian byte strings.

2. LmdbStream Spliterator advertised SORTED with a null comparator

createEntryComparator/createReversedEntryComparator returned null (real body commented out), yet every spliterator reported ORDERED | DISTINCT | SORTED | NONNULL and getComparator() returned that null. A SORTED spliterator with a null comparator implies natural ordering, but KeyVal is not Comparable.

  • Dropped SORTED and DISTINCT (DUPSORT can repeat keys) → ORDERED | NONNULL.
  • getComparator() now throws IllegalStateException, as required for a non-sorted spliterator.
  • Removed the dead entryComparator plumbing (also clears the "useless rangeComparator parameter" scan alerts).
  • Note: a real Comparator<KeyVal<T>> + SORTED isn't safe without first emitting per-element snapshots, because the spliterator emits a single reused KeyVal (collect(toList()) returns one aliased instance for every row). That's a larger design change and is left to the maintainers.

3. KeyRange.getType() returned null for builder-created ranges

The private constructors used by KeyRange.builder() never set type, so getType() returned null and would NPE the legacy CursorIterable path. Builder/empty ranges now derive the equivalent KeyRangeType. Prefix ranges still have no KeyRangeType (documented) as they are only consumed by the new iterators.

Tidy-ups

  • Dbi.stream()/newIterate() javadoc now warns that entries are a single reused KeyVal holder (so collect/sorted/distinct observe aliased entries) and that the stream is ORDERED but not SORTED.
  • Dbi.getNameAsString(Charset): removed a dead try/catch and misleading "assume UTF8" comment (new String(byte[], Charset) never throws).
  • Fixed a stale javadoc link CursorIterable.JavaRangeComparator → JavaRangeComparator.

New/updated tests

  • BufferProxyPrefixTest — containsPrefix + incrementLeastSignificantByte across all four proxies, incl. edge cases (empty/oversized prefix, unsigned bytes, trailing-0xFF truncation, all-0xFF → null).
  • CompareAsIntegerKeysTest — length-mismatch guard + non-4/8-byte lexicographic fallback for the integer-key comparators.
  • KeyRangeBuilderTest — builder/prefix/inclusive accessors + derived KeyRangeType.
  • LmdbStreamCharacteristicsTest — locks in the corrected spliterator contract.
  • DbiIterateApiTest — newIterate(EntryConsumer) overloads, ranged stream, LmdbIterable single-use, getNameAsString, toString.
  • LmdbPrefixReversedSuccessorTest — the reverse-prefix regression above.

Verification

Full test suite (excluding the pre-existing, environment-flaky TestLmdbStreamBenchmark, which fails at Env.open unrelated to these changes): 1687 tests, 0 failures. mvn fmt:check is clean for all touched files. Java 8 source level unchanged.

Deliberately left open (branch decisions, not addressed here)

These came up while reviewing the branch and are for the maintainers, not this PR:

  • Approach approval; renaming the placeholder newIterate(...) and retiring/adapting CursorIterable for API compatibility.
  • Whether to split the bundled FlagSet / DbiBuilder / ByteUnit / The new MDB_UNSIGNEDKEY DbiFlag should probably be the default behaviour #249 / CursorIterable used with BACKWARD_AT_LEAST and BACKWARD_CLOSED is not returning expected ranges of results when using DUPSORT #267 work from the iterator perf change.
  • Rebasing onto current master; squashing WIP history and purging the accidentally-committed native binaries (src/main/resources/org/lmdbjava/*.so|*.dll) that live in branch history.
  • Codecov gate (new Env/Dbi/Cursor methods still thinly covered) and the remaining code-scanning alerts (deprecated Env.openDbi in tests, NumberFormatException in CSV parsing, field shadowing) — some intentionally not touched here as they live in branch-owned test helpers.
  • The KeyVal-snapshot redesign that would let SORTED + a real comparator return.

🤖 Generated with Claude Code

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
bernardladenthin and others added 2 commits July 24, 2026 14:09
@bernardladenthin @claude
Add tests for iterator/stream helpers; fix LmdbStream SORTED contract
716e7f0
Adds unit coverage for lmdbjavagh-269 iterator/stream helpers that had none, and
corrects the LmdbStream spliterator characteristics.

Tests:
- BufferProxyPrefixTest: containsPrefix + incrementLeastSignificantByte across
  all four proxies, with edge cases (empty/oversized prefix, unsigned bytes,
  trailing-0xFF carry, all-0xFF -> null).
- CompareAsIntegerKeysTest: length-mismatch guard and non-4/8-byte lexicographic
  fallback for the integer-key comparators.
- KeyRangeBuilderTest: builder/prefix/start-stop-inclusive accessors; also
  documents that getType() is null for builder()/prefix()-created ranges.
- LmdbStreamCharacteristicsTest: locks in the corrected spliterator contract.

Fix (LmdbStream):
- Drop the false SORTED and DISTINCT characteristics: entries are produced in key
  order but no KeyVal comparator is exposed, and DUPSORT can repeat keys.
- getComparator() now throws IllegalStateException (required for a non-SORTED
  spliterator) instead of returning null.
- Remove the dead entryComparator plumbing (stubbed factories returning null),
  which also clears the unused-parameter warnings.

The fuller alternative (real Comparator<KeyVal<T>> + SORTED) was investigated and
rejected: the spliterator emits a single reused KeyVal instance, so advertising
SORTED would be unsound without first emitting per-element snapshots.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
@bernardladenthin @claude
Fix reverse-prefix successor bug; add API coverage; tidy KeyRange/Dbi
d7c6369
Fix (correctness):
- incrementLeastSignificantByte over-shot the prefix successor when the prefix
  ended in 0xFF: {0x01,0xFF} produced {0x02,0xFF} instead of the tight {0x02},
  keeping the trailing 0xFF. In reverse prefix iteration this seeks past the
  prefix range and steps back onto an unrelated higher key, so iteration wrongly
  yields nothing. Now truncates trailing 0xFF across all four proxies (big-endian
  byte-string branch). LmdbPrefixReversedSuccessorTest reproduces and guards it.

KeyRange:
- Builder/empty ranges now derive the equivalent KeyRangeType, so getType() is no
  longer null (a null type would NPE the legacy CursorIterable path). Prefix
  ranges still have no KeyRangeType (documented).

Dbi:
- Document that stream()/newIterate() emit a single reused KeyVal holder (so
  collect/sorted/distinct observe aliased entries) and that the stream is ORDERED
  but not SORTED.
- Simplify getNameAsString(Charset): drop the dead try/catch and misleading
  "assume UTF8" comment; new String(byte[], Charset) never throws.
- Fix stale javadoc link CursorIterable.JavaRangeComparator -> JavaRangeComparator.

Tests:
- DbiIterateApiTest: newIterate(EntryConsumer) overloads, ranged stream,
  LmdbIterable single-use, getNameAsString, toString.
- BufferProxyPrefixTest / KeyRangeBuilderTest updated for the corrected successor
  truncation and the derived KeyRangeType.

Full suite (excluding the pre-existing flaky TestLmdbStreamBenchmark) green: 1687
tests, 0 failures.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
@bernardladenthin bernardladenthin mentioned this pull request Jul 24, 2026
Improve iterator performance and add better stream support #269
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

1 participant

@bernardladenthin
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.