Description

bp-alex

opened on Oct 26, 2025

Issue body actions

I have finished refactoring LmdbJava Benchmarks to use the latest third party database libraries, Java 25, automatic scaling of workloads to match available system memory, and execution on every commit (including both JMH benchmarking and report production with charts) via GitHub Actions.

We need to decide how we best want to manage benchmarks going forward, particularly given the importance of maintaining performance as part of #259.

The current state of the benchmark project enables us to run benchmarks on a timer basis or indeed on every commit to the main LmdbJava repository. But a proper benchmark requires a more stable environment than a GitHub Action can provide. Plus we don't currently have a mechanism for comparing branches, eg FFM compared with the present Unsafe-based buffers.

Comments and suggestions are invited.

Activity

1. bp-alex

   self-assigned this

   on Oct 26, 2025
2. bp-alex

   added

   enhancementAdditional or new functionalityAdditional or new functionality

   on Oct 26, 2025
3. bp-alex

   added 2 commits that reference this issue on Oct 29, 2025

   Fix major performance regression

   e634b3f

   Replace Zig cross-compilation with native library artifact

   e5b510c
4. stroomdev66 commented on Oct 30, 2025

   

   stroomdev66

   on Oct 30, 2025

   Collaborator

   More actions

   I'm just running the benchmarks now. I agree that they probably need to be run on a stable environment and not GH infrastructure. Unless somebody has a server to spare then I think the only option is for us to run them locally when we produce a new release.

   It would be useful to compare different LmdbJava versions. We might be able to do this by using shadow jar to craft version specific jars without overlapping package names. We'd also need to be careful about where the C lib gets written to avoid a clash.

   I intend to focus more on targeted performance testing of specific areas on LmdbJava. I think this is a separate task to providing benchmarks. I would like to performance test and profile general read and write behaviour in various scenarios. Some testing on buffer proxies, core C wrapper, bounded cursor iterations etc would also be helpful especially when evaluating #259. Preliminary profiling and code change has yielded some minor improvements that will be added to PRs in the coming weeks.
5. bp-alex commented on Oct 31, 2025

   

   bp-alex

   on Oct 31, 2025

   MemberAuthor

   More actions

   LmdbJava Benchmarks has been significantly refactored to provide continuous performance monitoring through two benchmark suites:

   • Library Comparisons: Compares the latest LmdbJava release against other embedded key-value stores (LevelDB, RocksDB, MapDB, MVStore, Xodus, Chronicle Map)
   • Version Regressions: Tracks LmdbJava performance across historical releases and the current master branch

   Benchmark Profiles

   Achieving reliable benchmark results requires large, dedicated machines and many hours of execution time. To balance accuracy with practical constraints, two distinct profiles have been introduced:

   • Smoketest (< 1 hour): Runs daily via GitHub Actions to verify the benchmark infrastructure and publishing pipeline remain operational
   • Benchmark (several hours): Uses multiple warmup iterations, forks, and extended execution times for accurate performance analysis

   Iteration Duration Selection

   To determine the optimal iteration duration, I conducted a quantification exercise (methodology detailed in commit e634b3f) comparing coefficient of variation across different durations:

   Duration	Score (ms)	Error (ms)	CV %	vs 60s Baseline
   10s	45.386	±0.034	0.0757%	2.2× worse
   30s	45.757	±0.037	0.0811%	2.4× worse
   60s	45.026	±0.015	0.0339%	baseline
   120s	45.192	±0.005	0.0110%	3.1× better

   Based on these results, 120-second iterations have been adopted for the benchmark profile, providing 3× better stability than the previous 60-second standard.

   Published Results

   All benchmark results are published to Cloudflare Pages with automatic deployment:

   Type	Profile	URL
   Library	Benchmark	https://libraries-benchmark.lmdbjava.org
   Library	Smoketest	https://libraries-smoketest.lmdbjava.org
   Version	Benchmark	https://versions-benchmark.lmdbjava.org
   Version	Smoketest	https://versions-smoketest.lmdbjava.org

   Additionally, http://lmdbjava.org now redirects to this GitHub project.

   Full benchmark results from a production run will be published to the above URLs within the next few days.
6. bp-alex

   closed this as completedon Oct 31, 2025

Issue actions