GeezerVison is showing github.com — Ports to aarch64 and ppc64le systems with OS pages 65536 bytes big by Karm · Pull Request #165 · lmdbjava/lmdbjava
Conversation
Karm
commented
Sep 25, 2020
•
edited
Loading
Uh oh!
There was an error while loading. Please reload this page.
Hello,
This PR surpasses #161.
This PR depends on lmdbjava/native#9
Notable changes:
- PAGE_SIZE env and sys property configurable at runtime
- allocations aligned to whole pages
- size of DBs chnages according to what is needed when big pages are in use
Notable documentation these changes are based on:
* https://github.com/LMDB/lmdb/blob/LMDB_0.9.24/libraries/liblmdb/mdb.c#L498:L514
* https://github.com/LMDB/lmdb/blob/LMDB_0.9.24/libraries/liblmdb/lmdb.h#L813:L845
How it was tested
git clone https://github.com/Karm/native.git
pushd native
git checkout secondaryArches
git clone https://github.com/LMDB/lmdb.git
pushd lmdb
git checkout LMDB_0.9.24
popd
mvn clean install
popd
git clone https://github.com/Karm/lmdbjava.git
pushd lmdbjava
git checkout secondaryArches
export PAGE_SIZE=$(getconf PAGESIZE)
mvn clean install RHEL 8 aarch64, 4 CPUs, 6G RAM
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.lmdbjava.KeyRangeTest
[INFO] Tests run: 19, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.14 s - in org.lmdbjava.KeyRangeTest
[INFO] Running org.lmdbjava.CursorParamTest
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.002 s - in org.lmdbjava.CursorParamTest
[INFO] Running org.lmdbjava.TxnTest
[INFO] Tests run: 21, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.086 s - in org.lmdbjava.TxnTest
[INFO] Running org.lmdbjava.ResultCodeMapperTest
[INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.006 s - in org.lmdbjava.ResultCodeMapperTest
[INFO] Running org.lmdbjava.LibraryTest
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.013 s - in org.lmdbjava.LibraryTest
[INFO] Running org.lmdbjava.MaskedFlagTest
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.018 s - in org.lmdbjava.MaskedFlagTest
[INFO] Running org.lmdbjava.DbiTest
[INFO] Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 35.873 s - in org.lmdbjava.DbiTest
[INFO] Running org.lmdbjava.CursorIterableTest
[INFO] Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.152 s - in org.lmdbjava.CursorIterableTest
[INFO] Running org.lmdbjava.MetaTest
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 s - in org.lmdbjava.MetaTest
[INFO] Running org.lmdbjava.EnvTest
[INFO] Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.494 s - in org.lmdbjava.EnvTest
[INFO] Running org.lmdbjava.ComparatorTest
[INFO] Tests run: 21, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.049 s - in org.lmdbjava.ComparatorTest
[INFO] Running org.lmdbjava.VerifierTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.029 s - in org.lmdbjava.VerifierTest
[INFO] Running org.lmdbjava.CursorTest
[INFO] Tests run: 15, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.06 s - in org.lmdbjava.CursorTest
[INFO] Running org.lmdbjava.ByteBufferProxyTest
[INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 s - in org.lmdbjava.ByteBufferProxyTest
[INFO] Running org.lmdbjava.TutorialTest
[INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.118 s - in org.lmdbjava.TutorialTest
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 187, Failures: 0, Errors: 0, Skipped: 0
RHEL 8 ppc64le, 50 CPUs, 252G RAM
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.lmdbjava.KeyRangeTest
[INFO] Tests run: 19, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.209 s - in org.lmdbjava.KeyRangeTest
[INFO] Running org.lmdbjava.CursorParamTest
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.275 s - in org.lmdbjava.CursorParamTest
[INFO] Running org.lmdbjava.TxnTest
[INFO] Tests run: 21, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.331 s - in org.lmdbjava.TxnTest
[INFO] Running org.lmdbjava.ResultCodeMapperTest
[INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.015 s - in org.lmdbjava.ResultCodeMapperTest
[INFO] Running org.lmdbjava.LibraryTest
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 s - in org.lmdbjava.LibraryTest
[INFO] Running org.lmdbjava.MaskedFlagTest
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 s - in org.lmdbjava.MaskedFlagTest
[INFO] Running org.lmdbjava.DbiTest
[INFO] Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 926.308 s - in org.lmdbjava.DbiTest
[INFO] Running org.lmdbjava.CursorIterableTest
[INFO] Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.52 s - in org.lmdbjava.CursorIterableTest
[INFO] Running org.lmdbjava.MetaTest
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 s - in org.lmdbjava.MetaTest
[INFO] Running org.lmdbjava.EnvTest
[INFO] Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 132.677 s - in org.lmdbjava.EnvTest
[INFO] Running org.lmdbjava.ComparatorTest
[INFO] Tests run: 21, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.022 s - in org.lmdbjava.ComparatorTest
[INFO] Running org.lmdbjava.VerifierTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.294 s - in org.lmdbjava.VerifierTest
[INFO] Running org.lmdbjava.CursorTest
[INFO] Tests run: 15, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.511 s - in org.lmdbjava.CursorTest
[INFO] Running org.lmdbjava.ByteBufferProxyTest
[INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.03 s - in org.lmdbjava.ByteBufferProxyTest
[INFO] Running org.lmdbjava.TutorialTest
[INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.649 s - in org.lmdbjava.TutorialTest
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 187, Failures: 0, Errors: 0, Skipped: 0
Fedora, amd64, 8 CPUs, 32G RAM
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.lmdbjava.ResultCodeMapperTest
[INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.113 s - in org.lmdbjava.ResultCodeMapperTest
[INFO] Running org.lmdbjava.TxnTest
[INFO] Tests run: 21, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.463 s - in org.lmdbjava.TxnTest
[INFO] Running org.lmdbjava.LibraryTest
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 s - in org.lmdbjava.LibraryTest
[INFO] Running org.lmdbjava.MetaTest
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 s - in org.lmdbjava.MetaTest
[INFO] Running org.lmdbjava.EnvTest
[INFO] Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.129 s - in org.lmdbjava.EnvTest
[INFO] Running org.lmdbjava.CursorIterableTest
[INFO] Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.036 s - in org.lmdbjava.CursorIterableTest
[INFO] Running org.lmdbjava.VerifierTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.002 s - in org.lmdbjava.VerifierTest
[INFO] Running org.lmdbjava.MaskedFlagTest
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 s - in org.lmdbjava.MaskedFlagTest
[INFO] Running org.lmdbjava.TutorialTest
[INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.026 s - in org.lmdbjava.TutorialTest
[INFO] Running org.lmdbjava.KeyRangeTest
[INFO] Tests run: 19, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.009 s - in org.lmdbjava.KeyRangeTest
[INFO] Running org.lmdbjava.CursorParamTest
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.219 s - in org.lmdbjava.CursorParamTest
[INFO] Running org.lmdbjava.ComparatorTest
[INFO] Tests run: 21, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.019 s - in org.lmdbjava.ComparatorTest
[INFO] Running org.lmdbjava.ByteBufferProxyTest
[INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 s - in org.lmdbjava.ByteBufferProxyTest
[INFO] Running org.lmdbjava.CursorTest
[INFO] Tests run: 15, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.014 s - in org.lmdbjava.CursorTest
[INFO] Running org.lmdbjava.DbiTest
[INFO] Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.768 s - in org.lmdbjava.DbiTest
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 187, Failures: 0, Errors: 0, Skipped: 0
Note about ppc64le
As you can see, Dbi test takes a very long time on the, ironically, most powerful ppc64le box, see:
[INFO] Running org.lmdbjava.DbiTest
[INFO] Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 926.308 s - in org.lmdbjava.DbiTest
I do not know why it takes so much time, but it happened more or less in the same way all three times I tried.
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
* Added more distinguished names for x86_64, aarch64, ppc64le and unified arch64 for all three linuxes
* Renamed unifying x86_64 varable to make spotbugs happy
- Page size is configurable via env and sys prop PAGE_SIZE
- Allocation happens page by page, not by an arbitrary chunk
Merging #165 into master will decrease coverage by 0.52%.
The diff coverage is 53.57%.
@@ Coverage Diff @@
## master #165 +/- ##
============================================
- Coverage 90.93% 90.40% -0.53%
- Complexity 383 385 +2
============================================
Files 30 30
Lines 1346 1366 +20
Branches 118 124 +6
============================================
+ Hits 1224 1235 +11
- Misses 58 63 +5
- Partials 64 68 +4
Continue to review full report at Codecov.
Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 26d76be...dacfdc8. Read the comment docs.
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
|
This pull request introduces 1 alert when merging dacfdc8 into 26d76be - view on LGTM.com
new alerts:
- 1 for Result of multiplication cast to wider type
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
?
- private long mapSize = 1_024 * 1_024;
+ // On a system with pages 4096 bytes big, this translates to default max map size of 4 MB
+ private long mapSize = PAGE_SIZE * 1_024;
=> NEWPotential overflow in int multiplication before it is converted to long by use in an assignment context.
It is correct, but not deserved.
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
It is correct, but not deserved.
There's more information in the LGTM "Result of multiplication cast to wider type" article. Try making PAGE_SIZE a long.
Yup. I understand that, and long is direct fix. But considering how the page_size is used in java, jni and lmdb... I'm quite unable to judge to consequences of the retypeing.
On contrary, we can safely assume, that PAGE_SIZE * 1_024 will remain in the int range
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
|
As Env.getPageSize() guarantees the page size will never exceed 32768, I agree the mapSize maximum of 33,554,432 (32786 * 1024) is well within the Java int range as you observed.
The Env.mapSize field is a Java long for two reasons:
- Its value is eventually passed to the LMDB function
mdb_env_set_mapsize(MDB_env *env, size_t size) (API docs). The size_t on a 64-bit machine is of course larger than a Java int.
- Users need the ability to specify they require a larger map via
Env.Builder.setMapSize(long mapSize). Otherwise their LMDB maps would be limited to 2,147,483,647 bytes (ie about 2 Gigabytes).
Hope this clears things up. I see no issue with performing the cast.
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
|
Hello ! Any activity ongoing on this one ? Just waiting on the cast ?
Thanks for your time :3
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
|
I'm afraid nothing si happening here:(
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
|
And afaik should be merged :)
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
|
Can this be rebased and then merged please?
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
|
As per #217 a Zig-based cross-compilation process has now been added to LmdbJava and deprecates the old LmdbJava Native project.
I would welcome an update to this PR or a new PR that provides or improves any cross-platform support.
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
|
The lmdb library states that the map size should be a multiple of the page size.
The failing tests are examples which succeed when tested on systems with conventional page sizes but fail with larger page sizes.
The two options are a) massage the requested map size to a "proper" size in lmdbjava or b) adopt the same view as lmdb and document that it is the responsibility of client to choose a map size and simply alter tests to request a larger map size.
See https://github.com/LMDB/lmdb/blob/LMDB_0.9.24/libraries/liblmdb/lmdb.h#L813:L845 referenced in above PR.
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
|
I've taken the view that if LMDB feels setting the right map size is the job of the client, then lmdbjava should do the same.
As such, I've created a minimal change PR #218 to make the tests run on architectures with a different page size.
|
bp-alex reacted with thumbs up emoji
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
|
Closing this ticket as #218 has added the required functionality and has been merged.
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Assignees
No one assigned
Milestone
No milestone
Development
Successfully merging this pull request may close these issues.
Uh oh!
There was an error while loading. Please reload this page.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
|