GeezerVison is showing github.com — Change back to getLong: why ? · Issue #121 · lmdbjava/lmdbjava
Change back to getLong: why ? #121DescriptionHi to all,
This is more a clarification request than an issue per se:
-
we're currently evaluating lmbdjava and while we didn't experience any problem on OS X and Linux, we had some random and rather frequent crashes on Windows (access violations while retrieving buffers or keys). After investigation, we couldn't really pinpoint anything wrong with our code or discern any general rule as to the crashes' cause (multithreading, memory, etc.). We're using JDK 8 but using other JDKs did not help.
-
we finally managed to reproduce the problem consistently using ByteBufferProxy.PROXY_SAFE, creating a large env size (> 1Gb) and a bit of our code in a JUnit set to 2048m max heap size (letting the default buffer, PROXY_OPTIMAL, makes it much harder to crash, but it crashes eventually -- the same with Agrona buffers or even byte arrays). Our piece of code does not really stress lmdb, nor fills the map or anything extreme.
-
then I saw issue 97 and your change from getLong to getAddress in the buffer proxies, and decided to give it another go. It improved things a lot. Early tests seem to indicate that the problem went away, at least with ByteBufferProxy.PROXY_SAFE.
Which leads to my question:
- since we originally had the problem with
OPTIMAL proxy versions and you reverted back to using getLong for these, the problem is still there unless we write our own proxy. Which we can do, of course, but I wanted to know if there was a specific issue that made you revert your changes?
Thanks! Activitybp-alex commented on Mar 13, 2019 In commit a5ef8b5 I changed the Unsafe usages back to long (not address) because I wanted to make the minimal changes necessary to resolve the bug reported in #97 and my local VM testing had shown the issue related to Pointer usages, not Unsafe usages.
Furthermore I had concerns about retaining the Unsafe use given JavaDocs include the following applicable warning for both getAddress(long) and putAddress(long, long):
If the address is zero, or does not point into a block obtained from {@link #allocateMemory}, the results are undefined.
We do not obtain the memory from Unsafe.allocateMemory(long) but rather LMDB itself manages it. Perhaps the warning is not cause for concern, but given my local testing demonstrated it wasn't necessary to resolve #97 I decided I should remove such usage.
I would be grateful if you could attach a minimal test case that shows the Windows issue though. We've had several unusual and often intermittent Windows bugs over the years, such as byte order boundary conditions, pointer endoing etc. As I do not use Windows at all, I appreciate any assistance with finding bugs and/or reproducing them in a local VM so we can get them resolved. altesse commented on Mar 13, 2019 ContributorAuthor More actions Thanks for your quick response,
I understand your caution. We will give it a try on our end and stress it a bit, just to see.
What I have difficulties to understand is the strong difference of behavior betweenOPTIMAL and UNSAFE, at least on Windows platforms. After all, only the methods to access the buffer fields differ, and nonetheless with our 'crash setup' the former crashes fast and hard, while the latter resists much longer. Barring getAddress, the only settings that improve the situation are either reducing the map size or increase the max heap size, which would hint at a memory / GC pressure problem.
Our setup unfortunately has lots of proprietary code but I'll let you informed of what we find or find a way to reproduce the problem with simple code. bp-alex commented on Mar 13, 2019
while the latter resists much longer
Does this mean that regardless of OPTIMAL, UNSAFE and getAddress it still crashes (eventually) anyway?
the only settings that improve the situation are either reducing the map size or increase the max heap size, which would hint at a memory / GC pressure problem
If you watch LmdbJava in a profiler you will see it does minimal allocations in the hot path. The only objects that are created somewhat regularly are Txns, and even that can be often avoided (depending on your workload characteristics and ACID requirements). So I don't see why LmdbJava would be using enough heap to behave differently with a larger heap size.
Our setup unfortunately has lots of proprietary code but I'll let you informed of what we find or find a way to reproduce the problem with simple code.
I'd encourage you to double-check your application isn't holding onto any of the references being returned by LmdbJava. The pointers are only valid until the Txn ends, or the next LmdbJava operation (eg another Dbi or Cursor call), or an LmdbJava instance is closed (eg Txn, Dbi, Env). You mentioned you had tested successfully on Linux and OS X, so that would suggest the application usage is fine, but is it possible a different code path is being exercised between the platforms?
Have you tried taking the LMDB file produced by your application on Linux or OS X and trying to read it on Windows? In other words, is this failing during read-time operations or only write-time operations? That may help narrow down the code paths to check.
I've also seen Windows bugs in the past where NTFS was not the partition type. For example, it doesn't like "shared folders". So maybe have a look at the partition type you're using, and/or try it on another type of partition.
If you had a go at writing a quick standalone test class, it just needs to roughly approximate what your application is doing in terms of key, value size, order of inserts and order of reads. It doesn't need real data (eg the value can be random bytes of the same size as your typical application's values). It just needs to be similar enough that LMDB would create similar types of pages and rebalance the B+ Tree behind the scenes. If I can reproduce it locally I am very happy to look further into it. altesse commented on Mar 13, 2019 ContributorAuthor More actions
Does this mean that regardless of OPTIMAL, UNSAFE and getAddress it still crashes (eventually) anyway?
With getAddress, I did not crash it yet, except "classic" LMDB crashes where the map is full and somehow LMDB can't even return a "map full" return code. With the previous code with get|putLong in the proxies, the crash would happen eventually, fast and predictable for OPTIMAL, random for UNSAFE.
So I don't see why LmdbJava would be using enough heap to behave differently with a larger heap size.
Yes, we made the same observations. Might be our code.
I'd encourage you to double-check your application isn't holding onto any of the references being returned by LmdbJava. The pointers are only valid until the Txn ends, or the next LmdbJava operation (eg another Dbi or Cursor call), or an LmdbJava instance is closed (eg Txn, Dbi, Env). You mentioned you had tested successfully on Linux and OS X, so that would suggest the application usage is fine, but is it possible a different code path is being exercised between the platforms?
We thought of that too, but did not find anything. I added some code to ensure the map is open whenever a key/value is read/written, I even tried never closing the Env and Dbi, but it still crashed. I'll perform another check, though. There's no code path depending on the platform that I know of, except of course the native LMDB part.
Have you tried taking the LMDB file produced by your application on Linux or OS X and trying to read it on Windows? In other words, is this failing during read-time operations or only write-time operations?
As far as I can remember -- I still have the logs, will look into it -- it's almost always when reading a buffer returned by LMDB, not writing it. More rarely, in unsafe.copyMemory. I did not try to read the maps cross-platform. Clever idea.
If I can reproduce it locally I am very happy to look further into it.
Thanks a lot for your time and suggestions. I'll try and set something up. It may take a while, though. bp-alex commented on Mar 28, 2019 How did you go with this? Can we close the ticket if you're sorted it out? altesse commented on Mar 28, 2019 ContributorAuthor More actions Hi, sorry for the late reply -- I was letting some time for a sufficient number of our project builds to execute, the crashes occurring in some of our JUnit tests. Now that we are at 50+ builds and counting without any crash, I can safely post our conclusions:
So it turns out that:
- The random process crashes in our heavy JUnit tests had nothing to do with LmdbJava itself but, with high probability, rather the way my tests themselves opened and closed the map at each iteration, plus maybe the way I was locking the Env to ensure uniqueness in the JVM. A closer inspection from our lead architect solved the problem. Lesson learned: if the test crashes, maybe it's simply because the test is badly written.
- However, the
get|putLong / get|putAddress problem is real on Windows platforms, and even with the refactor we did on our classes, the tests will crash with Long on PROXY_SAFE. With the growing number of consecutive passed tests we have now, I can conclude that get|putAddress on PROXY_SAFE is a solution and not a mitigation. I have yet to run refactored tests with PROXY_OPTIMAL, which we'd ultimately rather use, but I'll let you know.
So there were two unrelated problems, one on our end -- sorry for that. The ticket can be closed.
Thanks for your time and, as I said, if I can find time, I'll try to set something up for you to reproduce the Windows getLong problem. bp-alex commented on Mar 31, 2019 It sounds like you have a pretty extensive test suite.
If you are happy to submit a PR containing the changes to LmdbJava with confirmation that you have tested this extensively on Windows I am happy to merge them. I just don't want to introduce bugs, as my Windows use is very limited. altesse commented on Apr 3, 2019 ContributorAuthor More actions
It sounds like you have a pretty extensive test suite.
Oh yes we have.
If you are happy to submit a PR containing the changes to LmdbJava with confirmation that you have tested this extensively on Windows I am happy to merge them. I just don't want to introduce bugs, as my Windows use is very limited.
Ok, thanks -- for now we went with PROXY_SAFE and get|putAddress, so, no changes to your code. The root cause of our JUnit crashes has been determined with high confidence to be env_info / env_stat calls after a close().
We'll shortly try PROXY_OPTIMAL and tell you what happens on Windows with get|putLong. Honestly, with what we throw at it, if LmdbJava does not crash, then it simply works. That seems to be the case. bp-alex commented on Apr 4, 2019
for now we went with PROXY_SAFE and get|putAddress, so, no changes to your code.
Just for clarification, you are saying that your extensive test suite passed when using the officially-released (ie unmodified) LmdbJava 0.6.3 with PROXY_SAFE?
We'll shortly try PROXY_OPTIMAL and tell you what happens on Windows with get|putLong.
Look forward to hearing how this goes. altesse commented on Apr 4, 2019 ContributorAuthor More actions
Just for clarification, you are saying that your extensive test suite passed when using the officially-released (ie unmodified) LmdbJava 0.6.3 with PROXY_SAFE?
It's a 0.6.2-SNAPSHOT compiled with a JDK 8 and get|putAddress for PROXY_SAFE, so it's very close. Sooner or later, we're going to upgrade to 0.6.3+ anyway.
Look forward to hearing how this goes.
So far everything works. bp-alex commented on Apr 5, 2019 I just added a Verifier class (see issue #124) that explores boundary conditions with the ByteBuffer buffer type. You may like to copy the contents of the Verifier source into your project or upgrade to the 0.6.4-SNAPSHOT (see the Snapshot Repository) and give it a try with PROXY_OPTIMAL and PROXY_SAFE. It would be good to know you can't crash it with the new Verifier. altesse commented on Apr 5, 2019 ContributorAuthor More actions Ok, thanks a lot! We'll give it a try shortly and let you know.
So far, extensive use with either PROXY_OPTIMAL, PROXY_SAFE or Agrona did not show any problem. altesse commented on Apr 25, 2019 ContributorAuthor More actions Hi, sorry for the long delay -- we were busy with other matters.
Verifier works with PROXY_OPTIMAL and PROXY_SAFE, and still no crash of LmdbJava on Windows! I'd argue that this version at least is rock-solid. We're going to decide soon whether to upgrade to 0.6.3, I'll let you know how this goes. bp-alex commented on May 31, 2019 Can I close this ticket given it all seems good now? I hope you can try it with 0.6.3. altesse commented on May 31, 2019 ContributorAuthor More actions Yes you can, thanks. I'll you informed as soon as we switch to 0.6.3.
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|