GeezerVison is showing github.com — Seeing records that are malformed/corrupted after write · Issue #207 · lmdbjava/lmdbjava
Seeing records that are malformed/corrupted after write #207DescriptionHey, we have a pretty simple lmdb setup - no env flags, all single-threaded, no nested transactions. This is running on many hosts which receive the same change streams and therefore make identical LMDB transactions. Each host will make at most ~400 write transactions per minute, and across the entire fleet, there are maybe like 100k transactions total per minute (just giving an idea of how rare this is, I've seen it happen about once per month at this volume). Transaction size is variable, could include anywhere from 1-1000 operations. But given that every host makes the same transactions, and seeing nothing out of the ordinary in the logs, it seems like there is some race condition or other intermittent bug with lmdbjava/our implementation/the OS. We've also confirmed that this doesn't match any existing bug with the C library.
The malformed records we see contain garbage data replacing the bytes from left to right:
Example normal record (key, value):
(b'\x1akey-prefix-129632d9951a047d9\x1akey-prefix-8302eb7207a2c859d', b'\x01\x00\x00\x00\x00\x00\x00\x00\x00')
Example malformed record:
(b'\x80\xe1.\xad\xc6\x7f\x00\x00ix-60cf8260ecfb1741f\x1akey-prefix-38357a98a4\xf0\x00\x00\x00\x00\x00', b'\xe0\xdd/\xad\xc6\x7f\x00\x00\x00')
The presence of 7f seems to indicate these are virtual memory addresses.
I've also run the Verifier in our env for 1 hour for 61855873 rows and no errors. The issue hasn't been able to be reproduced.
Any pointers on where else to look? Activitybp-alex commented on Feb 2, 2023 Are the errors always showing up on the same host(s)? Could it possibly be hardware? Is there ECC memory? What is the underlying storage (SSD or magnetic, file system)? Is this running in a VM or on bare metal? Are all hosts identical in terms of software versions (operating system, JVM)? Are there different reads occurring on each host (eg are the hosts processing different incoming requests or similar)?
It's an interesting problem and the infrequency it occurs must be a frustration as you can never be certain if you have reproduced it or corrected it. If I was faced with this I'd be producing a CRC of the key and value bytes and then putting that CRC at the start of the value. I'd then always verify the CRC on each read operation given it's unlikely to be an appreciable cost and you can guarantee your data is free of corruption. I'd also have a configuration setting that I could enable to perform a verification read after every write, therefore ensuring the entry was stored against the expected key and the CRC was valid. Your workload is pretty light (< 10 writes per second) so you're unlikely to notice the overhead and it would offer a lot of confidence given the extremely intermittent nature of this issue.
Depending on your change stream and fleet coordination design you might be able to keep a cumulative CRC of your write CRCs since a particular checkpoint (eg start of an hour as per the change stream-reported event timestamp). Then at the end of each hour your nodes could write their final CRC for that hour into a distributed data structure and check to see if it matches a majority of other reporters. If a node detects its CRC is different from the majority, it would log an error and abort. You could also potentially keep a snapshot of the database after each successful hour, giving you a replay point to see if another CRC variance occurs or it passes on a second attempt. In any event at least you will have a way of detecting errors relatively quickly and protecting your data integrity. As an aside I have previously used Atomix for non-persistent shared data structures but there are plenty of alternatives.
Hope some of these suggestions help. 2018ik commented on Feb 3, 2023 ContributorAuthor More actions It shows up on random hosts - when it is detected, we replace the entire host (they are virtualized). There is ECC memory, the underlying storage is SSD, same operating system, and the exact same writes occur on all hosts, however there are different reads. Perhaps it is worth mentioning that there is a separate process which makes read-only transactions, though we don't see any appreciable correlation between the read TPS of this process and the frequency of corruption.
We have been testing out a similar verification read after every write as you have suggested, so we should be mostly fine on data integrity. However, this kind of forces us to implement a similar check for all our future applications of lmdb, so finding out why this is happening is of course perferable. It's worth noting that internally, there is another completely independent team who also uses lmdbjava that has seen the exact same issue (corrupt bytes, often ending in 7f, very rare, unsure about their hardware setup right now).
One thing I forgot to mention is that the malformed record is out-of-sort with the rest of the sub db, which causes multiple existing records to fail to be found with GET, not just the corrupt one. In my testing it seems to be only keys after the corrupt one (rest of the page, maybe?). So knowing this, if the corruption occurred somewhere in lmdbjava/jnr-ffi which tried to write a bad value, I would think that it would be written successfully to lmdb, but also sorted correctly. However this is not the case here.
Lastly I have not confirmed firsthand, but the other team has been able to perform this verification with success:
- open txn
- put a key
- attempt read for same key
- if it fails, abort the transaction and retry everything again
And it will succeed the second time. This seems to indicate that the corruption is happening in-memory, as the transaction has not been committed/flushed to disk? Appreciate the help. bp-alex commented on Feb 4, 2023 I just released LmdbJava 0.8.3 and this includes an update to the underlying LMDB system library. Would you be able to give the new version a try? apmatthei commented on Feb 7, 2023 @benalexau Are there specific changes in the updates to LmdbJava 0.8.3 or in the update to LMDB native library 0.9.29 that could prevent record corruption @2018ik indicated? bp-alex commented on Feb 7, 2023
Are there specific changes in the updates to LmdbJava 0.8.3?
The previously-linked LmdbJava Change Log identifies the changes in LmdbJava 0.8.3. There are no changes that would impact corruption.
or in the update to LMDB native library 0.9.29 that could prevent record corruption
LmdbJava 0.8.2 included LMDB native library version 0.9.24. The change that updated the included version to 0.9.29 was included in commits b9b331b and lmdbjava/native@a442be3.
To review what changed in the LMDB native library:
Issue ITS 9155 fixes a spill page not being freed. Likewise issue ITS 9376 originates in Mithril-mine/libmdbx#121 and there is the remark:
This seems to be another bug inherited from LMDB. The difference is that libmdbx returns an error, whereas LMDB crashes, returns invalid data, or damages the database.
Whether @2018ik is impacted by ITS 9376 is unclear as I do not know whether they are using DUPSORT for example.
In any event LMDB 0.9.24 was released in 2019 and LmdbJava 0.8.3 now includes the latest release. Given LmdbJava observes semantic versioning principles it is definitely worth trying a low-risk upgrade from 0.8.2 to 0.8.3. 2018ik commented on Feb 8, 2023 ContributorAuthor More actions I've observed the issue on both dupsort and non-dupsort tables so I had ruled out that change. But yea, upgrading is in the works. Only time will tell if it is actually fixed, though. 2018ik commented on Mar 16, 2023 ContributorAuthor More actions Uninteresting update, rather than upgrading, we switched to ByteBuf w/ its buffer pool instead of ByteBuffer and the bug has never occurred since. We think this didn't actually fix the problem, but just made it much rarer. Since we're allocating way fewer buffers, probably less of a chance for one of them to get GC'd or deleted by something else. Still looking into the root cause though. simonis commented on Mar 23, 2023 @benalexau, I'm not familiar with the exact code of this project, but managing native resources (e.g. native memory) referenced from Java objects is not trivial. You must make sure that Java objects like e.g. DirectByteBuffer which hold references to native memory are kept alive until all the native code which uses this memory has finished execution.
E.g. in the following code example:
{
ByteBuffer direct = ByteBuffer.allocateDirect(1000);
long address = MyUnsafeClassWhichExtractsDirectByteBuffersNativeAddress(direct);
nativeCall(address);
}
The JVM can garbage collect direct already before the call to nativeCall() because no further reference to direct exists any more at that point (it is a common misconception that local variables are kept alive until control flow exits the scope where they have been created). In the case of a DirectByteBuffer object, this can also trigger the invocation the corresponding Cleaner and free up the buffers native memory before or while it is used in your native code.
In order to prevent such sitations, JKD 9 added the concept of a reachabilityFence. With this, you can rewrite the above code as:
{
ByteBuffer direct = ByteBuffer.allocateDirect(1000);
long address = MyUnsafeCodeWhichExtractsDirectByteBuffersNativeAddress(direct);
nativeCall(address);
Reference.reachabilityFence(direct);
}
This will make sure that direct will not be garbage collected until you return from you native call.
I couldn't find any reference to Reference.reachabilityFence() in this code base, but maybe you have other means to ensure that objects are kept alive in which case you're free to ignore these comments :) 2018ik commented on Mar 23, 2023 ContributorAuthor More actions @simonis Interesting, so the nativeCall is performed within a function where the direct buffer is passed as a parameter https://github.com/lmdbjava/lmdbjava/blob/master/src/main/java/org/lmdbjava/Dbi.java#L365-L390.
So how LmdbJava is used would be:
putBuffer() {
ByteBuffer direct = ByteBuffer.allocateDirect(1000);
lmdbJavaPut(direct)
}
lmdbJavaPut(direct) {
long address = MyUnsafeCodeWhichExtractsDirectByteBuffersNativeAddress(direct);
nativeCall(address);
// other stuff that does not use direct
}
putBuffer()
I guess the implicaition here is that direct is considered unreachable during the native call even though it is in scope as a function parameter? Our various "fixes" for the issue all coincidentally involved using direct in some way after lmdbJavaPut, so that makes sense. Is there a way to programmatically tell if an object is unreachable? simonis commented on Mar 23, 2023 Well, an object becomes unreachable right after the last reference to it got cleared :) In fact it doesn't even have to be explicitly overwritten or cleared. The JVM keeps a so called "OopMap" for each Safepoint, i.e. a data structure which saves the references of all objects in the corresponding scope which are "alive" (i.e. reachable) at that program location. A Safepoint is a program location where the JVM can stop a thread in order to perform e.g. a GC. Native calls are considered to be Safepoints and a GC can run asynchronously while the corresponding native code is executing (the thread will be suspended when it returns from native and the GC is still running).
Especially in highly optimized JIT code objects can become unreachable very eagerly because of inlining, code movement and dead code elimination. E.g. in your code example above it probably makes no difference that you pass direct as a function parameter because in the end lmdbJavaPut() and putBuffer() will both be inlined into putBuffer().
The problem is that the VM has no way to detect if a primitive field of your otherwise dead object is still referencing an otherwise valid "resource". And this problem is by no means limited to native memory references (although that usually makes the problem worse :). The API documentation of Reference::reachabilityFence() has a pure Java example of this problem.
Using Reference::reachabilityFence() usually has zero costs so I'd really advice to use it explicitly instead of relying on the object being kept alive implicitly by a later usage.
Notice that for the current example, it is really, really unlikely to run into a problem, because this requires not only a GC which frees up the native resource but also another native allocation which returns the same memory region and another thread which overwrites the original data in that region. And all this has to happen while your nativeCall() is working on address. 2018ik commented on Mar 23, 2023 ContributorAuthor More actions Cool, so to consistently reproduce the issue, I added a System.gc() call to L377 of DBI (between these two https://github.com/lmdbjava/lmdbjava/blob/master/src/main/java/org/lmdbjava/Dbi.java#L377-L378) and ran this script:
package org.lmdbjava;
import java.io.File;
import java.nio.ByteBuffer;
import static java.nio.ByteBuffer.allocateDirect;
import static java.nio.charset.StandardCharsets.UTF_8;
import static org.lmdbjava.DbiFlags.MDB_CREATE;
import static org.lmdbjava.Env.create;
public class Main {
static void test(Dbi<ByteBuffer> db, Txn<ByteBuffer> txn, int i) {
ByteBuffer key = allocateDirect(24);
ByteBuffer val = allocateDirect(24);
key.put(String.format("Uncorruptedkey%d", i).getBytes(UTF_8)).flip();
val.put(String.format("Uncorruptedval%d", i).getBytes(UTF_8)).flip();
db.put(txn, key, val);
}
public static void main(String[] args) {
final File path = new File("/tmp");
final Env<ByteBuffer> env = create()
.setMapSize(2_085_760_999)
.setMaxDbs(1)
.open(path);
final Dbi<ByteBuffer> db = env.openDbi("testdb", MDB_CREATE);
// Write a bunch of stuff
try (Txn<ByteBuffer> txn = env.txnWrite()) {
for(int i = 0; i < 1000; i ++) {
test(db, txn, i);
}
txn.commit();
}
// Print corrupt keys
try (Txn<ByteBuffer> txn = env.txnRead()) {
try (Cursor<ByteBuffer> c = db.openCursor(txn)) {
if (c.first()) {
do {
byte[] rkey = new byte[c.key().remaining()];
c.key().get(rkey);
byte[] rval = new byte[c.val().remaining()];
c.val().get(rval);
String skey = new String(rkey, UTF_8);
String sval = new String(rval, UTF_8);
if (!skey.startsWith("Uncorrupt")) {
System.out.println("Key " + skey);
}
if (!sval.startsWith("Uncorrupt")) {
System.out.println("Val " + sval);
}
} while (c.next());
}
}
}
env.close();
}
}
The output would be something like
Key ����������dkey661
Val ���������dval661
Key ����������dkey670
Val `���������dval670
Key ���������dkey778
Val ���������dval778
Key ���������dkey680
Val ����������dval680
Key @⨇����4�dkey970
Val `樇����\�dval970
Val `�������*�dval657
Val �������>�dval660
Val `�������>�dval662
Val �l������Z�dval664
Val �_������A�dval665
Val ���������dval668
Key �$��������dkey744
Val @:��������dval744
Val ��������Z�dval893
Val @������� �dval923
Key ����������dkey887
Val ��������F�dval887
Key �뮇������dkey663
Val `�����dval663
Key �뮇������dkey669
Val `�����dval669
Key �뮇������dkey672
Val `�����dval672
Key ��������i�dkey721
Val ����������dval721
Key ����������dkey656
Val ����������dval656
Key ������dkey686
Val ������dval686
Key �8��������dkey719
Val 5��������dval719
Key �8��������dkey736
Val @8��������dval736
Key �ڨ�������dkey659
Val �㨇����x�dval659
2018ik commented on Mar 23, 2023 ContributorAuthor More actions @benalexau @simonis That's my proposed fix #214 ^ Based on the changelog, we are trying to make this buildable for JDK 8 which I think reachabilityFence is not supported in? By creating that arbitrary reference in keyVal it ensures the buffer remains reachable through all LmdbJava calls, whether cursor/txn/get/put/etc. 2018ik commented on Apr 20, 2023 ContributorAuthor More actions bp-alex commented on Apr 21, 2023 @2018ik did you wish to amend the PR and give it a test? added 3 commits that reference this issue on Apr 21, 2023This issue will close once commit fb7dd1b is merged into the 'master' branch.fb7dd1b This issue will close once commit d01270b is merged into the 'master' branch.d01270b This issue will close once commit 0c636f1 is merged into the 'master' branch.0c636f1 2018ik commented on Apr 21, 2023 ContributorAuthor More actions Amended it. If you remove the reachability fence in put the unit test will fail. This comes with the caveat of a breaking change if a user is for some reason synchronizing their buffer before passing it to lmdbjava
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|