Description

fbiricotti

opened on Jan 20, 2022

Issue body actions

The APIs for creating a database (openDbi) don't take a transaction as input parameter (it's generated internally and immediately committed). This code works:

    Dbi<ByteBuffer> dbi = env.openDbi("TEST", new DbiFlags [] { DbiFlags.MDB_CREATE });
    Txn<ByteBuffer> txn = renv.txnWrite();
    dbi.put(txn, key, val);
    txn.commit();

This code DOES NOT:

    Txn<ByteBuffer> txn = repoEnv.txnWrite();
    Dbi<ByteBuffer> dbi = repoEnv.openDbi("TEST", new DbiFlags [] { DbiFlags.MDB_CREATE });
    dbi.put(txn, key, val);
    txn.commit();

by throwing this exception:

Exception in thread "main" org.lmdbjava.LmdbNativeException$ConstantDerivedException: Platform constant error code: EINVAL Invalid argument (22)
at org.lmdbjava.ResultCodeMapper.checkRc(ResultCodeMapper.java:114)
at org.lmdbjava.Dbi.put(Dbi.java:411)
at test.TestMain.main(TestMain.java:34)

This seems to be a blocking issue in a complex application where databases may be require to be created on-demand while a write transaction is already up and running. Or there is an alternative ? Please note that the native LMDB API for opening/creating a database do require a transaction., why not exposing it as parameter?

Activity

1. bp-alex commented on Jan 20, 2022

   

   bp-alex

   on Jan 20, 2022

   Member

   More actions

   Please feel free to submit a PR if an overloaded signature including the Txn parameter would assist.
2. fbiricotti commented on Feb 16, 2022

   

   fbiricotti

   on Feb 16, 2022

   Author

   More actions

   Just submitted a pull request. I modified the code and verified it by compiling, but I didn't add any specific tests for the new API versions (I did it on my env but by using JUnit in another project context. Thank you
3. bp-alex

   added

   pr-received

   on Apr 15, 2022
4. dacr commented on Apr 16, 2022

   

   dacr

   on Apr 16, 2022

   More actions

   I confirm this is an important issue, I'm struggling with my unit tests for several days. Those tests are run in parallel by default, and as soon as I'm using a shared (between tests) lmdb environment I encounter random issues and crashes. I'm creating a lot of new databases during the tests, I've achieved a single writer semantic using a semaphore which protect database creation and all write transactions operations.
   My current config :

                                Env
                                  .create()
                                  .setMapSize(100_000_000_000L)
                                  .setMaxDbs(10_000)
                                  .setMaxReaders(100)
                                  .open(
                                    databasesPath,
                                    EnvFlags.MDB_NOTLS,
                                    EnvFlags.MDB_NOLOCK, // Locks managed using ZIO Semaphore
                                    EnvFlags.MDB_NOSYNC  // Acceptable, in particular because EXT4 is used
                                  )
   

   Could it be possible that a database creation during a readTransaction may also generates those crashes, because in my situation with the semaphore and with the pull request from @fbiricotti I still have random crashes or issues. (But everything goes fine if I do not use a shared environment between all tests OR if I run the tests sequentially)

   I've tried with 0.8.2, 0.8.3-SNAPSHOT (built by my self from the original repo and alsor from the PR repo)

   Some examples of the issues :

       - many updates within multiple collection - Lightening Memory Mapped Database abstraction layer spec
         Exception in thread "zio-fiber-99" org.lmdbjava.Txn$BadException: org.lmdbjava.Txn$BadException: Transaction must abort, has a child, or is invalid (-30782)
         	at com.orange.thingin.micro.LMDBOperationsSpec$.spec.macro(LMDBOperationsSpec.scala:188)
         	at com.orange.thingin.micro.LMDBOperationsSpec$.spec.macro(LMDBOperationsSpec.scala:189)
         	at com.orange.thingin.micro.LMDBOperationsSpec$.spec.macro(LMDBOperationsSpec.scala:56)
         	at com.orange.thingin.micro.LMDBOperationsSpec$.spec.macro(LMDBOperationsSpec.scala:222)
       - basic CRUDL operations
         Test failed after 23 iterations with input: ( ,,)
   ...
       - stream collection content
         ✗ 49999 was not equal to 50000
         returnedCount.toInt == count / 2
         .toInt = 49999
         returnedCount = 49999
         at LMDBOperationsSpec.scala:215
   

   # A fatal error has been detected by the Java Runtime Environment:
   #
   #  SIGSEGV (0xb) at pc=0x00007ff7435ee232, pid=17121, tid=17175
   #
   # JRE version: OpenJDK Runtime Environment (17.0.1+12) (build 17.0.1+12-39)
   # Java VM: OpenJDK 64-Bit Server VM (17.0.1+12-39, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
   # Problematic frame:
   # C  [lmdbjava-native-library-11859737896682889782.so+0x4232]  mdb_page_get.isra.13+0x72
   #
   # Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport %p %s %c %d %P %E" (or dumping to /home/dcr/dev-pro/tmini/core.17121)
   #
   # An error report file with more information is saved as:
   # /home/dcr/dev-pro/tmini/hs_err_pid17121.log
       - many updates within multiple collection - Lightening Memory Mapped Database abstraction layer spec
         Exception in thread "zio-fiber-104" org.lmdbjava.Txn$BadException: org.lmdbjava.Txn$BadException: Transaction must abort, has a child, or is invalid (-30782)
   

   #
   # A fatal error has been detected by the Java Runtime Environment:
   #
   #  SIGSEGV (0xb) at pc=0x00007f53c0040043, pid=24630, tid=24682
   #
   # JRE version: OpenJDK Runtime Environment (17.0.1+12) (build 17.0.1+12-39)
   # Java VM: OpenJDK 64-Bit Server VM (17.0.1+12-39, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
   # Problematic frame:
   # C  [lmdbjava-native-library-549967747319896674.so+0x5043]  mdb_page_get.isra.0+0x73
   #
   # Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport %p %s %c %d %P %E" (or dumping to /home/dcr/dev-pro/tmini/core.24630)
   #
   # An error report file with more information is saved as:
   # /home/dcr/dev-pro/tmini/hs_err_pid24630.log
   

   + Lightening Memory Mapped Database abstraction layer spec
       + platform check
       + create and list databases
       + try to get an non existent key
       + safe update in place
       + many overwrite updates
       - try to set/get a key - Lightening Memory Mapped Database abstraction layer spec
         Exception in thread "zio-fiber-99" scala.None$: None
       - many updates within multiple collection - Lightening Memory Mapped Database abstraction layer spec
         Exception in thread "zio-fiber-101" org.lmdbjava.Txn$BadException: org.lmdbjava.Txn$BadException: Transaction must abort, has a child, or is invalid (-30782)
       - list collection content - Lightening Memory Mapped Database abstraction layer spec
         Exception in thread "zio-fiber-100" org.lmdbjava.LmdbNativeException$ConstantDerivedException: org.lmdbjava.LmdbNativeException$ConstantDerivedException: Platform constant error code: EFBIG File too large (27)
       - stream collection content - Lightening Memory Mapped Database abstraction layer spec
         Exception in thread "zio-fiber-99" org.lmdbjava.LmdbNativeException$ConstantDerivedException: org.lmdbjava.LmdbNativeException$ConstantDerivedException: Platform constant error code: EFBIG File too large (27)
       - stream collection content - Lightening Memory Mapped Database abstraction layer spec
         Exception in thread "zio-fiber-99" org.lmdbjava.LmdbNativeException$ConstantDerivedException: org.lmdbjava.LmdbNativeException$ConstantDerivedException: Platform constant error code: EFBIG File too large (27)
         ✗ Result was false
         listed1.contains(updatedValue) ?? "for key  "
         listed1 = Nil
         at LMDBOperationsSpec.scala:123
   

   /home/dcr/.jdks/openjdk-17.0.1/bin/java --add-opens java.base/java.nio=ALL-UNNAMED --add-opens java.base/sun.nio.ch=ALL-UNNAMED -javaagent:/home/dcr/.local/share/JetBrains/Toolbox/apps/IDEA-U/ch-0/221.5080.210/lib/idea_rt.jar=35165:/home/dcr/.local/share/JetBrains/Toolbox/apps/IDEA-U/ch-0/221.5080.210/bin -Dfile.encoding=UTF-8 ...
   + Lightening Memory Mapped Database abstraction layer spec
       + create and list databases
       + platform check
       + try to get an non existent key
       + try to set/get a key
       + safe update in place
       + many overwrite updates
   mdb.c:3158: Assertion 'pglast <= env->me_pglast' failed in mdb_freelist_save()
   
   Process finished with exit code 134 (interrupted by signal 6: SIGABRT)
   
5. dacr commented on Apr 19, 2022

   

   dacr

   on Apr 19, 2022

   More actions

   I've left my previous comment, but now I know that my issue was related to the lock management I've implemented, from the documentation about MDB_NOLOCK :

   1. the caller must enforce single-writer semantics
   2. the caller must ensure that no readers are using old transactions while a writer is active

   I've only managed the first prerequisite, I'm now moving from a simple semaphore to a reentrant lock system.

   And thank you for your great library !

   David.

   Reacted by Ian Pojman
6. bp-alex

   closed this as completedin 4182ad1on Feb 3, 2023
7. at055612

   added a commit that references this issue on Feb 28, 2024

   Dbi.openDbi(..) to support user-provided Txn (fixes lmdbjava#192)

   This issue will close once commit 4182ad1 is merged into the 'master' branch.

   4182ad1

Issue actions