Description

ghudo

opened on Apr 20, 2020

Issue body actions

Hi @benalexau - you've been so helpful... I was wondering if you could sanity check the below... if not, or if there's a better forum for these types of questions please let me know!

LMDB really does seem to be best-in-class for read focused performance - I'm surprised there's not more people using/discussing it... it's made it challenging to get as far as I've got!

Background: I've been using LMDB for a while with a lot of success - all my keys are incrementing longs (put with MDB_APPEND flag) and my values are binary encoded, compressed (using LZ4-fast) with checksums (using xxHash64).

My questions;

1. I've been using LMDB a bit like a relational database and creating a separate LMDB file/Env for each entity (i.e. think SQL table). This works very well, but there are rare cases where I have to open transaction open across multiple LMDB file/Env's.

2. I group things like foreign-key relationships into the same LMDB file/Env as a separate named database to avoid having to always open multiple read or write transactions.

3. There are cases where I want to partition information - so I have a separate LMDB file/Env for each calendar month or complex user specific databases - so I keep only the most commonly/recently used LMDB file/Env's open to limit the number of environments I have open at a time.

4. There are rare (and performance critical) cases where I want to upgrade from a read lock to a write lock. In these cases, I first release the read lock, then open the write lock and re-read everything again with this write lock. I've noted from What does Environment maxreaders reached (-30790) actually mean? #65 (comment) that you prefer JVM-based locking. I thought I might get performance gains by using JVM-based locking and setting EnvFlags.MDB_NOLOCK - but didn't... so gave up.

Is there anything obviously wrong with any of the above? Any recommendations?

Thanks in advance ;-)

Reacted by Bertozzi Julien

Activity

1. bp-alex commented on Apr 21, 2020

   

   bp-alex

   on Apr 21, 2020

   Member

   More actions

   By way of background to comments I've made in other tickets, I mostly use LMDB as a fast, durable, key-value store, with the data arriving from various external, authoritative, replayable sources that can be used to deterministically build (or rebuild) a given temporally-scoped LMDB file. As such LMDB is not the system of record, but it provides a way to transform large message-based event streams into a form that becomes vastly more space-efficient (thanks to chunking of what are typically very similar values) and amenability to random querying.

   Like you, I use lots of separate LMDB files (or Envs), in my case one per calendar day. There are DAOs which usually only deal with one "type" of data, such as pricing data. There are many databases (Dbis) to separately store the data and indexes. My primary keys and indexes always use Big Endian so I can create consistent composite keys (eg a string then a Big Endian long timestamp might be a key for a chunk of data). A single read-write transaction is held by the DAO for the duration of its operation. All read and write operations are coordinated using Java locking primitives where required, but it's highly infrequent to need this given isolation is rarely necessary for my workload. The two major trade-offs of this model are (a) it does not provide data visibility to other processes and (b) if the DAO does not close correctly, it will not commit the transaction and the contents will need to be rebuilt.

   Overall your approach sounds good. Some quick thoughts:

   • I assume you are aware there are no transactions across Envs.
   • You didn't clarify the Endianness of your keys. With incrementing longs you might consider setting DbiFlags.MDB_INTEGERKEY and keeping them Little Endian. If you are not using this flag you will need to be inserting as Big Endian so future retrieval order (eg in Cursors) is correct.
   • Make sure you have some unit tests that verify the insertion and retrieval across byte orders, especially if you're composing your own composite keys.
   • Value compression may not be as helpful as it first appears given they will always consume at least one LMDB page. There's a fuller discussion here.

   Reacted by Viktor Szathmáry and Bertozzi Julien
2. ghudo commented on Apr 21, 2020

   

   ghudo

   on Apr 21, 2020

   Author

   More actions

   Thanks @benalexau!

   I'm much more confident now I'm on the right track! I was aware transactions don't cross Env's and DbiFlags.MDB_INTEGERKEY requires little endian to be ordered correctly :-)

   I wasn't aware entries always filled an LMDB page!

   • But this doesn't occur with PutFlags.MDB_APPEND as you commented here https://github.com/lmdbjava/benchmarks/issues/9#issuecomment-344074558 ? You said "but the trade-off is slower future inserts if MDB_APPEND can no longer be used" - I'm a little confused by this - say I always insert/add new values with MDB_APPEND, but occasionally update existing values with older ids - what are the space/performance consequences of this?

   • I'm starting to understand why you transform your data into a more space efficient form... I'm less concerned about storage size, more concerned about reliability and read performance for now... but thanks for sharing how you're using LMDB - I might look into using a more space efficient form later as well! :-)

   Thanks for letting me pick your brains :-)
3. bp-alex commented on Apr 22, 2020

   

   bp-alex

   on Apr 22, 2020

   Member

   More actions

   But this doesn't occur with PutFlags.MDB_APPEND

   Sorry, I overlooked your earlier mention of MDB_APPEND.

   In July 2017 @hyc and I discussed storage consumption over email. His reply was very informative on how LMDB optimises page space, so I hope he doesn't mind me quoting a portion of it here:

   When we do inserts in random order, if a page fills up, we split it in half so that one or the other half has room for the new entry. That's standard procedure for any Btree.

   When we do inserts in sequential order, we bias the split so that the new page has more room than the old page, on the assumption that inserts will continue to be sequential. When we do sequential inserts using MDB_APPEND, there is no page split at all - we just allocate a new page and fill it, sequentially. These behaviors are both unique to LMDB.

   For a random insert pattern, you'll be left with a lot of half-full pages. This is normal for all Btrees, and it's intentional. It leaves room for more future inserts, and minimizes the total number of page splits in a sustained random insert workload.

   If you take a DB that was generated sequentially, with fully packed pages, and then start adding new records to it in random order, it will be a lot slower because so many inserts will require page splits.

   Reacted by Bertozzi Julien
4. phraktle commented on Apr 22, 2020

   

   phraktle

   on Apr 22, 2020

   Contributor

   More actions

   @benalexau These should go into a guide / tips & tricks doc.
5. bp-alex commented on Apr 22, 2020

   

   bp-alex

   on Apr 22, 2020

   Member

   More actions

   @phraktle I added issue #151 to track this.
6. bp-alex

   added

   questionQueries on usage of the library, architectural patterns and alikeQueries on usage of the library, architectural patterns and alike

   on Apr 22, 2020
7. hyc commented on Apr 22, 2020

   

   hyc

   on Apr 22, 2020

   More actions

   For the most part, stuff like LMDB's internal behavior doesn't get published because it's such a fine detail. (And it's open source; people who care enough can go read it to understand what it actually does.) In reality, your use case determines whether you can load data in random order or sorted order. If your data arrives in random order and you can't control that, the fact that sorted order is faster (and why) is irrelevant. If your data arrives in sorted order, things will go faster, but it really doesn't matter why.
8. bp-alex

   closed this as completedon May 22, 2020

Issue actions