Description

jubax

opened on Nov 17, 2017

Issue body actions

In LMDB it is possible to use DbiFlags.MDB_INTEGERKEY when opening a database. When this flag is used then it is assumed that the keys are stored in native byte order.

When I have one database with MDB_INTEGERKEY then I use ByteBuffer.order(ByteOrder.nativeOrder()). Otherwise I keep the order.

When I retrieve values from the MDB_INTEGERKEY store I need to change the byte order of the returned ByteBuffer. Note that "db1" uses MDB_INTEGERKEY and "db2" not. The ByteBuffer which is returned is shared between both databases.

			try (Txn<ByteBuffer> txn = env.txnRead();) {
				ByteBuffer result = db1.get(txn, key); // <-- This buffer is re-used
				result.order(ByteOrder.nativeOrder());
				assertEquals(100, result.getInt());
			}
			try (Txn<ByteBuffer> txn = env.txnRead();) {
				ByteBuffer result = db2.get(txn, key);
				assertEquals(333, result.getInt()); // FAIL because "result" has native order
			}

The API does not mention that I cannot change e.g. the byte order of the returned ByteBuffer object.

Activity

1. jubax commented on Nov 17, 2017

   

   jubax

   on Nov 17, 2017

   Author

   More actions

   Here a test. The test fails, but it should succeed:

   	@Test
   	public void testLmdbJavaIntegerKeyBehavior() throws IOException {
   		Builder<ByteBuffer> b = Env.create(ByteBufferProxy.PROXY_OPTIMAL).setMapSize(100_000).setMaxDbs(2).setMaxReaders(100);
   		try (Env<ByteBuffer> env = b.open(Files.createTempDirectory("lmdb-test").toFile());) {
   			Dbi<ByteBuffer> db1 = env.openDbi("db1", DbiFlags.MDB_CREATE);
   			Dbi<ByteBuffer> db2 = env.openDbi("db2", DbiFlags.MDB_CREATE);
   			
   			// Put 10->100 to db1
   			ByteBuffer key = ByteBuffer.allocateDirect(4);
   			key.putInt(10);
   			key.flip();
   			ByteBuffer val = ByteBuffer.allocateDirect(4);
   			val.putInt(100);
   			val.flip();
   			
   			try (Txn<ByteBuffer> txn = env.txnWrite();) {
   				db1.put(txn, key, val);
   				ByteBuffer result = db1.get(txn, key);
   				assertEquals(100, result.getInt());
   				txn.commit();
   			}
   			
   			// Put 20->200 to db2
   			ByteBuffer key2 = ByteBuffer.allocateDirect(4);
   			key2.putInt(20);
   			key2.flip();
   			ByteBuffer val2 = ByteBuffer.allocateDirect(4);
   			val2.putInt(200);
   			val2.flip();
   			
   			try (Txn<ByteBuffer> txn = env.txnWrite();) {
   				db2.put(txn, key2, val2);
   				ByteBuffer result = db2.get(txn, key2);
   				assertEquals(200, result.getInt());
   				txn.commit();
   			}
   		}
   		
   		b = Env.create(ByteBufferProxy.PROXY_OPTIMAL).setMapSize(100_000).setMaxDbs(2).setMaxReaders(100);
   		try (Env<ByteBuffer> env = b.open(Files.createTempDirectory("lmdb-test-v2").toFile());) {
   			Dbi<ByteBuffer> db1 = env.openDbi("db1-v2", DbiFlags.MDB_CREATE, DbiFlags.MDB_INTEGERKEY);
   			Dbi<ByteBuffer> db2 = env.openDbi("db2-v2", DbiFlags.MDB_CREATE);
   			
   			// Put 10->100 to db1
   			ByteBuffer key = ByteBuffer.allocateDirect(4);
   			key.order(ByteOrder.nativeOrder());
   			key.putInt(10);
   			key.flip();
   			ByteBuffer val = ByteBuffer.allocateDirect(4);
   			val.order(ByteOrder.nativeOrder());
   			val.putInt(100);
   			val.flip();
   			
   			try (Txn<ByteBuffer> txn = env.txnWrite();) {
   				db1.put(txn, key, val);
   				ByteBuffer result = db1.get(txn, key);
   				result.order(ByteOrder.nativeOrder());
   				assertEquals(100, result.getInt());
   				txn.commit();
   			}
   			
   			// Put 20->200 to db2
   			ByteBuffer key2 = ByteBuffer.allocateDirect(4);
   			key2.putInt(20);
   			key2.flip();
   			ByteBuffer val2 = ByteBuffer.allocateDirect(4);
   			val2.putInt(200);
   			val2.flip();
   			
   			try (Txn<ByteBuffer> txn = env.txnWrite();) {
   				db2.put(txn, key2, val2);
   				ByteBuffer result = db2.get(txn, key2);
                                   // workaround: set byte order
                                   // result.order(ByteOrder.BIG_ENDIAN);
   				assertEquals(200, result.getInt());
   				txn.commit();
   			}
   		}
   	}
   
   
2. krisskross commented on Dec 16, 2017

   

   krisskross

   on Dec 16, 2017

   Contributor

   More actions

   Sorry for leaving you hanging. Last few months have been really busy.

   Reusing the buffer cache between databases was probably wrong from the beginning. And what we need is to be able to specify the ByteOrder for each database. As of now the buffer cache is attached to a static variable in AbstractByteBufferProxy. We could move the cache to a protected instance variable in BufferProxy and let users specify ByteOrder per proxy. However, this would require separate Env's since each proxy belongs to Env (not Dbi).

   Opinions?
3. bp-alex commented on Dec 17, 2017

   

   bp-alex

   on Dec 17, 2017

   Member

   More actions

   Quoting the ByteBuffer JavaDocs:

   The initial order of a byte buffer is always BIG_ENDIAN.

   Is there a reason we couldn't just set the order back to the Big Endian default in ByteBufferBufferProxy's deallocate(...) method? This will be the byte order of the first ByteBuffer returned regardless, as LMDB offers no way to store the byte order.

   I acknowledge that we could infer byte order for key buffers in certain specific cases (eg a Dbi with DbiFlags such as MDB_INTEGERKEY) but this would feel very "magical" when users still need to set the byte order correctly on the ByteBuffer they present for get, put and cursor operations anyway.

   Reacted by Christofer Sjögren
4. krisskross commented on Dec 18, 2017

   

   krisskross

   on Dec 18, 2017

   Contributor

   More actions

   Yes, I agree. That's a lot more consistent and easier to reason about.
5. bp-alex

   closed this as completedin d16ab43on Mar 21, 2018

Issue actions