The LDBC FinBench Driver is a powerful tool designed for benchmarking the performance of graph databases.

Note: The main branch is a work-in-progress for the upcoming v0.2 release aiming scales larger than SF100. For the stable version, please refer to version 0.1.0 on the v0.1.0 branch.

Related repositories:

• FinBench Specification: https://github.com/ldbc/ldbc_finbench_docs
• FinBench DataGen: https://github.com/ldbc/ldbc_finbench_datagen
• FinBench Driver: https://github.com/ldbc/ldbc_finbench_driver
• FinBench Transaction Reference Implementation: https://github.com/ldbc/ldbc_finbench_transaction_impls

The Driver initiates by reading the configuration file. The default configuration file path for DummyDB is src/main/resources/example/ldbc_finbench_driver_dummy.properties.

If required, the System Under Test (SUT) can customize the configuration parameters to connect to the database or read the query file. This can include the request host, port, path, etc.

host=localhost
port=9091
user=admin
pass=123456
path=cypher/

For a new implementation of the SUT, update this configuration parameter accordingly.

db=org.ldbcouncil.finbench.impls.dummy.DummyDb

Please update based on your parameter path. The parameters_dir is the ComplexRead parameter path, and updates_dir is the Write and ReadWrite incremental data path.

ldbc.finbench.transaction.queries.parameters_dir=src/main/resources/example/data/read_params
ldbc.finbench.transaction.queries.updates_dir=src/main/resources/example/data/incremental_data

The driver supports the following user-facing modes:

• CREATE_VALIDATION
• VALIDATE_DATABASE
• EXECUTE_BENCHMARK
• AUTOMATIC_TEST
• OPTIMIZATION_RECOMMENDATION

With CREATE_VALIDATION mode, you create a database result. validation_parameters_size denotes the number of results created, while validate_database refers to the file where the created results are stored.

mode=CREATE_VALIDATION
validation_parameters_size=100
validate_database=validation_params.csv

VALIDATE_DATABASE mode allows you to verify the SUT. The validate_database is the result created by CREATE_VALIDATION mode.

mode=VALIDATE_DATABASE
validate_database=validation_params.csv

Perform the performance test with EXECUTE_BENCHMARK mode. Here are some crucial configuration parameters that need adjustment when operating the driver:

1. thread_count: Represents the number of concurrent requests that the driver can handle, corresponding to the number of active threads running simultaneously within the driver.
2. time_compression_ratio: Controls the intensity of the driver's workload. A lower value yields a higher workload in a shorter timeframe.
3. ignore_scheduled_start_times: Determines whether the driver should follow the scheduled timings for sending requests. If set to true, the driver sends requests as soon as they are prepared, regardless of the schedule.
4. warmup: Denotes the number of preliminary test items processed before the actual benchmarking begins.
5. operation_count: Sets the number of test items executed during the actual benchmarking phase after the warm-up.

mode=EXECUTE_BENCHMARK
thread_count=1
time_compression_ratio=0.001
ignore_scheduled_start_times=false
warmup=5
operation_count=10000

Perform the performance test automatically with the AUTOMATIC_TEST mode, utilizing an adaptive testing approach. This approach leverages pre-execution on hardware devices for initial parameter estimation and achieves optimal performance, meeting requirements through automatic tuning. The entire process is divided into two phases: (1) Rapid Estimation Phase and (2) Precision Tuning Phase.

1. estimate: Quickly calculate the duration of each test phase (in milliseconds). If set to -1, it indicates the completion of operations to achieve the specified warmup value. If the parameter is not used, the default is 300,000 ms.
2. accurate: Specifies the duration of each test in the precise tuning phase (in milliseconds). If set to -1, it indicates the completion of operations to reach the specified operation_count. If the parameter is not used, the default is 7,200,000 ms.
3. error_range: Binary end condition, indicating the tolerance range. If the parameter is not used, the default is 1E-5.
4. tcr_min: Minimum time compression ratio limit. If the parameter is not used, the default is 1E-9.
5. tcr_max: Maximum time compression ratio limit. If the parameter is not used, the default is 1.
6. timeout_rate: Specifies the ratio of timeouts to total operations at which the threshold is allowed to be exceeded. If the parameter is not used, the default is 0.05.
7. time_compression_ratio: When utilized in the initial testing round, setting an appropriate value for the machine can significantly reduce the overall test duration. If the parameter is not used, the default is the average of (tcr_min + tcr_max) / 2.

mode=AUTOMATIC_TEST
estimate=300000
accurate=7200000
error_range=1E-5
tcr_min=1E-9
tcr_max=1
timeout_rate=0.05
time_compression_ratio=0.1

Runs a normal benchmark as a baseline, ranks operation types by optimization recommendation score, profiles each selected operation through the optional SUT OptimizationSupport capability, then reruns the full workload for every selected operation and reduction step. During each scenario run, only the selected operation is replaced by the SUT synthetic execution hook; all other operations use their normal handlers. Existing benchmark modes do not require this capability.

The driver captures one real operation, including its parameters, for every operation type during the baseline. It executes the selected operations through their normal database handlers while sampling resources. A SUT only needs to implement getCurrentCpuCores, getCurrentMemoryBytes, and executeSyntheticLoad; it does not need to construct or dispatch FinBench operations.

CPU is expressed as cores, memory as bytes, and synthetic duration as nanoseconds. The SUT must release requested resources before executeSyntheticLoad returns. Generated scenario configuration stores duration in optimization.simulation.target_duration_nanos, while reports display fractional milliseconds. See the Dummy implementation for an integration example.

mode=OPTIMIZATION_RECOMMENDATION
results_dir=results/optimization
operation_count=10000
optimization.top_n=3
optimization.reduction_steps=0.05,0.10,0.20,0.30
optimization.profile_repetitions=3
optimization.repetitions=3
optimization.sample_interval_millis=50

The mode writes optimization-report.json and optimization-report.md. Scenario throughput gains are measured from those full-workload reruns and should be verified again after implementing the real SUT optimization.

The DummyDB example configuration is available at src/main/resources/example/ldbc_finbench_optimization_recommendation_dummy.properties.

To get started, clone the repository and build it with Maven:

git clone https://github.com/ldbc/ldbc_finbench_driver.git
cd ldbc_finbench_driver
mvn clean package -DskipTests

For a quick trial of the driver, utilize the DummyDB shipped with it by running the following command:

java -cp target/driver-0.2.0-alpha.jar org.ldbcouncil.finbench.driver.driver.Driver -P src/main/resources/example/ldbc_finbench_driver_dummy.properties

To automatically find the suitable time_compression_ratio for your system on this machine, execute the following command:

java -cp target/driver-0.2.0-alpha.jar org.ldbcouncil.finbench.driver.driver.Driver -P src/main/resources/example/ldbc_finbench_automatic_test_dummy.properties

To run the optimization recommendation example:

java -cp target/driver-0.2.0-alpha.jar org.ldbcouncil.finbench.driver.driver.Driver -P src/main/resources/example/ldbc_finbench_optimization_recommendation_dummy.properties