KalamDB Benchmarking Tool

Overview

The KalamDB Benchmarking Tool is designed to facilitate performance testing of the KalamDB database. It provides a framework for running various benchmark tests, generating detailed reports in both JSON and HTML formats, and allows for easy addition of new benchmark tests.

Project Structure

• src/: Contains the core library code for the benchmarking tool.

  • lib.rs: Core library code defining main structures and functions.
  • main.rs: Entry point for the benchmarking tool, initializes the application and triggers benchmark runs.
  • config.rs: Handles configuration settings for the benchmarking tool.
  • runner.rs: Logic for executing benchmarks and collecting metrics.
  • reporter/: Module for different reporting strategies.
    • json_reporter.rs: Generates JSON reports of benchmark results.
    • html_reporter.rs: Generates HTML reports of benchmark results.
  • client.rs: Client logic for interacting with the KalamDB server.
  • metrics.rs: Structures and functions for collecting and processing benchmark metrics.
  • benchmarks/: Organizes benchmark tests.
    • insert_bench.rs: Benchmark test for insert operations.
    • select_bench.rs: Benchmark test for select operations.
    • update_bench.rs: Benchmark test for update operations.
    • delete_bench.rs: Benchmark test for delete operations.
    • bulk_insert_bench.rs: Benchmark test for bulk insert operations.
    • concurrent_read_bench.rs: Benchmark test for concurrent read operations.
    • concurrent_write_bench.rs: Benchmark test for concurrent write operations.
    • mixed_workload_bench.rs: Benchmark test for mixed workload operations.

• templates/: Contains templates for generating reports.

  • report.html: HTML template for the benchmark report.
  • styles.css: CSS styles for the HTML report.

• results/: Directory for storing benchmark results.

  • .gitkeep: Keeps the results directory in version control.

• tests/: Contains tests for the benchmarking tool.

  • integration.rs: Integration tests for the benchmarking tool.
  • reporter_tests.rs: Unit tests for the reporting functionality.

• run-benchmarks.sh: Shell script to run the default benchmark suite in a Unix-like environment.

• run-connection-scale.sh: Dedicated runner for the opt-in connection_scale stress benchmark on macOS/loopback hosts.

• run-benchmarks.ps1: PowerShell script to run benchmarks in a Windows environment.

• Cargo.toml: Configuration file for the Rust project.

Getting Started

1. Clone the Repository:

   git clone <repository-url>
   cd KalamDB

2. Build the Project:

   cargo build --release

3. Build the KalamDB Server:

cd ../backend
cargo build --release --bin kalamdb-server

4. Run Benchmarks: Use the provided scripts to run benchmarks:
   • For Unix-like systems:

     ./run-benchmarks.sh

   • For the dedicated connection-scale stress run on macOS / loopback hosts:

     ./run-connection-scale.sh

   • For Windows:

     .\run-benchmarks.ps1

By default, run-benchmarks.sh manages a dedicated local benchmark server for loopback runs. The default suite excludes the opt-in connection_scale benchmark because that test often needs extra loopback aliases or an explicit override on single-host macOS runs. It will:

• Verify a release kalamdb-server binary exists.
• Refuse to run if a local server is already listening on the target port.
• Clear benchv2/data before startup.
• Start the server with benchv2/server.toml.
• Wait for /health and then continue into the Rust benchmark pre-flight checks.

If you explicitly want connection_scale, use ./run-connection-scale.sh or pass --bench connection_scale yourself with the required environment/setup.

If you pass a non-loopback or multi-URL --urls value, the script skips managed-server startup and uses the supplied servers directly.

Adding New Benchmarks

To add a new benchmark test:

1. Create a new Rust file in the src/benchmarks/ directory.
2. Implement the benchmark logic and metrics collection.
3. Update src/benchmarks/mod.rs to include the new benchmark.

Reporting

The tool generates reports in both JSON and HTML formats:

• JSON reports are generated by json_reporter.rs and saved to the specified location.
• HTML reports are generated by html_reporter.rs using the report.html template.

Contribution

Contributions are welcome! Please follow the standard Git workflow:

1. Fork the repository.
2. Create a new branch for your feature or bug fix.
3. Submit a pull request with a description of your changes.

License

Licensed under the Apache License, Version 2.0 (Apache-2.0). See ../LICENSE.txt and ../NOTICE.