If you want to benchmark you code, the Java Microbenchmark Harness is the tool of choice.
In our example we shall use the refill-rate-limiter project

 

Since refill-rate-limiter uses Gradle we will use the following plugin for gradle

plugins {
...
  id "me.champeau.gradle.jmh" version "0.5.3"
...
}

We shall place the Benchmark at the jmh/java/io/github/resilience4j/ratelimiter folder.

Our Benchmark should look like this.

package io.github.resilience4j.ratelimiter;

import io.github.resilience4j.ratelimiter.internal.RefillRateLimiter;
import org.openjdk.jmh.annotations.*;
import org.openjdk.jmh.infra.Blackhole;
import org.openjdk.jmh.profile.GCProfiler;
import org.openjdk.jmh.runner.Runner;
import org.openjdk.jmh.runner.RunnerException;
import org.openjdk.jmh.runner.options.Options;
import org.openjdk.jmh.runner.options.OptionsBuilder;

import java.time.Duration;
import java.util.concurrent.TimeUnit;
import java.util.function.Supplier;

@State(Scope.Benchmark)
@OutputTimeUnit(TimeUnit.MICROSECONDS)
@BenchmarkMode(Mode.All)
public class RateLimiterBenchmark {

    private static final int FORK_COUNT = 2;
    private static final int WARMUP_COUNT = 10;
    private static final int ITERATION_COUNT = 10;
    private static final int THREAD_COUNT = 2;

    private RefillRateLimiter refillRateLimiter;

    private Supplier<String> refillGuardedSupplier;

    public static void main(String[] args) throws RunnerException {
        Options options = new OptionsBuilder()
                .addProfiler(GCProfiler.class)
                .build();
        new Runner(options).run();
    }

    @Setup
    public void setUp() {

        RefillRateLimiterConfig refillRateLimiterConfig = RefillRateLimiterConfig.custom()
                                                                                 .limitForPeriod(1)
                                                                                 .limitRefreshPeriod(Duration.ofNanos(1))
                                                                                 .timeoutDuration(Duration.ofSeconds(5))
                                                                                 .build();

        refillRateLimiter = new RefillRateLimiter("refillBased", refillRateLimiterConfig);

        Supplier<String> stringSupplier = () -> {
            Blackhole.consumeCPU(1);
            return "Hello Benchmark";
        };

        refillGuardedSupplier = RateLimiter.decorateSupplier(refillRateLimiter, stringSupplier);
    }

    @Benchmark
    @Threads(value = THREAD_COUNT)
    @Warmup(iterations = WARMUP_COUNT)
    @Fork(value = FORK_COUNT)
    @Measurement(iterations = ITERATION_COUNT)
    public String refillPermission() {
        return refillGuardedSupplier.get();
    }

}

Let’s now check the elements one by one.

By using Benchmark scope all the threads used on the benchmark scope will share the same object. We do so because we want to test how refill-rate-limiter performs in a multithreaded scenario.

@State(Scope.Benchmark)

We would like our results to be reported in microseconds, therefore we shall use the OutputTimeUnit.

@OutputTimeUnit(TimeUnit.MICROSECONDS)

On JMH We have various benchmark modes depending on what we want to measure.

Throughput is when we want to measure the number operations per unit of time.
AverageTime when we want to measure the average time per operation.
SampleTime when we want to sample the time for each operation including min, max time, more than just the average.
SingleShotTime: when we want to measure the time for a single operation. This can help when we want to identify how the operation will do on a cold start.

We also have the option to measure all the above.

@BenchmarkMode(Mode.All)

Those options configured on the class level will apply to the benchmark methods we shall add.

Let’s also examine how the benchmark will run

We will specify the number of Threads by using the Threads annotation.

@Threads(value = THREAD_COUNT)

Also we want to warm up before we run the actual benchmarks. This way our code will be initialized, online optimizations will take place, and our runtime will adapt to the conditions before we run the benchmarks.

@Warmup(iterations = WARMUP_COUNT)

Using a Fork we shall instruct how many times the benchmark will run.

@Fork(value = FORK_COUNT)

Then we need to specify the number of iterations we want to measure/

@Measurement(iterations = ITERATION_COUNT)

We can start our test by just using

gradle jmh

The results will be save in a file.

...
2022-10-28T09:08:44.522+0100 [QUIET] [system.out] Benchmark result is saved to /path/refill-rate-limiter/build/reports/jmh/results.txt
..

Let’s examine the results.

Benchmark                                                         Mode       Cnt      Score   Error   Units
RateLimiterBenchmark.refillPermission                            thrpt        20     13.594 ± 0.217  ops/us
RateLimiterBenchmark.refillPermission                             avgt        20      0.147 ± 0.002   us/op
RateLimiterBenchmark.refillPermission                           sample  10754462      0.711 ± 0.025   us/op
RateLimiterBenchmark.refillPermission:refillPermission·p0.00    sample                  ≈ 0           us/op
RateLimiterBenchmark.refillPermission:refillPermission·p0.50    sample                0.084           us/op
RateLimiterBenchmark.refillPermission:refillPermission·p0.90    sample                0.125           us/op
RateLimiterBenchmark.refillPermission:refillPermission·p0.95    sample                0.125           us/op
RateLimiterBenchmark.refillPermission:refillPermission·p0.99    sample                0.209           us/op
RateLimiterBenchmark.refillPermission:refillPermission·p0.999   sample              139.008           us/op
RateLimiterBenchmark.refillPermission:refillPermission·p0.9999  sample              935.936           us/op
RateLimiterBenchmark.refillPermission:refillPermission·p1.00    sample            20709.376           us/op
RateLimiterBenchmark.refillPermission                               ss        20     14.700 ± 4.003   us/op

As we can see we have the modes listed.
Count is the number of iterations. Apart from throughput where we measure the operations by time, the rest is time per operation.
Throughput,Average and Single shot are straightforward, Sample lists the percentiles. Error is the margin of error.

That’s it! Happy benchmarking