If you want to benchmark you code, the Java Microbenchmark Harness is the tool of choice.
In our example we shall use the refill-rate-limiter project
Since refill-rate-limiter uses Gradle we will use the following plugin for gradle
plugins {
...
id "me.champeau.gradle.jmh" version "0.5.3"
...
}
We shall place the Benchmark at the jmh/java/io/github/resilience4j/ratelimiter folder.
Our Benchmark should look like this.
package io.github.resilience4j.ratelimiter;
import io.github.resilience4j.ratelimiter.internal.RefillRateLimiter;
import org.openjdk.jmh.annotations.*;
import org.openjdk.jmh.infra.Blackhole;
import org.openjdk.jmh.profile.GCProfiler;
import org.openjdk.jmh.runner.Runner;
import org.openjdk.jmh.runner.RunnerException;
import org.openjdk.jmh.runner.options.Options;
import org.openjdk.jmh.runner.options.OptionsBuilder;
import java.time.Duration;
import java.util.concurrent.TimeUnit;
import java.util.function.Supplier;
@State(Scope.Benchmark)
@OutputTimeUnit(TimeUnit.MICROSECONDS)
@BenchmarkMode(Mode.All)
public class RateLimiterBenchmark {
private static final int FORK_COUNT = 2;
private static final int WARMUP_COUNT = 10;
private static final int ITERATION_COUNT = 10;
private static final int THREAD_COUNT = 2;
private RefillRateLimiter refillRateLimiter;
private Supplier<String> refillGuardedSupplier;
public static void main(String[] args) throws RunnerException {
Options options = new OptionsBuilder()
.addProfiler(GCProfiler.class)
.build();
new Runner(options).run();
}
@Setup
public void setUp() {
RefillRateLimiterConfig refillRateLimiterConfig = RefillRateLimiterConfig.custom()
.limitForPeriod(1)
.limitRefreshPeriod(Duration.ofNanos(1))
.timeoutDuration(Duration.ofSeconds(5))
.build();
refillRateLimiter = new RefillRateLimiter("refillBased", refillRateLimiterConfig);
Supplier<String> stringSupplier = () -> {
Blackhole.consumeCPU(1);
return "Hello Benchmark";
};
refillGuardedSupplier = RateLimiter.decorateSupplier(refillRateLimiter, stringSupplier);
}
@Benchmark
@Threads(value = THREAD_COUNT)
@Warmup(iterations = WARMUP_COUNT)
@Fork(value = FORK_COUNT)
@Measurement(iterations = ITERATION_COUNT)
public String refillPermission() {
return refillGuardedSupplier.get();
}
}
Let’s now check the elements one by one.
By using Benchmark scope all the threads used on the benchmark scope will share the same object. We do so because we want to test how refill-rate-limiter performs in a multithreaded scenario.
@State(Scope.Benchmark)
We would like our results to be reported in microseconds, therefore we shall use the OutputTimeUnit.
@OutputTimeUnit(TimeUnit.MICROSECONDS)
On JMH We have various benchmark modes depending on what we want to measure.
Throughput is when we want to measure the number operations per unit of time.
AverageTime when we want to measure the average time per operation.
SampleTime when we want to sample the time for each operation including min, max time, more than just the average.
SingleShotTime: when we want to measure the time for a single operation. This can help when we want to identify how the operation will do on a cold start.
We also have the option to measure all the above.
@BenchmarkMode(Mode.All)
Those options configured on the class level will apply to the benchmark methods we shall add.
Let’s also examine how the benchmark will run
We will specify the number of Threads by using the Threads annotation.
@Threads(value = THREAD_COUNT)
Also we want to warm up before we run the actual benchmarks. This way our code will be initialized, online optimizations will take place, and our runtime will adapt to the conditions before we run the benchmarks.
@Warmup(iterations = WARMUP_COUNT)
Using a Fork we shall instruct how many times the benchmark will run.
@Fork(value = FORK_COUNT)
Then we need to specify the number of iterations we want to measure/
@Measurement(iterations = ITERATION_COUNT)
We can start our test by just using
gradle jmh
The results will be save in a file.
... 2022-10-28T09:08:44.522+0100 [QUIET] [system.out] Benchmark result is saved to /path/refill-rate-limiter/build/reports/jmh/results.txt ..
Let’s examine the results.
Benchmark Mode Cnt Score Error Units RateLimiterBenchmark.refillPermission thrpt 20 13.594 ± 0.217 ops/us RateLimiterBenchmark.refillPermission avgt 20 0.147 ± 0.002 us/op RateLimiterBenchmark.refillPermission sample 10754462 0.711 ± 0.025 us/op RateLimiterBenchmark.refillPermission:refillPermission·p0.00 sample ≈ 0 us/op RateLimiterBenchmark.refillPermission:refillPermission·p0.50 sample 0.084 us/op RateLimiterBenchmark.refillPermission:refillPermission·p0.90 sample 0.125 us/op RateLimiterBenchmark.refillPermission:refillPermission·p0.95 sample 0.125 us/op RateLimiterBenchmark.refillPermission:refillPermission·p0.99 sample 0.209 us/op RateLimiterBenchmark.refillPermission:refillPermission·p0.999 sample 139.008 us/op RateLimiterBenchmark.refillPermission:refillPermission·p0.9999 sample 935.936 us/op RateLimiterBenchmark.refillPermission:refillPermission·p1.00 sample 20709.376 us/op RateLimiterBenchmark.refillPermission ss 20 14.700 ± 4.003 us/op
As we can see we have the modes listed.
Count is the number of iterations. Apart from throughput where we measure the operations by time, the rest is time per operation.
Throughput,Average and Single shot are straightforward, Sample lists the percentiles. Error is the margin of error.
That’s it! Happy benchmarking
Emmanouil Gkatziouras 