After years of struggling with one-off benchmarking harnesses, Kevin B and I started Caliper in 2009. It started out as a project for benchmarking Android and Java. It knew about
java, and could benchmark an attached device. Later, Greg Kick took over the project and gave it a better understanding of HotSpot JVM, plus a new web UI.
The JMH project was first published in 2013. It addresses the same problems as Caliper but with even better access to HotSpot internals.
JMH’s Runner Is Very Sophisticated
It has flexible concurrency control, a powerful
State abstraction, and a thoughtful API. JMH has some amazing features like
@CompilerControl(DONT_INLINE) which is going to be quite fun for me later!
The JMH runner is more precise than Caliper’s. Caliper uses reflection to invoke the benchmarking method; this has lots of bad consequences such as the need for a
reps parameter. JMH just code-gens the entry point to avoid interference from reflection. Way better.
No Love For Android
Because of broken history between Oracle