Today I’m playing with JMH, the Java benchmarking harness written by Oracle. I’m hoping to try out some optimizations for Moshi.
Some History
After years of struggling with one-off benchmarking harnesses, Kevin B and I started Caliper in 2009. It started out as a project for benchmarking Android and Java. It knew about dalvikvm and java, and could benchmark an attached device. Later, Greg Kick took over the project and gave it a better understanding of HotSpot JVM, plus a new web UI.
The JMH project was first published in 2013. It addresses the same problems as Caliper but with even better access to HotSpot internals.
JMH’s Runner Is Very Sophisticated
It has flexible concurrency control, a powerful State abstraction, and a thoughtful API. JMH has some amazing features like @CompilerControl(DONT_INLINE) which is going to be quite fun for me later!
The JMH runner is more precise than Caliper’s. Caliper uses reflection to invoke the benchmarking method; this has lots of bad consequences such as the need for a reps parameter. JMH just code-gens the entry point to avoid interference from reflection. Way better.
No Love For Android
Because of broken history between Oracle and Android, it’s unlikely that we’ll see the JMH team make this happen. It’s too bad, because battery-powered Android devices need just as much optimization as plugged-in servers.
Android’s ART and dalvikvms have significantly different performance characteristics from the JVM. Which means that optimizing code for the JVM may be pessimizing that same code for Android. We saw this with Gson.
Sigh.
No Fancy Pivot Tables
One of the early design goals of Caliper was making it easy to see how different parameters impacted performance. We built two generations of JavaScript pivot tables to help explore and make sense of a benchmark’s results.
Again, I’d love to see some effort bringing this frontend to JMH’s powerful engine. (Though I can’t seem to find the Caliper web source code anywhere; was it a victim of the Google Code migration?)
I’m frustrated by the process of actually using JMH. I’d like to tag a run as a baseline, change my code, and do a comparison. I think my workflow is going to involve a spreadsheet and copy-pasting, sadly.
Dat Documentation
The JMH samples provide a great tour of the API. The genetic algorithm example is particularly fun.
The documentation frequently refers to the Javadoc, but the JMH maintainers haven‘t bothered to publish the Javadoc on the web. Weak. Googling for the Javadoc returns out of date docs covered in banner ads.
When I used their recommended mvn archetype:generate script to set up my project, it spewed COPYRIGHT ORACLE poison into the template. To avoid granting the litigious Oracle corporation ownership of my code I had to set up my benchmark manually.
Try It Out
JMH is good stuff. You can get it on the JMH project page.