The JSON benchmarks are wrong

Claiming “We’re the fastest!” is a great way to grow a project’s popularity, particularly for mechanical tasks like JSON parsing. This month there's been plenty of comparison between Jackson and Boon.

Boon author Rick Hightower:

“Boon JSON parser and the new Groovy 2.3 JSON parser are faster than Jackson.”

I contribute to Gson which occasionally gets mentioned. But Gson lags behind Jackson and Boon in benchmarks.

But benchmarks don't tell the whole story.

They measure best-case or common-case scenarios. But both Jackson and Boon are vulnerable to hash denial of service attacks, and behave extremely poorly in the worst case.

I have a 1.2 MiB JSON file with 100,000 keys that all share a common hash code. Gson parses this file in 150 milliseconds. But Boon and Jackson take 100x longer: over 15 seconds to parse the same file. Hash collision performance is exponential, so doubling the file size quadruples parse time for Boon and Jackson.

An attacker could tie up all of your servers with just a few megabytes of malicious data. If you use current releases of Boon or Jackson, you’re vulnerable to DoS attacks.

Unfortunately, the benchmark race focuses on top speed at the expense of safety.

Update (May 12, 2014)

The HashMap in Java 1.8 defends against hash DoS attacks for you, so it should be safe to deploy Boon and Jackson there without consequence.