Menu

OkHttp Certificate Pinning Vulnerability!

Bad news. When I added certificate pinning in OkHttp 2.1, I didn’t sanitize the server’s certificate chain. An attacker could exploit this weakness to defeat the protection offered by certificate pinning.

The vulnerability was disclosed to Square by security researcher John Kozyrakis. Whew! We fixed it in OkHttp 3.1.2, and backported that to OkHttp 2.7.4. Matthew McPherrin has requested a CVE for this vulnerability.

If you’re using OkHttp, you should upgrade to the latest version immediately! Staying up to date on OkHttp is a good idea – we track the latest HTTPS cipher suites & TLS versions to balance connectivity with security. It’s like keeping the browser up to date on your computer: staying current is the best way to stay safe.

LinkedHashMap is always better than HashMap

Deterministic code is much easier to debug & understand than non-determinstic code. Got a test that’s failing? Launch the debugger, step through the code, and find the exact statement where an assumption is violated. For me this often looks like a binary search. I add breakpoints and make assertions like “everything is fine before we call pruneCache()”. I’ll run the app, learn something, and repeat.

When you use HashMap in your programs, you’re introducing needless nondeterminism. This is because HashMap’s iteration order is different between platforms (Android vs. Java) and between executions for classes that don’t override equals() and hashCode().

This nondeterminism is toxic to debugging. I think it’s even worse than the nondeterminism caused by multithreading because it’s so unexpected.

Does this program always return the same result?

private static float massPerUnit(  
    float totalMass, float w, float h, float d) {
  Map<CharSequence, Float> dimensions = new HashMap<>();
  dimensions.put(dimension("width", "x"), w);
  dimensions.put(dimension("height", "y"), h);
  dimensions.put(dimension("depth", "z"), d);

  float massPerUnit = totalMass;
  for (Float dimension : dimensions.values()) {
    massPerUnit /= dimension;
  }

  return massPerUnit;
}

Surprise, it doesn’t. And HashMap’s non-deterministic iteration order is to

Your strict naming conventions are a liability

Programmers tend to be pretty obsessive over consistency. Most of that consistency is worthwhile, but some of it is foolish.

For example, let’s take some JSON from GitHub’s gists API:

{
  "files": {
    "Hello.txt": {
      "type": "text/plain",
      "content": "Hello World!\n"
    }
  },
  "created_at": "2014-05-27T02:31:35Z",
  "updated_at": "2015-08-29T14:01:51Z"
}

And we’ll map this JSON structure to some Java classes:

public final class Gist {  
  Map<String, GistFile> files;
  Date createdAt;
  Date updatedAt;
}
public final class GistFile {  
  String type;
  String content;
}

It’s light work for Gson to decode the document into a Java objects:

Gist gist = gson.fromJson(json, Gist.class);  

And moments later, heartbreak! Because createdAt and updatedAt fields are both null! The problem of course is that these field names are camelCase in Java (createdAt) vs. snake_case in JSON (created_at).

Well that sucks. We’ve got three bad choices.

Choice 1: Get the framework to flex its muscle

Gson has built-in magic to convert snakes into camels with its LOWER_CASE_WITH_UNDERSCORES FieldNamingPolicy. We make a global configuration change to our Gson instance and the problem is solved.

Gson gson = new GsonBuilder()  
    .setFieldNamingPolicy(LOWER_CASE_WITH_UNDERSCORES)
    .create();

Choice 2

Sneaking Data into an OkHttp Interceptor

OkHttp Interceptors are a fun & powerful way to implement cross-cutting behavior in your app. You can use ’em for authentication, performance, monitoring, and more. There’s even open source interceptors for OAuth signing and curl logging!

Suppose we want to write an interceptor that tracks which activities use the network the most. For each HTTP call we’ll measure how big it is, how long it took, and which activity triggered it:

public final class CallMeasurement {  
  long requestSize;
  long responseSize;
  long requestDurationMillis;
  long responseDurationMillis;
  String activityName;
}

Our interceptor watches all calls & measures them:

public final class MeasuringInterceptor implements Interceptor {  
  final List<CallMeasurement> measurements = new ArrayList<>();

  @Override public Response intercept(Chain chain) {
    ...
  }
}

We have a problem: there’s no obvious way for the interceptor to find out which activity triggered the call. We could cheat by doing pattern matching on the URLs, or by looking at which activity is in the foreground, but both are clumsy and could be unreliable.

It turns out that we can sneak data into an interceptor with a request header! When initiating each request, include a header that contains the data of interest:

Request request = new Request.Builder()  
    .url("...")
    .header

The Play Store is bad at their job

Today Google removed Podcast Addict from the Play Store. Sometimes people publish podcasts containing sexually explicit material, and Podcast Addict is capable of accessing such material.

“Podcast Addict was the #1 podcast app on the Play Store with 4M downloads, 175K reviews and an average rating of 4.6/5 The app almost had 500K daily users and more than 1M episodes were listened to everyday through the app...”

So the Play Store prevents podcast listeners from using a popular podcasting app.

Last week I discovered that my app, Shush Ringer Restorer has an unauthorized clone in the Play Store. That app has the same exact name as my app, which I assume is in an attempt at tricking users into downloading the wrong app. I petitioned Google to take down this lookalike app, with links to all the media attention Shush has received.

They declined because I haven’t registered a trademark:

“Thanks for submitting your trademark complaint, dated 01/04/2015. We will not be taking action against the apps in question at this time as we are unable to ascertain the validity of your trademark rights. If you'd like us to investigate further, please send us evidence

4 Brainy Books

Like books? I don’t have the attention span for reading but I listen to plenty of audiobooks. Here are some of my all-time favorites.

On Intelligence

Non-fiction by Jeff Hawkins

How your brain works.
Amazon, Audible

Daemon

Fiction by Daniel Suarez

In Terminator, Skynet just launches the nukes. In Daemon, the AI is much better: it uses coercion!
Amazon, Audible

Antifragile

Non-fiction by Nassim Nicholas Taleb

Feedback loops and failure.
Amazon, Audible

The Martian

Fiction by Andy Weir

“I'm gonna have to science the shit out of this.”
Amazon, Audible

OkUrlFactory is going away

OkHttp exposes three APIs:

  • Request/Response. This is the most fully-featured API. It can be used synchronously or asynchronously. It avoids mutable state. Plus interceptors!
  • OkUrlFactory, the HttpURLConnection API. This is a complete implementation of an awkward API. The API’s awkardness has resulted in an implementation that is complex and fragile.
  • The Apache HTTP client shim. This is intended to simplify migration. It’s a partial implementation; some methods throw UnsupportedOperationException when invoked.

In OkHttp 3.0, we’re deprecating both OkUrlFactory and Apache HTTP client shim. And we’ll be deleting them in a follow-up release.

If you’re using the Request/Response API, this is good news. It means we’ll be able to significantly simplify the OkHttp internals yielding better performance and stability.

If you’re using OkUrlFactory this is good news. Finally an excuse to shed some of your own tech debt and adopt a better API. Did I mention interceptors?! If for whatever reason you can’t upgrade, you’ll need to rely on the platform’s built-in HttpURLConnection going forward.

And if you’re on the Apache HTTP client shim, this is merely annoying. You can make your own shim by forking ours

com.squareup.okhttp3

We’re about to start working on OkHttp 3.0. It’ll be like OkHttp 2.7, but with small backwards-incompatible API changes. For example, we’re going to make multipart better.

How can we roll this out without breaking everything? As Jake explains, we’re going to cheat! By changing the package name and Maven group ID, you’ll be able to have both OkHttp 2.7 and OkHttp 3.0 in the same app at the same time.

Shipping two copies of a library is a terrible long-term solution! It bloats your app, consumes more memory, and exacerbates the 65K dex limit. But it’ll enable you to update OkHttp in your application independently from its libraries. Preferably you can get everything upgraded by the time you release your app.

Note that our approach is quite different from Guava’s versioning philosophy. Our package renames will require you to find & replace imports – how annoying! But it also means we can change things without a drawn out deprecation period. Like ripping off the band-aid!

We’re doing this for both OkHttp 3 and Retrofit 2. I’m looking forward to both.

A Convincing Argument

I’m always disagreeing.

I think this might be inevitable. The stuff we agree on is boring, and so the conversation accelerates until it slams into something interesting: a disagreement.

  1. With a neighbour: climate change.
  2. With a family member: basic income.
  3. With a colleague: HBase.
  4. With a friend: messaging software.

I’m not sure where I got this from, but I’ve found a handy–though potentially manipulative–way to advance the conversation:

“What would convince you?”

Suppose we’re arguing about whether to use a blue theme vs. a black theme for a new app. Rather than scattering a bunch of reasons why blue is superior, I ask “What would convince you?” inviting you to tell me exactly which evidence I must present.

You’ll either become convincible by providing concrete problems I must address: “Convince me it won’t get lost in the blue sea of Facebook, Twitter and Inbox. Convince me it will be usable by people who are colorblind. Convince me that blue will strengthen our brand.”

I won’t necessarily be able to address these concerns! Maybe I do the research and discover blue won’t strengthen our brand. But now our disagreement is actionable

Don’t use interfaces for values

If you’re defining an Java API, value objects are your friend. And interfaces aren’t suited to that task.

Java gets this wrong all over:

  • Annotations. They use interfaces to define value objects, and consequently it’s quite awkward for tools like Guice to define annotation values. NamedImpl.java is boilerplate hell.

  • Generic type descriptors. Hidden inside each of Guava, Gson, and Guice are mechanical implementations of GenericArrayType, ParameterizedType, TypeVariable, and WildcardType. This wasted code is particularly tragic because Java 8’s TypeVariable interface broke backwards compatibility.

  • Collections. The List and Set interfaces are too big to implement directly, so we have AbstractList and AbstractSet. If List and Set were just abstract classes, Java 8 wouldn’t have needed default methods!

I’m annoyed that this mistake continues to be made. JAX-RS defined its headers type as an interface without a good way to build an instance in a unit test. Instead every application needs its own implementation for testing.

Interfaces are wonderful! They’re useful in all kinds of modeling problems. But for defining value objects, interfaces are a bad fit.

Publishing Javadoc for GitHub Projects

I recently standardized the location of Javadoc for several of Square’s Java open source projects. Here’s a sample:

We post Javadoc for the latest release of each major version. Each artifact’s docs are published separately.

I wrote Osstrich to make this simple & repeatable. It copies Javadocs from Maven Central to GitHub Project Pages.

If you want to follow this pattern to document your own GitHub projects, release to Maven Central and create a gh-pages branch. Then run the following:

git clone git@github.com:square/osstrich.git  
cd osstrich  
mvn compile  
mvn exec:java \  
  -Dexec.mainClass=com.squareup.osstrich.JavadocPublisher \
  -Dexec.args="temp git@github.com:square/moshi.git com.squareup.moshi"

In this example, Osstrich will download Javadoc for the latest com.squareup.moshi artifacts from Maven central and post ’em to the git@github.com:square/moshi.git GitHub project.

FlatBuffers aren’t fast, they’re lazy

Lots of great Android developers have been promoting FlatBuffers.

Miroslaw Stanek recently posted benchmarks comparing JSON, FlatBuffers, and a hybrid that uses both. He mentioned that decoding JSON to FlatBuffers is about 30-40% faster than decoding JSON to Java models with Gson.

Colt McAnlis has been referring Android developers to his video on FlatBuffers, where he explains some FlatBuffers implementation details:

“This layout allows FlatBuffers to be type safe and further allows you to read from the serialized type format without having to do any type conversions, memory allocations, or unpacking during load time which is a huge speed boost”

Note that FlatBuffers don’t do memory allocations during load time. Well, then when does the memory allocation happen?

FlatBuffers allocate every single time you access a non-primitive property of your object. Every. Single. Time. This is bad. It means allocation is likely happening in your onDraw() method, the dangers of this are well-explained by Ian Ni-Lewis in Avoiding Allocations in onDraw().

When you use FlatBuffers in an Android app, you’re moving work from the background I/O thread to the main thread. The best way to avoid this is by avoiding FlatBuffers.

OkHttp, HTTP/2 & NGINX 1.9.5

OkHttp’s HTTP/2 doesn’t interop with NGINX 1.9.5. HTTP requests made from OkHttp to impacted NGINX servers will fail like this:

java.io.IOException: stream was reset: PROTOCOL_ERROR  
    at com.squareup.okhttp.internal.spdy.SpdyStream.getResponseHeaders(SpdyStream.java:145)
    at com.squareup.okhttp.internal.http.SpdyTransport.readResponseHeaders(SpdyTransport.java:104)
    at com.squareup.okhttp.internal.http.HttpEngine.readNetworkResponse(HttpEngine.java:830)
    at com.squareup.okhttp.internal.http.HttpEngine.access$200(HttpEngine.java:95)
    at com.squareup.okhttp.internal.http.HttpEngine$NetworkInterceptorChain.proceed(HttpEngine.java:823)
    at com.squareup.okhttp.internal.http.HttpEngine.readResponse(HttpEngine.java:684)
    at com.squareup.okhttp.Call.getResponse(Call.java:272)

They’ve merged a fix. In the interim, you can either downgrade your NGINX or disable HTTP/2 in OkHttp:

OkHttpClient client = new OkHttpClient();

// Disable HTTP/2 for interop with NGINX 1.9.5.
// TODO: remove this hack after 2015-12-31.
client.setProtocols(Collections.singletonList(Protocol.HTTP_1_1));  

HPACK is what HTTP/2 uses to compress headers. It’s a bit tricky to get right, partially because there are multiple ways to encode headers. In this case OkHttp’s compression strategy triggered a different code path than the major browsers

#enumsmatter

Colt McAnlis has been arguing that Android developers avoid enums because they have a non-negligible runtime cost compared to int constants. In his talk he measures it, and suggests that pervasive use of enums can harm your Android app.

“And the worst part is that you don't really know that enums are causing a problem until they're already infecting your codebase. And by that point trying to fix it is really a horrible process.”

The implied alternative here is a codebase that uses a single type – int – instead of a variety of different enum types. He wants us to go from this:

  public PaymentState cancel(String id, CancellationReason reason);

to this:

  public int cancel(String id, int reason);

That codebase is harder to maintain and I wouldn’t want to work on it! The reason the entire Internet was disgusted by Colt’s advice is that it takes one of the few tools we have to help us to write readable code (types!) and discourages its use.

#enumsmatter

ProtoParser is Going Away

ProtoParser is a simple Java project that parses .proto files. It’s a dumb parser: whenever you reference a type, it lacks any mechanism to resolve that reference.

Wire is a much more sophisticated Java project that builds upon ProtoParser. It performs that essential linking to make ProtoParser’s output useful, and then uses that message graph to generate nice model classes.

I’m in the process of folding ProtoParser into Wire, combining the two projects into one. The result will be a new subproject (called wire-schema) that does both parsing and linking. This is how we should have done it all along. That new API isn't stable yet, but it will be soon.

If you’re using ProtoParser, be warned: we’re halting development on that project. That’ll allow us to move faster on Wire, which will has a better, more powerful schema model.

Reactions to JMH

Today I’m playing with JMH, the Java benchmarking harness written by Oracle. I’m hoping to try out some optimizations for Moshi.

Some History

After years of struggling with one-off benchmarking harnesses, Kevin B and I started Caliper in 2009. It started out as a project for benchmarking Android and Java. It knew about dalvikvm and java, and could benchmark an attached device. Later, Greg Kick took over the project and gave it a better understanding of HotSpot JVM, plus a new web UI.

The JMH project was first published in 2013. It addresses the same problems as Caliper but with even better access to HotSpot internals.

JMH’s Runner Is Very Sophisticated

It has flexible concurrency control, a powerful State abstraction, and a thoughtful API. JMH has some amazing features like @CompilerControl(DONT_INLINE) which is going to be quite fun for me later!

The JMH runner is more precise than Caliper’s. Caliper uses reflection to invoke the benchmarking method; this has lots of bad consequences such as the need for a reps parameter. JMH just code-gens the entry point to avoid interference from reflection. Way better.

No Love For Android

Because of broken history between Oracle

Null

In announcing OkHttp's new URL class, I wrote about how parsing returns null instead of throwing exceptions:

Instead, parse() just returns null when it doesn’t understand what you passed it.

Several developers thought this was a lousy API. Derek Morr tweeted:

"parse() just returns null" so, not sane.

And Alex Hvostov redditted:

Are you fucking kidding me? What is this, PHP?

Well, the method returns null because I think it’s the best tool for the job. Let’s review the options:

Checked Exceptions

I reject this option immediately because it punishes callers who know their input is valid.

Unchecked Exceptions

This is tempting. But now I’m punishing callers who expect that some of the inputs to parse() to fail. For example, OkHttp’s own redirect handler needs to parse an arbitrary HTTP header, which should be an HTTP URL but could be anything.

public @Nullable Request followUpRequest() throws IOException {  
  String location = response.header("Location");
  if (location == null) {
    return null; // No "Location" header? No follow up.
  }

  HttpUrl redirectUrl = userRequest.httpUrl().resolve(location);
  if (redirectUrl == null) {
    return null; // Location header isn't an HTTP URL? No follow up.
  }

  ...

  return requestBuilder.url(redirectUrl).build();

Wrapping this code in try/catch ceremony

OkHttp 2.4.0-RC1 has HttpUrl

Neither of Java’s built-in APIs make it easy to get at a URL’s query parameters. Instead, you end up having to muck around with string concatenation, string splitting, UrlEncoder, UnsupportedEncodingException, and MalformedURLException.

HttpUrl is a new class that makes URLs easy:

   HttpUrl url = new HttpUrl.Builder()
       .scheme("https")
       .host("www.google.com")
       .addPathSegment("search")
       .addQueryParameter("q", "polar bears")
       .build();

The Javadoc goes on and on explaining why Java needs yet another URL class, and why this one is different.

Get HttpUrl in OkHttp 2.4.0-RC1.

It’s ready on Maven Central and eager to simplify your URL-manipulation code.

<dependency>  
  <groupId>com.squareup.okhttp</groupId>
  <artifactId>okhttp</artifactId>
  <version>2.4.0-RC1</version>
</dependency>  

I invite you to try out this release candidate right away. Unless there are surprises we’ll do a final 2.4.0 release soon.

No beer emoji for java.io.Reader

We’re in the age of Emoji. We fought in the charset wars, obsoleted old bad encodings like ISO-8859-1, and emerged victorious. Being able to sprinkle our writing with sprinked donut emoji is the English speaker’s upside to ubiquitous UTF-8.

Unfortunately, java.io.Reader is stuck in the UTF-16 ghetto: all of the hard work of internationalization but without the donut emoji.

An example

Let’s read characters from a Reader until we hit ‘🍺’ (0x0x1f37a), and then we'll stop. The naïve solution doesn't work:

byte[] data = new byte[] {  
    (byte) 0x68, (byte) 0x65, (byte) 0x6c, (byte) 0x6c,
    (byte) 0x6f, (byte) 0xf0, (byte) 0x9f, (byte) 0x8d,
    (byte) 0xa9, (byte) 0x77, (byte) 0x6f, (byte) 0x72,
    (byte) 0x6c, (byte) 0x64, (byte) 0xf0, (byte) 0x9f,
    (byte) 0x8d, (byte) 0xba
};
Reader reader = new InputStreamReader(new ByteArrayInputStream(data)));  
for (int c; (c = reader.read()) != 0x1f37a; ) {  
  System.out.printf("%08x: %s%n", c, new String(new int[] { c }, 0, 1));
}

This crashes because the single codepoint ‘🍺’ is returned in two halves. We miss the beer altogether and run off the end of the string.

00000068: h  
00000065: e  
0000006c: l  
0000006c: l  
0000006f: o  
0001f369: 🍩  
00000077: w  
0000006f: o  
00000072: r  
0000006c: l  
00000064: d  
0000d83c: ?  
0000df7a

Checked exceptions vs. API design

Java has checked exceptions and unchecked exceptions.

Checked exceptions are for expected problems that you should deal with and recover from at runtime. Flaky network? Catch the IOException and deal with it.

Unchecked exceptions are for unexpected problems that should trigger a crash. Is the byte count negative? Got a string when you expected a URL? Don't recover; just crash. In 2015 even crashing is luxurous with cloud services to make failure delightful.

But when designing general purpose APIs, it's not always possible to differentiate unexpected vs. expected problems. The classic example is NumberFormatException. When you accept raw data from a user and Integer.parseInt() fails, you probably shouldn't crash. But if you're parsing a trusted JSON file and the input is malformed, you should give up so the problem can be found & fixed.

Today I'm annoyed because I'm parsing data that will be clean sometimes and dirty at other times. Do I risk crash good programs on bad input? Or do I punish safe calls with needless try/catch ceremony?

Guava works around this in LoadingCache with two methods that only differ in the types of exceptions they throw: get() and getUnchecked(). This both works and is sad

How do HTTP caching heuristics work?

Suppose you’ve requested a webpage with a Last-Modified date and no other caching headers:

HTTP/1.1 200 OK  
Last-Modified: Tue, 16 Dec 2014 06:00:00 GMT

...

The HTTP client will happily store this response in the cache indefinitely. But it won't serve it from the cache unless it’s still fresh at the time of request.

How do we decide whether it’s still fresh? There are three timestamps we’re interested in:

  • Last requested at: Timestamp when we made the last request.
  • Last modified at: What the Last-Modified header said on that response.
  • Now: Timestamp at the time of the current request.

We use the time between last requested at and last modified at to estimate how frequently a document is edited. If a page was modified 5 minutes before the request, it’s assumed to be frequently modified. If it was last modified 5 years before the request, it’s assumed to be infrequently modified.

A page is fresh for 10% of that duration: 10% of 5 minutes is 30 seconds; 10% of 5 years is 6 months. A page is considered fresh until that 10% has elapsed since the document was last requested.

An

Tracking OkHttp Download Progress

Anton Rieder just contributed an OkHttp code sample that monitors a download’s progress using an interceptor. With it, you get a callback as bytes arrive:

  interface ProgressListener {
    void update(long bytesRead, long contentLength, boolean done);
  }

I like this example because it’s all layered decorators:

  • The Interceptor puts code between application-and-OkHttp, or OkHttp-and-network.
  • The ResponseBody augments the response body to support progress tracking.
  • The Source does the work: updating the progress listener as bytes are delivered from network to application.

In each case, the adaptable interface is deliberately limited. Decorating okio.Source is easy! To contrast, InputStream is hostile to decoration.

OkHttp 2.3 has HTTP/2

Lately I've been building HTTP APIs for Square Cash, to be used by our website and partners like Snapcash. It turns out this is a particularly difficult task because I need to balance two competing concerns:

Keep separate things separate. If the app wants to look up payments 123, 246 and 369, it should make three requests for /payments/123, /payments/246, and /payments/369. This is simple and manageable, but it means both client and server need to pay HTTP call overhead for every payment. That slows down the experience and increases our datacenter load.

Batch things up. REST APIs are nice and all, but what's going to run fastest is a aggregating everything into a single call. Perhaps it’s /payments-list/123,246,369 and now there’s more APIs to worry about, but fewer calls.

Since /payments-list/123,246,369 doesn’t quite identify a resource, batching things also makes it more difficult to use HTTP caching. If the browser later wants to load payments 246 and 482, the URLs won’t match and we have to do our own caching in the application layer.

Enter HTTP/2

By significantly lowering the per-call overhead, we can use

Okio 1.3 has segment sharing

Okio’s buffer reads like an InputStream: when you read 50 bytes from the buffer, you implicitly remove those bytes from the buffer. This is a handy optimization and it means we can move data rather than copying it by default.

But sometimes you need non-destructive reads. So in Okio 1.3 we've changed the Buffer.clone() method to do segment sharing. Previously, cloning a 2 megabyte buffer would require 2 megabytes of memory to be allocated and copied. With segment sharing, only the bookkeeping data is copied, which is about 3% of the size of the buffer’s data.

To read a buffer without consuming it, just create a (cheap) copy and read that. And our fundamental optimization is preserved: reading from the clone will move data rather than making even more copies.

Okio 1.3 also adds some nice ergonomic features to ByteString:

  • URL-safe Base64 encoding
  • Handy MD5 and SHA-256 hashes
  • Substrings!

There’s also new APIs in BufferedSource and BufferedSink:

  • Read and write signed decimal values without the string intermediates that you'd need with Long.valueOf(String). We can use this in DiskLruCache’s journal file, among other places.
  • Read and write unsigned hexadecimal values, also without

Custom Java Trust Stores without Bouncy Castle

Java’s crypto APIs are flexible and powerful. But just awful to use. There are too many contexts, managers, factories, and providers. Want a certificate? Take your pick!

Gross.

Suppose you want to replace the host platform’s (scary) set of trusted certificate authorities with your own trimmed down set. This happens if you want to do HTTPS without paying a certificate authority each time your configuration changes. (You’d use certificate pinning if you wanted to avoid trusting all certificate authorities).

Most of the online advice I've seen suggests that you need Bouncy Castle and/or keytool to first load the PEM files into a JKS or BKS keystore, which you then use to build an SSLContext. That's annoying because it adds another opaque binary file to an already confusing process.

Fortunately, you can skip that step and read the PEMs directly. Hidden in plain sight within the certificate APIs is a mechanism to build an SSL socket factory from a PEM file like this one:

-----BEGIN CERTIFICATE-----
MIIFVTCCBD2gAwIBAgIRAKgHBM+t9Yx3v9G9tGZECWkwDQYJKoZIhvcNAQELBQAw  
b21haW4gQ29udHJvbCBWYWxpZGF0ZWQxFDASBgNVBAsTC1Bvc2l0aXZlU1NMMRkw  
FwYDVQQDExBwdWJsaWNvYmplY3QuY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8A  
oqEld6bOSoMlYavj9GCBSNIx2+mGS0Tg6A==  
-----END CERTIFICATE-----

OkHttp's CustomTrust example has

proto3 + JSON are a great match

On Cash we communicate with the web and mobile apps via both protocol buffers and JSON. For both data formats, we use proto2 as a schema language. It defines what our message types are, what their fields are, and gives us a language-neutral place to document all the things. This approach is very lowest-common-denominator: we can't use features that JSON lacks (default values, bytestring values) nor those that proto2 lacks (map types). It’s imperfect but productive!

Google recently released an alpha of proto3, which is very suited to our use case. It sheds features that JSON can't express and adds a map type. This is fantastic.

Here's a sample of proto3 output if you're curious. The code generated by the alpha looks & works like the code from proto2.

I think the Wire generated code is still the best API for most projects. We'll add support for proto3 once the syntax & feature set is final.

Droidcon Montréal, April 9–10

I'm proud to share the keynote stage with Jake at Droidcon Montréal this April. We're planning to talk about open source. That’s a broad topic, and one I’ve presented on previously.

But it isn’t just about bug reports and pull requests. I’m thinking a lot about what Ashe Dryden said in The Ethics of Unpaid Labor and the OSS Community:

“OSS contribution takes time; I don't think anyone would contest that. Getting familiar with a project, finding out where you can fit into it, reading and responding to issues, testing and submitting patches, writing documentation. All of that requires a good deal of time.

Marginalized people in tech - women, people of color, people with disabilities, LGBTQ people, and others - have less free time for a few major reasons: dependent care, domestic work and errands, and pay inequity.”

Yikes. I’ve been doing open source for a decade, but only because I’ve had the privilege of time & resources to do so.

It’ll be an interesting conversation, and it’s happening at Droidcon Montréal April 9–10. Early bird registration ends January 31!

Fancy Flow Control: Did it throw?

Exceptions are powerful control flow: mainline code handles the mainline cases, and exception cases can be handled elsewhere. But sometimes you want to observe whether an exception occurred, without necessarily handling that exception.

For example, suppose we're writing a service that sends webhooks to a far-away webserver. Webhooks are naturally flaky: there are usually many target servers, each managed a different team with a different expectation of availability and correctness.

Our method wants to run webhooks and keep statistics on their results:

public int sendWebhook(String url, String request)  
    throws IOException {
  try {
    int statusCode = httpPost(url, request);
    if (statusCode >= 400) {
      failureCount++;
    }
    return statusCode;
  } catch (IOException e) {
    errorCount++;
    throw e;
  }
}

What happens if httpPost() throws an unchecked exception, like WebServiceException? Well, we can handle that too. And also Error so that assertion failures aren't dropped:

public int sendWebhook(String url, String request)  
    throws IOException {
  try {
    int statusCode = httpPost(url, request);
    if (statusCode >= 400) {
      failureCount++;
    }
    return statusCode;
  } catch (IOException | RuntimeException | Error e) {
    errorCount++;
    throw e;
  }
}

This works. But it's unfortunate to list the three different Throwable types that can pass-through this method, especially if those types could change. Should we add another checked exception like URISyntaxException, it's easy to lose

Okio Watchdog

I was running a jstack dump on my project's app server this afternoon & I saw Okio Watchdog among the threads:

"Okio Watchdog" #1972 daemon prio=5 os_prio=0 tid=0x00007f4a8c01e800 nid=0x362 in Object.wait() [0x00007f48f2351000]  
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at java.lang.Object.wait(Object.java:502)
    at okio.AsyncTimeout.awaitTimeout(AsyncTimeout.java:297)
    - locked <0x0000000733eaa0e0> (a java.lang.Class for okio.AsyncTimeout)
    at okio.AsyncTimeout.access$000(AsyncTimeout.java:40)
    at okio.AsyncTimeout$Watchdog.run(AsyncTimeout.java:272)

This is a thread Okio uses to implement fancy timeouts for OkHttp.

The JDK's Network Timeouts

There are two timeout mechanisms in Java's networking stacks:

Notably, there's no API for write timeouts on blocking I/O. Oracle advises you switch to NIO if you need that.

Okio Timeouts

Okio uses a watchdog thread to implement read & write timeouts for blocking I/O. When the watchdog sees that an operation has gone on too long

Posting Images on GitHub's wiki

GitHub's wiki permits embedded images, but offers no UI to upload your own images. Here's how to do it from the command line.

Clone the Wiki's Git Repo

GitHub's wiki is itself a git repository. Clone it to your desktop using an SSH URL to make git push just work later. (This assumes you're already configured SSH keys with GitHub).

Here I'm editing the wiki for square/okhttp. Substitute square with your GitHub username or organization, and okhttp with your GitHub project.

git clone git@github.com:square/okhttp.wiki.git  
cd okhttp.wiki/  

Add the image

There's no pull requests or other process here. Just push master once you've added the image.

mv ~/Desktop/interceptors@2x.png ./  
git add interceptors@2x.png  
git commit -m "Add interceptors diagram."  
git push  

Embed the image

Raw wiki resources are web-accessible at raw.githubusercontent.com. The URL for my image is here and the corresponding markdown is this:

 ![Interceptors Diagram](https://raw.githubusercontent.com/wiki/square/okhttp/interceptors@2x.png)

I edited my wiki on the web; editing it in the local git repository also works. Just don't forget to push.

Enjoy