I didn't encounter a crash in the following two cases:

Setting minSdkVersion to 24 or higher

Explicitly overriding the default method in the class that implements TestListener

In addition to those two, as I learned from this answer, enabling minifyEnabled also resolved the issue.