Turns out, that this is a known bug: https://youtrack.jetbrains.com/issue/IDEA-361235/Test-classpath-is-incorrect-when-dependencies-with-classifiers-are-specified