Some of my assumptions were incorrect. The tests were slowing down, but the culprit wasn't memory creep, but an issue with Robolectic and Compose.

https://github.com/robolectric/robolectric/issues/9043

In the end, creating a junit rule that resets the AndroidUIDispatcher's Choreographer solved the issue.

Here's a gist for junit rule: https://gist.github.com/johngray1965/24d7a3f1e5ae5f0fc1adc24444fe12ac

Note: it was very important that the rule runs before the compose rule (otherwise the compose rule may fail).