I know this is a really old thread but the answer is a simple modifier of
.allowsHitTesting(false)
on the Map object.
This disables all Map interaction.