You were not doing anything wrong, this was an implementation limitation.
Your example compiles successfully since Scala 3.4, thanks to the changes introduced in SIP-56: "Proper specification for match types" (text, PR).