Looking at Kotlin generic class to math numbers.

There seems to be no way to do what you want. As the math operators have no relation to Number.

The only thing you can work with is the toDouble / toLong / .. operations. So indeed like you said the only solutions you have is typecasting it.