It's not just you.

I set up CLASP today, and ran into the same issue. I was vexed, and stumbled on your question.

TL;DR: There's no way to go from .gs (.js) -> .ts

Because of the loss of specificity when you compile a TS file into JS, it becomes an irreversible operation. And because CLASP uses ts2GAS to compile prior to uploading your file, only the JS version is sent to Google drive.

But what if I work in a team?
I think the CLASP architects expect you to use git (or some other version control) and store it in an external repo. Because the TS file is not stored in drive, there is no way to use that as your repository.

In my case, I'm working solo, so keeping it in a git repo on my machine is sufficient. (I'll add it to GitHub for a clean, versioned backup of the TS).

Also a shoutout for using git if you are working with a team on GAS:
From what I understand, CLASP updates are destructive. If someone is edited a file and you push without pulling: Boom. Gone.
Someone edited a file and you pull with conflicting changes: Boom. Overwritten.

Solution? Everyone uses git.