The problem appears to be that I wasn't using the same keystore as previous versions of the app. I ran keytool -list on it to make sure I was using the correct alias, which I wasn't, and that recommend to migrate it to PKCS12 by using the command:

keytool -importkeystore -srckeystore keystorename.keystore -destkeystore newkeystorename.keystore -deststoretype pkcs12

That migrated version imported into Visual Studio without errors, and worked in the Play Console's Play App Signing page.