There was a warning when creating the keystore:

Warning: Different store and key passwords not supported for PKCS12 KeyStores. Ignoring user-specified -keypass value.

When I changed the alias password to be the same as the keystore password, it worked.