Fixed! The trick to using @dave_tompson_085's excellent answer was to get the PrivateKeyEntry in order to get the full private key. In addition, I changed all instances of PEMWriter to JcaPEMWriter.

With that I used the following code to successfully write the private key:

KeyStore.ProtectionParameter protParam =
        new KeyStore.PasswordProtection(secret.toCharArray());
KeyStore.PrivateKeyEntry pkEntry =
        (KeyStore.PrivateKeyEntry) certificate.getEntry(alias, protParam);
PrivateKey key = pkEntry.getPrivateKey();
writer.writeObject(new JcaPKCS8Generator(key, null));