Found the answer while in the middle of submitting the question.

1. Ensure to run dotenvx decrypt to not lose secrets
2. Remove the keys (back them up just to be safe)
   • DOTENV_PUBLIC_KEY from .env
   • DOTENV_PRIVATE_KEY from .env.keys
3. Run dotenvx encrypt - Solves the problem, now there is a new public and private key pair
4. Cleanup .env & .env.keys from extra generated comments due to new key pair