What you describe is exactly how I do it. It is convenient to store the key file in a source control system, so that newly generated key files can be deployed with code, as old ones expire. Usually you don't want your secrets unencrypted in source code, so encrypting them with a certificate gets around that problem. The X509 certificate can be maintained by our IT group and installed on servers as they come up, or kept in our cloud vendors' secrets vault.

The certificate is only used to house the PEM (encryption key) on your system. You can generate a PEM using any encryption utility like openSSL, and import it into an x.509 certificate using your OS' certificate utility. This is why it doesn't need to be signed by an authority, because you aren't using it to establish trust with a 3rd party but to hold a secret that you yourself created.

If you were configuring the key from a separate source than the rest of your application, it may not be important to encrypt it and you can just ignore the warning. But that is usually a hassle since key files need to be maintained and kept current, and different keys go with different applications, etc.