Just as a workaround, create an SSH tunnel between the remote host (where Minikube is running) and the client host. Then simply use https://127.0.0.1:<APISERVER_PORT> in the Kubeconfig.
https://127.0.0.1:<APISERVER_PORT>