You need to run everything with GPU image

https://milvus.io/docs/install_standalone-docker-compose-gpu.md

https://milvus.io/docs/install_cluster-helm-gpu.md

You can adjust the amount of memory.

adjusting the GPU memory settings in the milvus.yaml

Also in version 2.4.14 and beyond there is a fix that makes sure it just works with no warnings or issues. So wait for the next release or adjust accordingly.