I was able to solve it by adding the mingw complier as the compiler in compile flags in the clangd config.yaml file.

Here is a the detailed medium post I made: https://medium.com/@adarshroy.formal/setting-up-neovim-on-windows-a-beginner-friendly-no-nonsense-guide-with-cpp-clangd-without-wsl-f792117466a0