In my case, it was the same the Docker desktop installation installed on Windows and removed afterwards as discussed by @Sebastian Bujak which led to non-existent symlinks. The fixed mentioned above did not work for so I used this github comment for anybody else who had the same problem as me. I could not add this to the comments.
https://github.com/docker/buildx/issues/262#issuecomment-1744354626
Quoting the solution here.
In fact /usr/local/lib/docker/ contained only invalid symlinks.
So, running this command fixed it: sudo rm -fr /usr/local/lib/docker/