I dug around in the nvim logs present in the "nvim-data" directory. I noticed there that the crash was due to the pyright LSP which was using python 3.12.

Here was the problem, the python 3.12 installation was done using the Microsoft store and that was causing the problem. I uninstalled python using MS store and reinstalled it using the official python installer from python.org .

This fixed the crashing issue, though I am still unsure of what exactly was different in the MS store version that was causing the crash.