I had the same issue. Turns out Vite was picking up changes to the Apache log files. So the solution was to move them out of the umbrella of the application. I know I could have set them up as external, but now I have all my projects dump their logs into a central dir, which makes life a little easier. It took a while to discover this, in the end I set up a script:

"debug": "vite --debug hmr"

in package.json which ultimately gave the game away.