I had the same issue on my Apple M4. I just installed MacOS 15.4 (beta, 24E5238a) and the problem disappeared.
https://developer.apple.com/forums/thread/775833 hints that Apple actually fixed something.