Looks like Overlay's are improvised in recent versions of Emacs since this question was posted. The recent versions of Emacs doesn't seem to have this issue anymore.

You can refer to the following links for more reference:

• EmacsConf 2023: Improving compiler diagnostics with overlays - Jeff Trull

• The "noverlay" branch was merged to master. This should speed up overlays in emacs