To answer my own question, it seems like buildSrc modules share the same classloader with gradle and somehow lead to dependency leaks that end up triggering internal compiler errors. So the solution I chose to go with was to create a separate module (i.e. build-logic) and then include that into my project, that way the build-logic module gets its own classloader and such issues are avoided.

So the structure looks something like:

.
├── build-logic/
│   ├── build.gradle.kts
│   ├── settings.gradle.kts
│   └── src/main/kotlin/com/package/specifics/JacocoCoveragePlugin.kt
├── app/
│   └── build.gradle
├── settings.gradle.kts
└── build.gradle.kts

And then in the root project's build.gradle.kts file, I just include it via includeBuild("build-logic").

If anyone has knowledge of the specifics please share it!