Ok i was able to resolve it using java 17 by changing some configurations.

I used scala 2.12.15 and updated sparkTestsVersion to 1.1.0 (this helped solve the ReflectiveOperationException)

As for the java options, I didnt find a good way of setting this in build.sbt, so I just added it as a step in git actions as following:

  - name: Set JAVA_OPTS
    if: ${{ inputs.JAVA_VERSION == '17' }}
    run: echo "JAVA_OPTS=--add-exports=java.base/sun.nio.ch=ALL-UNNAMED" >> $GITHUB_ENV