Thanks to onlyIf hint from @Cisco I managed to create this helper method which now lives inside my convention plugins:

fun allExistingArtifactChecksumsInRepositoryMatch(
    repository: MavenArtifactRepository,
    publication: MavenPublication,
): Boolean {
    //...
    return when (val repositoryScheme = repositoryURI.scheme) {
        "http", "https" -> {
            // Create the HttpClient and use it to fetch existing checksums
        }
        "file" -> {
            // Use Path objects to do the same thing
        }
        else -> {
            // Maybe add support later, if necessary
            println("Unsupported repository scheme $repositoryScheme")
            false
        }
    }
}

After fetching the existing checksums, compute checksums of new artifacts using DigestOutputStream and compare them.

A somewhat important detail seems to be to do this:

val artifactsToCheck = when (publication) {
    is MavenPublicationInternal -> publication.asNormalisedPublication().allArtifacts
    else -> publication.artifacts
}

This downcast to internal gradle API seems to be necessary to also check metadata artifacts. This is valid for gradle version 8.13.