Thanks to programmer for their question/answer and derek-mccallum for their answer as well. The Java interfaces for this task are not intuitive at all and their content saved me some time.

Here is a more succinct example and the code is available on GitHub here.

static java.security.PublicKey toJavaPublicKey(byte[] publicKey, int off, int len) {
  byte[] reversed = ByteUtil.reverse(publicKey, off, len);

  int last = reversed[0] & 0xFF;
  boolean xOdd = (last & 0b1000_0000) == 0b1000_0000;
  reversed[0] = (byte) (last & Byte.MAX_VALUE);

  var y = new BigInteger(reversed);
  var edECPoint = new EdECPoint(xOdd, y);

  var publicKeySpec = new EdECPublicKeySpec(NamedParameterSpec.ED25519, edECPoint);
  try {
    return ED_25519_KEY_FACTORY.generatePublic(publicKeySpec);
  } catch (InvalidKeySpecException e) {
    throw new RuntimeException(e);
  }
}