My working implementation boils down to two things:

1. Set clockSkew to 0 s.

OAuth2AuthorizedClientProvider authorizedClientProvider = OAuth2AuthorizedClientProviderBuilder.builder()
                .clientCredentials(
                        clientCredentialsGrantBuilder -> clientCredentialsGrantBuilder.clockSkew(Duration.ZERO))
                .build();

2. Add retry mechanism in case of 401:

public class WebClientRetryHelper {

    public static Retry retryUnauthorized() {
        return Retry.fixedDelay(1, Duration.ofMillis(500))
                .filter(throwable -> {
                    if (throwable instanceof WebClientResponseException ex) {
                        return ex.getStatusCode() == HttpStatus.UNAUTHORIZED;
                    }
                    return false;
                });
    }
}