If I understand your requirements, you want the @Retry annotation to actually (skip retrying) on specific types of exceptions.
So you are looking for abortOn. https://download.eclipse.org/microprofile/microprofile-fault-tolerance-4.0.2/microprofile-fault-tolerance-spec-4.0.2.html#_retry_usage
/**
* In case the underlying service throws an exception, it will be retried,
* unless the thrown exception was an IO exception.
*/
@Retry(retryOn = Exception.class, abortOn = IOException.class)
public void service() {
underlyingService();
}