It seems like that the errorHandler doesn't record in the IdempotentRepository that the retries have exhausted and, therefore, the upload of the files is retried over and over again.

Kudos to https://stackoverflow.com/a/45235892/5911228 for pointing this implicitly out.

Changing the exception handling to

onException(Exception.class)
    .maximumRedeliveries(maxRetries)
    .redeliveryDelay(max(0, config.getRetriesDelayMs()))
    .handled(true)
    .logHandled(true)
    .logExhausted(true);

solved the issue.