I've managed to find the right combination of configuration to get this to work.

Retries work as expected - either automatic or explicit - and the Reject exception can be raised either in the task or in the on_failure handler.

If a retry is triggered in the on_failure handler then the Reject exception isn't handled in the same way and the message doesn't get routed to the dead letter queue.

I've created a full example here:

https://gist.github.com/grahamlyons/5e7053e5fc9e56bec0cb62aca4232991