Increase retry_after to at least 2–3× your timeout, e.g.:
'retry_after' => 180,

Use the same queue name in your Notification:
$this->onQueue('reminder');

Make sure only one queue worker handles the same queue name.

If mails are slow, switch to Mail::queue() with a dedicated mail queue.