As @Dogbert pointed out, use r a in magit status buffer.
And if you have other keybindings like evil, you could type M-x magit-rebase-abort, which is the execution command of r a.