Somewhat counter-intuitively, you have to double-click anywhere within the timeline. This should deselect/unselect it so that the limited selection is cleared.