Thanks to @Anton Malyshev for guiding me to the answer. For those who don't want to click on a link and hunt down the answer (and in case the link disappears in the future), here's the solution.

In Android Studio, Preferences -> Advanced Settings -> Tool Windows (scroll waaaay down!) and make sure that "Always show tool window header icons in the new UI" is checked. Here's a picture.