webrightnow's answer led me to the solution.

For me, product reviews were disabled while I was creating the majority of my listings and for some reason the review section didn't appear for these products when I enabled it globally later, even though it did work for products that I've created after enabling it.

Enabling reviews on the edit product page of these products didn't work either, BUT it seemed to work for me when I clicked the "Quick Edit" in my Products page for my product and enabled it there.