I had the same question and it turns out that the figure needs to be in it's own paragraph (preceded and followed by a newline and two spaces) in order to show the caption, as described here.