If you open the html file in the browser, the \mathcal symbol shows normally. Maybe it's just R studio's page problem.