As indicated in my comment to @Quinton.Quagliano, the use of Mathjax version 3 allows showing math in figure labels in the HTML file but messes up the figures in the jupyter notebook, which is the original format I work with. The figures lose their interactivity and the labels completely disappear due to that!
Here is a hack I found to do what I want (Cannot explain why it works though...):
Here is a qmd code showing what the notebook content looks like.
---
format:
html:
embed-resources: true
execute: true
jupyter: python3
---
```{python}
from IPython.display import display, HTML
# Run this first
display(HTML(
'<script async src="https://cdn.jsdelivr.net/npm/mathjax@2/MathJax.js?config=TeX-AMS_SVG"></script>'
))
# Run this afterwards
# display(HTML(
# '<script async src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-svg.js"></script>'
# ))
```
```{python}
import plotly.express as px
fig = px.line(x=[0, 1, 2, 3, 4],
y=[1, 2, 4, 8, 16],
)
fig.update_layout(yaxis_title = '$2^x$',
xaxis_title = '$x$')
fig.show()
```