I was be able to solve this by changing starlette version. I had:
fastapi==0.101.1
starlette==0.46.0
It turns out they were incompatible.
After changing to
starlette==0.27.0
everything works fine.