Fixed by setting codec to libx264 and max B-frames to 0:

ffmpeg -stream_loop -1 -re -i ./video.mp4 -c:v libx264 -bf 0 -rtsp_transport tcp -f rtsp rtsp://localhost:8554/output