Turns out, it's because of new version of flask-session is not compatible with the old version of airflow. I limited my flask-session < 0.6 and it works just fine!