Yes, Airflow currently only supports SqlAlchemy <2.0. You can see here in the constraints they recommend using when installing Airflow.
To get around Airflow constraints, you can use the PythonVirtualEnvOperator or the ExternalPythonOperator.