Figured it out. As of today, the precompiled binaries for pyarrow are not yet available for Python 3.14 on wheel. Either I build it by myself or install a nightly release. Installed a nightly release and it unblocked me.

pip install -i https://pypi.anaconda.org/scientific-python-nightly-wheels/simple pyarrow