To get a clean, working pip install on Ubuntu 24.04 I needed to use:

CMAKE_ARGS="-DGGML_CUDA=on -DCMAKE_CUDA_COMPILER=`which nvcc`" FORCE_CMAKE=1 pip install --upgrade --force-reinstall llama-cpp-python --no-cache-dir

Which may be due to self-inflicted pain with my nvcc but I didn't intentionally do anything exotic.