Add the relaxed constexpr flag:
CUDA suggests using the --expt-relaxed-constexpr flag. You can add this in your CMakeLists.txt before you build OpenCV. For example:

set(CUDA_NVCC_FLAGS "${CUDA_NVCC_FLAGS} --expt-relaxed-constexpr")

After that, re-run CMake so it regenerates the Visual Studio solution.

Check CUDA/cuDNN versions:
You’re on CUDA 12.9 and cuDNN 8.9.7, but it’s worth checking NVIDIA’s compatibility matrix for your GPU. Sometimes, even if versions look compatible, there can still be hiccups. If nothing works, try rolling back to a slightly older CUDA release (say, 12.3) along with a matching cuDNN version—this often solves hidden issues.

Consider OpenCV version:
OpenCV 4.12.0 is a bit dated. Newer releases (like 4.9.x or even OpenCV 5) tend to have better CUDA support, especially for the latest GPUs. Upgrading both OpenCV and opencv_contrib might prevent these conflicts altogether.

Verify CUDA installation:
Make sure your CUDA Toolkit is fully installed and your environment variables are set properly (CUDA_PATH, CUDA_PATH_Vxx_x, Path on Windows, etc.). Leftovers from old CUDA installs can sometimes trip up the build.

Do a clean rebuild:
Whenever you change build flags or environment settings:

• Delete your existing build folder.

• Re-run CMake from scratch.

• Open Visual Studio, clean the solution, and then rebuild.

Try lowering optimization:
In Visual Studio’s project settings, reduce the compiler optimization level (under C/C++ → Optimization). Sometimes aggressive optimizations cause CUDA code to fail in odd ways.

Keep an eye on Blackwell issues:
Since you’re working with the new RTX 50-series (Blackwell), there could be some growing-pains with CUDA or OpenCV that aren’t widely documented yet. Checking the NVIDIA developer forums or the OpenCV GitHub issues page might reveal others facing the same problem.