haz un debug a D3DXMatrixLookAtRH seria algo como
D3DXVECTOR3 forward = eye - at;
float forward_len = D3DXVec3Length(&forward);
D3DXVECTOR3 cross_test;
D3DXVec3Cross(&cross_test, &up, &forward);
float cross_len = D3DXVec3Length(&cross_test);
OutputDebugStringA(
("forward len = " + std::to_string(forward_len) +
", cross len = " + std::to_string(cross_len) + "\n").c_str()
);
Si obtienes:
forward len = 0.0 → eye == at
o
cross len = 0.0 → up paralelo a forward
Creo que ya encontrarias el problema. espero haberte dado una pista. un salud.