Precision loss you described is expected. Always normalize quaternions when using "Eigen" library.
To optimize transformations multiplication of the form "t1.inverse() * t2" in terms of precision, you can implement a custom function t1.inverseMul(t2).
qr = q1.inverse()q2; pr = qr(p2-p1);