Arhg! My desk object was marked as static. No idea why! Maybe that is how I downloaded it from internet? Or clicked by accident myself? After unchecking static, the object moves with box collider.
This question helped me

Animation are moving the box collider and not the object in unity