I finally found the problem...
The trigger had no problem at all. The issue came from our UPDATE script which was badly written and was updating all rows even if no modifications were done...

The trigger was then updating the "ModifiedTimeStamp" for all rows, which was perfectly right.