The most accurate way to calculate is to store the NTP value in the RTCP reports that clients send to the server every few seconds.