No, you cannot get by with just the offset. The math is complex, but in a nutshell, the delay is used to determine the best (offset, delay, dispersion, time) sets from the arriving packets to use to discipline the clock. The only/best place I've found that explains this well is in section 3.5, "Clock Filter Algorithm" on page 43 in the "Computer Network Time Synchronization: The Network Time Protocol" book by Dr. David L. Mills. As of this writing, a later version of this book can be found online here: http://lib.uhamka.ac.id/file?file=digital/47911-eBST-11030034.pdf. In this version, the relevant section is 3.7 on page 48. https://www.eecis.udel.edu/~mills/ntp/html/filter.html also explains it, but not as well. The SO link might help with the understanding, as well: How does NTP Clock Discipline work?