In particular I am interested in a time that doesn't have skips or doubles.

What about CLOCK_MONOTONIC or CLOCK_MONOTONIC_RAW?

They give you the most direct access to your system's clock that I know of. The value represents the amount of "what your system thinks is a second" since boot.

I'm wondering is there a similar format to Unix time that also includes leap seconds and has no special cases at all? In other words, one that simply measures the number of SI seconds that have passed since an arbitrary reference point.

That reference point would be normally the time your system has booted.

If you want to persist across reboots, I'd use TAI, as @awwright suggested in the comments. You can also pass it to clock_gettime(), like the other two options. Maybe you also want to look into linuxptp and how to synchronize your device time to a GPS signal here or here, to get a very precise clock.

To put my musings into perspective: for audio clocks it's a big no-no when your clock shifts or jumps by a few ms, that's why we're using CLOCK_MONOTONIC_RAW and the device and/or PTP in case multiple devices need to be in sync.