Thanks to @IanAbbott in the comments, I now understand why waking 1 waiter would be incorrect. Assuming a semaphore with 2 waiters (and thus a count of -1), here is how sem_post waking only 1 waiter, followed by another sem_post, would behave:
poster waiter 1 waiter 2
in sem_wait() in sem_wait()
sem_post() called
sem->count = 1
previous count -1
=> wake 1 waiter
gets woken up
sem->count = 0
returns from sem_wait()
...
sem_post() called again
sem->count++ (so 1)
previous count 0
=> no futex_wake
Waking all waiters ensures that the waiter that fails to grab the semaphore will decrement it to -1 again and not leave it at 0, indicating there are still waiters to be woken up on the next sem_post.
I should also note that it would not be correct to do the less naïve change of waking 2 (resp. N) waiters, since the same situation described above could arise again with 3 or more (resp. N+1 or more) waiters, if an additional (resp. N-1 additional) sem_post races its way before the woken up waiters attempt to grab the semaphore.
Implementations such as musl and glibc seem to implement semaphores differently, with a separate waiters count, and are thus able to wake only 1 waiter in sem_post when possible (i.e. in the absence of concurrent updates, I assume):