Thanks for LittleDuck sharing the root cause.

I have faced the same issue as d4redevil, my VideoToolBox generated data is

SPS.pic_order_cnt_type = 0, 
VUI.max_dec_frame_buffering not exits in SPS.

Solution 1:

on Chrome you could set VideoDecoder to prefer-software to get 1 in 1 out. But does not work for Safari.

Solution 2, (I used):

I end up solving it by manipulating the raw SPS data. I parsed the original SPS data, and updated vui_parameters_present_flag = 1, and carefully appended about 5 bytes of VUI data at the end of SPS(right after bit position of vui_parameters_present_flag). The VUI data contains this key value to avoid decoder frame buffering.

VUI.max_dec_frame_buffering=1

Also on VideoToolBox side, I used this autoLevel

kVTProfileLevel_H264_Baseline_AutoLevel: CFString