This was a bug in exoplayer 2.11.1 with certain video formats. It was solved when upgrading exoplayer to 2.13.3 or higher.