Indeed, as Tim Roberts pointed out in comments, the problem was that av_frame_clone does not allocate new memory. Instead, I create a completely new frame or call av_frame_unref on the old frames. This solved my problem.