Adding

    def scrollContentsBy(self, dx, dy):
        super().scrollContentsBy(dx, dy)
        self.viewport().update()

to the view solves it. The explanation can be found in the comment by musicamante.