I finally got it working! It's 2:44am here now and we're two weeks of fucking around later :-P

I don't known why, but I started getting in trouble with the VirtualizingWrapPanel constantly flickering up to the point it became unusable and when I breaked the program I saw in the stack trace that it was constantly measuring and arranging. So I reckoned it had trouble sticking to a constant item size so I set the ItemSize property of the VirtualizingWrapPanel to something constant. And now all of a sudden all of my problems are gone and everything is working:

<ListView.ItemsPanel>
    <ItemsPanelTemplate>
        <vwp:VirtualizingWrapPanel 
            SpacingMode="None" 
            Orientation="Vertical" 
            ItemSize="240,20">
        </vwp:VirtualizingWrapPanel>
    </ItemsPanelTemplate>
</ListView.ItemsPanel>