The new version of gitk included with Git 2.51.0 allows hiding these.
Go to Edit -> Preferences, and in the "Refs to hide" box enter prefetch/*.
prefetch/*