Disabling the willReadFrequently canvas context flag fixed the issue and paints do not take 30ms anymore.
willReadFrequently