Update: It turns out the slowness seems to have been caused by my ad blocker extension (Adblock Plus) (Thank you Why chrome cached requests are taking time?), after disabling it, the site loads perfectly fine.

(It also turns out this was happening to other cached sites I had on my computer.)