I found that skipping the index can actually help in this case.

select count(*) from "default" NOT INDEXED

> CPU times: user 637 ms, sys: 2.16 s, total: 2.79 s Wall time: 11.3 s

Is still slow, but I get and improvement from 28s to 11s