So it looks like it has to do with the serverless tier. I have tested a few different things and having serverless always hurts the latency. The documentation is a bit sneaky where they say that the point reads are <10 ms for both, but the provisioned are covered by an SLA (service level agreement) and the serverless is covered by a SLO (service level objective).

Moving to a provisioned tier gives 2-4 ms response times, with a few exceptions. Like promised.