It seems that the problem not in API itself.

The first reason maybe some kind of "sleeping" software which have to wake up and then works for some short time and fall asleep again.

And the second - your networking equipment which looses path to the server and each time spend these 4 seconds to find the network device.

As for http - it must work even faster than https