In this situation 50-60ms is normal latency. However if you want to increase performance you may use JWT auth instead of Basic Auth.
Why basic auth increase latency?
Basic auth may introduce latency increase because credentials are sent to server in Base64 encoded string, server has to decode and validate credentials. Using JWT will reduce the performance overhead as server will not have query database again and again significantly reducing the latency. However you may use a workaround if you don't want to use JWT, on startup load all users in some static Map, you may use username as key and User Model as value, this will also help in reducing latency.