Writing an answer as I cannot comment yet...
Usually when refreshing the access token libraries issue a new refresh token along with the new access token, and invalidate the previous refresh token. So, probably you'll need to replace the stored refresh token as well.