Are you using next-auth in your project? if yes then the might be in next-auth and not nextjs. By default next-auth refreshes session each time the window or tab gets focussed. to disable this behavior of next-auth, find SessionProvider in your project and set the "refetchOnWindowFocus" attribute to true like this:

<SessionProvider refetchOnWindowFocus={false}>

This solved the issue for me. There are other useful attributes for SessionProvider as well through which you can even better control session refresh.