I had a similar issue and in my case the problem was stemming from a default "django.contrib.sessions.middleware.SessionMiddleware" implementation. It seems that this vanilla implementation looks for session_key only in cookies of a request. However, when you explicitly pass allauth's "X-Session-Token" header, Django has no idea that it should look for session_key there. The solution that seems to work (with database-based sessions at least) is to inherit from default SessionMiddleware:

from django.conf import settings
from django.contrib.sessions.middleware import SessionMiddleware as _SessionMiddleware


class SessionMiddleware(_SessionMiddleware):
    def process_request(self, request):
        session_key = request.COOKIES.get(settings.SESSION_COOKIE_NAME)
        if session_key is None:
            session_key = request.headers.get("X-Session-Token")
        request.session = self.SessionStore(session_key)

and add this implementation into settings.INSTALLED_APPS in place of the original one.

Disclaimer: I'm very new to Django myself and haven't thought through all the security implications of this solution.