Found solve in tortoise-orm docs on Github. For prefetch only certain fields need to use tortoise.query_utils.Prefetch object.

query = Token.all().prefetch_related(
        Prefetch("provider", queryset=Provider.all().only("id", "description")),
    )