My bad, The AsyncNotifier automatically wraps the state into AsyncValue. So in my code, its a AsyncValue inside a AsyncValue which gave this behaviour. Converting state of AsyncValue<VirtualBook> to just VirtualBook solved the issue.
AsyncNotifier
AsyncValue
AsyncValue<VirtualBook>
VirtualBook