In my case using FocusScope.of(context).focusedChild?.unfocus(); instead of FocusManager.instance.primaryFocus?.unfocus(); solved my issue.
FocusScope.of(context).focusedChild?.unfocus();
FocusManager.instance.primaryFocus?.unfocus();