Calling FocusScope.of(context).unfocus(); before showing the dialog didn't solve the problem for me. The previously focused TextField was still getting the focus after closing the dialog.

So, I created a global function in an utility class. I call this every time I want to close a dialog. The focus doesn't go back to any focusable widget after closing the dialog.

void closeDialog(BuildContext context) {
    Navigator.of(context).pop();
    FocusScope.of(context).unfocus();
}