Now I found how to fix it. Just simply add focusRestorer to modifier can fix it like:

Box(modifier = Modifier.focusRestorer()) {...}

We can also add a focusRequester to focusRestorer, in case of using our own focus logic.