I think what you are looking is a prefixIcon instead of prefixText (as it will only be visible when text field is focused on non empty).

For better understanding you can checkout prefixIcon property