All the above hacks were working but partially, the real reason was that the component was not updating few of its internal states without re-render, and none of the hacks were forcing a re-render.
If you have no problem re-rendering your textfield then this will work like a charm. The answer posted above by @Ifpl might work, but here is the more cleaner version for triggering the re-render.
We can use this as long as the key prop is not a problem for us.

<TextField
  key={value ? 'filled' : 'empty'} // triggers re-render
  label="Your Label"
  value={value}
  onChange={(e) => setValue(e.target.value)}
  variant="outlined"
/>