The Dialog has the open property, and it depends on the open value from handleDialog. You need to add the open value as true to render the Dialog component. Add it on modifiedProps in your test.
Dialog
open
handleDialog
true
modifiedProps