I think I figure out the solution myself, I want to post the solution for windows local machine here; thanks for @Wayne 's suggestion that "It's just that making it effectively work can be super tricky depending on your system".

I open the powershell of windows, type the following command

[System.IO.File]::WriteAllBytes("$env:TEMP\ctrl-d.txt", @(4))

Then I open the file using command:(Open a folder,type the following command in the address field)

%TEMP%\ctrl-d.txt

then ctrl-A + ctrl-C to copy the character to clipboard in windows system

paste that character into the interactive mode prompt

I am able to get back to ipdb normal mode instead of interactive mode.

You can see the result in the picture: