This happens when the keybinding is not set to toggle the terminal. Follow these steps::
ctrl+shift+P, open command palette
ctrl+shift+P
search for View: Toggle Terminal, go to gear icon to configure the keybinding
View: Toggle Terminal
set key binding