@Marks's solution works if the panel never opens, but if it sometimes opens that can be annoying since it's a toggle. As far as I can tell there's no action to open the panel, but I cobbled something together with closePanel, togglePanel, and compound tasks:
{
"label": "close Panel",
"command": "${command:workbench.action.closePanel}",
"type": "shell",
"problemMatcher": [],
},
{
"label": "open Panel",
"command": "${command:workbench.action.togglePanel}",
"type": "shell",
"problemMatcher": [],
"dependsOn": [
"close Panel"
],
"runOptions": {
"runOn": "folderOpen"
}
},
Not the prettiest, but it gets the job done.