I spent a while working on this, and just found a solution.

I selected "Run only when user is logged on" (under General tab of the task's properties)

I think this is related to taskscheduler using a headless environment if you select "Run whether user is logged on or not" which caused issues for xlwings.