Did you ever find a solution or workaround to this issue? I'm experiencing the exact same problem. Takes ~120ms to close the port.