Not how Mac Apps should behave, but the best workaround I could come up with is simply quitting the entire app when closing the window.

func windowShouldClose(_ sender: NSWindow) -> Bool {
    NSApplication.shared.terminate(self)
    return true
}