After not receiving a response for a week, I at least had some time to analyze the problem myself. Perhaps my question was too specific and complicated. The solution I found is surprisingly complex, as I need to override the custom methods for entering and exiting full screen mode.
In an NSWindowDelegate, you need to define the following optional methods to accomplish this.
func customWindowsToEnterFullScreen(for window: NSWindow) -> [NSWindow]? {
return [window]
}
func window(_ window: NSWindow, startCustomAnimationToEnterFullScreenWithDuration: TimeInterval) {
if let fullScreenFrame = window.screen?.frame {
normalFrame = window.frame
NSAnimationContext.runAnimationGroup { context in
context.duration = startCustomAnimationToEnterFullScreenWithDuration
window.animator().setFrame(fullScreenFrame, display: true)
}
}
}
func customWindowsToExitFullScreen(for window: NSWindow) -> [NSWindow]? {
return [window]
}
func window(_ window: NSWindow, startCustomAnimationToExitFullScreenWithDuration: TimeInterval) {
window.styleMask = normalStyle
NSAnimationContext.runAnimationGroup { context in
context.duration = startCustomAnimationToExitFullScreenWithDuration
window.animator().setFrame(normalFrame, display: true)
}
}
I found this after many failed attempts and some flaws I discovered in the Apple API. The working version is added as a new commit to the original files, which means you have to check out the first version if you want to reproduce the original problem.
The link as above: https://github.com/mac-curver/TestZoomAndFullScreen15_4