The solution ended up being quite simple for me as I was able to add this flag to make it package for windows:
electron-forge package --platform win32 // or electron-forge make --platform win32