So I can't get the service worker to stop being built, but you can tell firebase not to upload it. Add this to your firebase.json in the hosting section:

    "ignore": ["flutter_service_worker.js"]

If you are running in a browser that already installer the service worker for your app, you'll need to remove it otherwise it will stay around.