Here is a pretty dirty workaround that nevertheless achieves desired behavior.

(credit to @user2357112 for explaining process spawning)

mymodule is modified as follows:

/mymodule
├── __init__.py
├── __main__.py 
├── app.py
└── worker.py

__init__.py is empty

worker.py is unchanged

app.py contains the code from original __main__.py

__main__.py is a new file:

import sys
import mymodule.app

__spec__.name = mymodule.app.__spec__.name
sys.exit( mymodule.app.main() )

Now running module any of the three ways: python -m mymodule or python -m mymodule.__main__ or python -m mymodule.app produces the same result on Windows:

[__main__] [DEBUG]: I am main. I manage workers
[__main__] [INFO]: I am main. I manage workers
[__main__] [WARNING]: I am main. I manage workers
[mymodule.worker] [DEBUG]: I am a worker. I do stuff
[mymodule.worker] [INFO]: I am a worker. I do stuff
[mymodule.worker] [ERROR]: I am a worker. I do stuff
[mymodule.worker] [ERROR]: Here is my logger: <Logger mymodule.worker (DEBUG)>

In practice my code is more complex and takes care of cli arguments using argparse, but that is out of scope of this questions. I have tested this with python 3.11 and so far have not encountered any unexpected side effects.