The Jetson machines all have unified memory architecture (CPU and GPU share memory) so increasing the amount of workers typically reduces the performance and uses up the shared memory quickly since we are splitting which cores have access to what. The standard allocated shared memory is typically only a small portion of the full possible amount hence why you get issues with running multiple workers even though the AGX Orion has the specs it does.

Instead you should ideally run --worker 1 or in some cases --workers 0 to not segment your memory, if you are still running into issues or slow inference/training rates you can increase the shared memory with something like --shm-size=8g in your docker or the like :)