Based on public documentation for creating custom training jobs there are 4 main points we must follow for us to successfully run a custom training job.
Before you submit a job
What a custom job includes
Configure distributed training
Create a CustomJob