Ok, apparently I'm not the only one to think this is not good, and many others had the same issue:

• Add an option for Dection Models to return cost in eval mode as well (validation, test loss) · Issue #1574
• [models] Suggestion of GeneralizedRCNN forward output · Issue #1775
• Calculate Training Accuracy on resnet152 · Issue #2578