InsightFace already has a model trained using a large dataset that considers a lot of races, angles, face shapes, etc. The data you built is very small, so unless you are aiming for overfitting, it will be difficult to achieve general performance better than the existing InsightFace model.

For models like InsightFace, it is better to focus on "how to use" those models well. If you want to create a face recognition model specialized for a specific dataset domain, it will be helpful to refer to the github of other face recognition SOTA models and train them.

https://github.com/deepinsight/insightface/tree/master/recognition/\_datasets\_

Please refer to this page. According to this page, they use at least 100,000 to 500,000 data as training data. Unless you build a dataset larger than this, it is difficult to judge that it will perform better than InsightFace or avoid overfitting.