According to the DeepFace [benchmarks](https://github.com/serengil/deepface/tree/master/benchmarks), FaceNet512 & RetinaFace pair is giving the highest accuracy score.
Also, my experiments make me feel that euclidean is overperforming than cosine.
Some models such as FaceNet, ArcFace are worth to try for your custom dataset because benchmark scores are retrieved on LFW dataset, but your custom dataset may overperform for any other configuration.