Try to use better models for the embeddings, that bad embedding models could cause a decrease in accuracy.
See mteb-embedding-leaderboard