I have the same issue. I have two classes one-to-many (parent -> [child]) relationship. If I put the SwiftData relationship (explicit or inferred), even adding the records, the performance is degraded too much (Talking about 100k+ records). My only option was remove the relationship and add manually foreign keys to child entities, and modifying the queries to look for Ids.
What helped too was:
How do you insert an array of objects into the SwiftData ModelContext