I managed to find a solution. I replaced flashcardSetRepository.delete(flashcardSet) with

flashcardSet.getUser().getFlashcardSets().remove(flashcardSet);

Full code:

@Transactional
public void deleteFlashcardSet(Long userId, Long flashcardSetId) {

    FlashcardSet flashcardSet = flashcardSetRepository.findById(flashcardSetId)
            .orElseThrow(() -> new ResourceNotFoundException("Flashcard set not found"));

    if(!flashcardSet.getUser().getUserId().equals(userId)){
        throw new UnauthorizedException("User does not have permission to delete this flashcard set");
    }

    flashcardSet.getUser().getFlashcardSets().remove(flashcardSet);
}

Now I don't have to use entityManager.clear() or direct query.