You can write in this way,So you can Reduce time.

    // Get all contacts without thumbnail (faster)
List<Contact> contacts = await ContactsService.getContacts(withThumbnails: false);

In OtherWay,I can found flutter_contacts plugin.

This plugin was tuned to achieve optimal performance for fetching the whole list of contacts. Loading 1000 contacts with this plugin should take ~200ms or less depending on the device.