What else is running on your tableau server ? Does the Tableau server have enough resources like available RAM and cpu, are they the same or more as your local device ? are there other jobs running on Tableau server when you are pulling the data ? How many instances of backgrounder and other services are on your tableau server ?

It could be due to a lot of things. If it is an extract, the data should already be there on tableau server if you have an extract refresh schedule setup, shouldn't take long as it doesn't need to fetch the data. If the extract is old, maybe it's fetching new records and saving them on the Tableau server and that's could be while it's taking longer. Try setting up an extract refresh schedule and see if it helps.