I tested some solutions and measured it with put my module between commands:

$ExecutionTime = [System.Diagnostics.Stopwatch]::StartNew() 

Import-Module -Name .\TestModule -Verbose -Force

$ExecutionTime.Stop(); $ExecutionTime.Elapsed

Best for my scenario was list of all functions and add that list into psd1 file:

FunctionsToExport = @(
        'Function1'
        'Function2'
        'Function3'
)

After I simply add them all, loading of my module increased. I will also remove in some scenarios '-Force' switch as suggested, when I specially know this module was already loaded before. This will probably handle with one more command to check is my module is already loaded, like in one of the answers of this topic: How do I check if a PowerShell module is installed?