• Do you have a clue why this is happening?

Many things are all running on your system at the same time. The system shares available resources (CPU, memory bandwidth, device I/O) among them.

Your script does not have unconditional first priority for system resources. In fact, there are probably some occasional tasks that have higher priority when they run, and plenty of things have the same priority. It is not surprising that every once in a while, one of your transfers has to wait a comparatively long time for the system to do something else. If you need guarantees on how long your script might need to wait to perform one of its transfers, then you need to make appropriate use of the features of a real-time operating system.

• How can I fix this, or speed up the code?

You probably cannot prevent occasional elapsed-time spikes, unless you're prepared to install an RT OS and run your code there. Details of what you would need to do are a bit too broad for an SO answer.

With sufficient system privileges, you may be able to increase the priority of a given running job. That might help some. Details depend on your OS.

The usual general answer to speeding up Python code that is not inherently inefficient is to use native code to run the slow bits.

• Are there some general Python settings to prevent this behavior?

I don't believe so. The spikiness you observe is not Python-specific.

• How would you debug this?

I wouldn't. What you observe doesn't seem abnormal to me.