There are numerous ways to increase the speed of code, using numpy arrays over standard lists for calculations is usually always faster (as you have already attempted).

-Secondary since your calculations are independent from each other (calculating distance a does not affect the calculation for distance b) you could try to parallelize your code by e.g. hyperthreading. \

-Tertiary, from my background in industrial engineering I know alot about the need to calculate distances. The trick here is not to (as you stated) implement conditions to NOT recompute things. Rather it is often way faster to look at exactly which points you do need to recompute!