The best fix is to turn off the x-axis for the colorbar's once it's been created. Try not use set_xticks([]). Instead, use:

cax.xaxis.set_visible(False)