You might want to try using click_and_hold instead of pointer_down, or maybe using pointer_up rather than release() might work better for you
click_and_hold
pointer_down
pointer_up
release()