I just read a bit of the Gesture article and came across hitSlop all I did was add control from above and it happened by itself
const panGesture = Gesture.Pan() .hitSlop({ top: 0, bottom: -height + 50 }) ...