I came to a solution where I just added a tap gesture to the superview of my button to absorb the tap and not pass it along to the other superviews.