If there are any other UI elements in front of the button make sure you disable "Raycast Target" on their Image component.